New Apps, New Data, New Resilience — Continuously Building Reliable Financial Data Infrastructure

[Shanghai, China, June 8, 2023] Dr. Peter Zhou, President of Huawei IT Product Line, delivered a keynote speech at the Huawei Intelligent Finance Summit 2023 in Shanghai.

He noted that the trend of "new apps, new data, new resilience" is creating five major challenges, such as long-term coexistence of stable and agile architecture and generative AI, for the financial sector. To build reliable financial data infrastructure and effectively handle these challenges, we require continuous innovations in multiple areas, including decoupled storage-compute new digital core distributed databases, cloud-native container storage, AI data storage, and storage-intrinsic resilience solutions.

The financial sector currently faces five key challenges due to unprecedented changes: the rapid development of data apps driving enterprises to embrace the cloud-native solution; explosive data growth and massive amounts of unstructured data (such as big models) entering enterprises' production decision-making systems; and artificial risks and ransomware attacks. Huawei provides the following suggestions to address the challenges:

First, long-term coexistence of stable and agile architecture is a common problem faced by customers in the financial sector. To maintain stable financial services, the core requirements of the financial architecture are μs-level latency and high data consistency. Therefore, all-flash primary storage is necessary. The key factors in maintaining agile financial services are quick service rollout and flexible resource scaling. This means the storage architecture needs both the primary and scale-out storage in order to meet various service requirements. Financial institutions must effectively deploy their data infrastructure to coordinate both primary and scale-out storage.

Next up, cross-cloud and -region data center deployment has become the new normal in the finance sector. Flexera's 2022 State of the Cloud Report shows that 89% of global enterprises are planning to transition to a multi-cloud business strategy, which not only fully utilizes both public and private clouds, but meets enterprises' requirements for data resilience and prevents cloud vendor lock-in.

But this deployment can easily lead to data silos. To handle this potential issue, the global file system (GFS) is employed to power cross-cloud data flowing and sharing, thus improving data access efficiency. Furthermore, high availability (HA) solutions, such as Huawei's geo-redundant 4DC deployment, help meet cross-region service collaboration requirements.

Cloud native technologies are widely used in cloud platforms, meaning efficient and reliable container storage is essential. Container Storage Interface (CSI) and Container Disaster Recover (CDR) plugins help interconnect container storage with mainstream container cloud platforms, thereby accelerating resource provisioning and realizing app-level disaster recovery across clusters. Additionally, in core distributed financial scenarios, the traditional coupled storage-compute distributed database solution is plagued by problems like complex performance improvement, insufficient availability, and low resource utilization. Therefore, a high-performance, high-reliability storage base that decouples storage and computing capabilities is essential for the distributed database solution.

The third point is that big data and generative AI are being applied by the financial industry. Big data is widely used in scenarios like precision marketing and real-time risk control. Typically, the ability to store and process huge data volumes is too costly and inefficient for traditional coupled storage and computing platform. These issues, however, are resolved with integrated data lakes and warehouses, using the decoupled storage and compute platform to improve resource utilization and lower storage costs.

Then, generative AI enables the transformation to new data paradigm and storage architecture and is widely used in scenarios such as personalized financial management and intelligent customer service. For example, financial institutions are using professional AI data storage to perform parallel training of large models and handle multifold increase in parameters and datasets. Through channel-associated processing of training data, vectorized indexing, and efficient backup and recovery of checkpoints, these organizations can slash data preprocessing from days/weeks to hours or even minutes.

Furthermore, organizations around the world are adopting energy efficiency initiatives to deal with the demand of less energy. The need for green development goals has seen mainstream agreements, with many international and industry policies focusing on low-carbon financial data centers. In the Huawei Global Industry Vision (GIV) report, it is expected that 1 yottabyte (YB) of data will be generated worldwide each year by 2030. Storage accounts for 20% of the total energy consumption in data centers, and in this background, there is a demand for both efficient storage of huge data volumes. High-density, all-flash storage helps both reduce power consumption in a smaller equipment room space while improving heat dissipation efficiency. Replacing one HDD with an SSD model reduces carbon emissions equivalent to the carbon uptake of 150 trees. In practice, all-flash storage has shown to help a top financial institution reduce power consumption by 30%.

Last but not the least, data threats are changing the focus of data protection from physical damage to human-made disaster. Traditional data resilience systems are unable to adapt to the new challenges of the digital world, and now, data storage has become the last line of defense for data resilience. Financial institutions need robust storage systems with multi-layer protection and intrinsic resilience through built on technologies like ransomware identification, secure snapshots, backup storage, and physical isolation zones.

This climate has been echoed by Dr. Peter Zhou, who said, "Facing the future trends of new apps, new data, and new resilience, Huawei Data Storage will continue to discuss the direction of developing IT infrastructure with financial partners and customers. We will keep launching innovative products and solutions to meet future business requirements and facilitate digital transformation of the financial industry."