In the late 1990s, something happened at a major branch of China Merchants Bank (CMB) in a provincial capital: Some banks dispatched several armored cars to this branch, telling CMB’s customers they can withdraw their money if CMB can’t find its way out. CMB employees called in family members to help carry the money, and the employees smiled through their tears when saying goodbye to customers. No one complained. A few weeks later, the branch was back to business as usual. Customers had brought back the money they had withdrawn from the branch along with money they had withdrawn from other banks. In a short time, deposits in this CMB branch had doubled.
This story is one small step in the development of CMB, China’s first share-holding commercial bank whose shareholders include legal enterprise entities.
CMB was also the first Chinese bank to hold umbrellas for customers coming in and out of the bank on rainy days, own the auto call distributors, and provide milk to customers. These kinds of thoughtful touches have helped CMB develop rapidly over the past 30 years. In 2017, the bank ranked 23rd in the Banker’s Top 1000 Banking Brands, and 216th in the Fortune Global 500.
Three Thoughts on CMB’s Digital Process
Over the past 17 years since joining CMB in 2001, I have witnessed the bank’s ongoing digital transformation initiatives. In recent years, technologies such as cloud computing and Big Data have developed quickly. Keeping pace with the times, CMB introduced these innovations to achieve rapid service development and better serve customers. Several of the changes have impressed me deeply.
First is the rapid development of infrastructure. From the bank’s Shekou equipment room to the data center in Nanshan and the Nanjing disaster recovery data center, or the data centers in Shanghai and Pinghu, the upgraded facilities and expanded capacity are meeting CMB’s increased service requirements.
Second is the impact of the Internet. CMB’s top executives believe technologies can bring disruptive changes to banks — more so than strict supervision, small loan companies, or Internet financial companies. Therefore, CMB is attaching great importance to new technologies. For example, 30 percent to 40 percent of the employees in the CMB business department spend 30 percent to 40 percent of their time on technology-related work and training. In addition, special funds have been established for innovative projects to support better service development.
These efforts have paid off. For example, CMB has built a unified risk control platform for credit cards based on Huawei’s FusionInsight Big Data solution. The FusionInsight platform reduced the number of problem cases by 50 percent and saved more than CNY 100 million in six months. This platform also shortened the time to issue a credit card from 15 days to five minutes. CMB now supports the ability to issue loans up to CNY 300,000 within minutes. Further, the bank is leading the market to provide innovative services such as withdrawal authentication and flash payments.
The third change is what has impressed me the most: CMB’s commitment to a customer-centric concept. With the slogan ‘We are here just for you,’ we bear in mind that providing good services for customers is our foundation. We use FinTech to facilitate our business operations and better serve our customers. We use every technology to satisfy customer needs, improve customer experience, and create larger value. Our goals are more benefits, greater convenience, faster service, and considerate services for customers.
Values Internet Financial Enterprises Cannot Match
The CMB Application and Database Management Office has been seeking answers to many questions: How do we provide better services for customers and businesses? How do we reduce costs? How should we set up our networks, and what architecture will be best?
CMB has strict architecture standards, such as read/write separation, database partitioning, active-active backup, and stateless multi-active operation. The bank has established a disciplined capability for implementation, and the gradual use of standard architectures has changed our standard for managing databases. In the past, faults had to be rectified immediately or services could not be restored. Now, with the standard architectures of today, faults have little or no effect on services. This use of standard architectures is an evolutionary way of thinking, as well as a great improvement in reliability.
CMB’s efforts in these areas differ from those of Internet financial enterprises to some degree. We implemented high-availability, high-scalability, and high-flexibility systems under strict supervision and predicated on customer security and experience. I can definitely say that CMB now bears favorable comparison to Internet companies in terms of database architecture standards and implementation capabilities.
Constructing the Optimal Distributed Relational Database in the Financial Industry
Service innovations depend on IT and data. Therefore, it is critical to find a way to ensure information security and optimize the database.
If only one machine is used, its upper limit is fixed and once that limit is reached more database instances are required. This requirement leads to new problems such as higher management costs. Before database partitioning, only one or two databases would be managed, but with partitioning, 10 or more databases must be managed. In addition, the probability of faults remains as high as when only one database is used. The result is that the overall availability has not increased.
Database partitioning and horizontal expansion reduce the dependency on a single database. This approach seeks to balance the tradeoffs among resources, costs, availability, and development difficulty. The solution is to have a distributed database, which represents a trend in database development. A distributed database offers several advantages. First, it reduces cost, including hardware, labor, development, and O&M costs. Second, a distributed database simplifies development and O&M so that IT personnel have a smaller workload. Most importantly, this type of database achieves better utilization of hardware resources and provides a higher unit output rate, so the database can better support more services.
Why is the trend to use distributed relational databases so widely recognized, and why are there so many people engaged in it? First, the relational database will never be outdated. Although this type of database has been under development for over 40 years, it is still used in scenarios with strict consistency requirements. Second, single databases are encountering bottlenecks. In addition to throughput demands, the number of users and concurrent transactions have reached unprecedented levels and will continue to grow. In the future, once breakthroughs are made in technologies such as quantum computing, the transaction volume will again increase explosively. In this context, the distributed relational database is one of the best choices.
Based on experience and a commitment to go further, CMB has decided to engage in a joint innovation program with Huawei in the distributed database field. We aim to build the optimal distributed database for the financial sector that will give CMB’s services a competitive edge. Both companies will rise to the challenges of ‘Cloud First’ by leveraging technologies such as cloud computing, Big Data, and AI, as well as leading financial business practices and high-quality resources to connect services and technologies. Further, we will jointly develop the distributed database and put the products into use, so as to migrate database applications to the cloud.
The bank’s existing open-source database kernel does not have performance and function comparable to that of a proprietary Oracle database. In addition, we have issues with scaling large clusters, low cost-effectiveness, and high maintenance costs. By cooperating with Huawei, CMB hopes to build a competitive distributed database for finance with high performance, high security, high reliability, and high scalability. We will use technologies such as latch-free data structures; NUMA-aware architectures (where NUMA is Non-Uniform Memory Access); 3D-XPoint memory; high-performance distributed transaction processing, computing, and storage separation; Remote Direct Memory Access (RDMA); Group Buffer Pools (GBPs); and Z-Paxos protocol. These technologies will be used to build cloud-oriented scale-up/scale-out capabilities.
Over the years the financial industry has gradually refined the core value of the database, which is all about consistency. Databases play a critical role in data processing and solving all difficulties regarding consistency. Problems that can be solved by applications should not be handed over to the database, as databases may not be appropriate for problems better solved by applications. Moreover, coping with such problems bears a high cost and affects database capabilities, including performance and capacity. In addition, fault probabilities are higher.
Based on these concepts, CMB has a clear specification for all online transaction systems. For example, even when using an Oracle database with the capacity for 500 SQL statements, CMB may only use 10 of them. This policy simplifies the database function requirement, which shortens the development cycle.
Mutual Trust Builds the Foundation of Cooperation
The customer-centric concept is the first reason why CMB chose Huawei. CMB is born to serve customers, and gives the highest priority to customer satisfaction. That is also true for Huawei. With a shared spirit of excellence, Huawei and CMB respect, trust, and appreciate each other. Second, we believe in the strength that Huawei has accumulated with over 10 years of experience in the database field — including both in-memory and disk-based databases. The company has many successes in multiple business domains.
We also appreciate Huawei’s service-minded approach and ability to tackle difficulties. About six years ago, Huawei left me with a deeply positive impression when CMB chose Hadoop products. At that time, several vendors, including Huawei, offered to provide Hadoop cluster products and services. After hearing CMB’s six challenges, two of the vendors said the project was too difficult and dropped out. Only Huawei was willing to take on the challenge. Five months later, Huawei reported to CMB that five of the issues had been completely resolved and only half of the sixth issue remained. This interaction demonstrated to me that Huawei is determined to deliver on projects and deal with difficulties with service awareness and cooperation. This is the foundation of our mutual trust.
Based on this trust and cooperation, how can the two companies use online trading systems and databases in the future? What services and capabilities need further development? CMB is a typical bank. Our interaction with Huawei is to propose advanced requirements and challenging functional attributes. Huawei sees that CMB has this capability and that the database developed by the two companies applies to banks as well as to businesses in other vertical markets supported by Huawei. I think that is why both parties choose each other to achieve win-win cooperation.
Huawei is planning to develop a database on the public cloud, and CMB can assist Huawei in developing such a database. By participating in this project, CMB will study to better understand the development trend for database technology, as well as the planning and design for core databases. This project will be highly beneficial to CMB, especially in terms of talent cultivation.
The joint innovation between CMB and Huawei has three phases:
• Initial phase: Focus on commercial pilot projects in 2018.
• Growth phase: Reach industrial scale in 2019.
• Stable development phase: Carry out large-scale promotion and replication activities in 2020.
CMB is responsible for the design of requirements and solutions, and Huawei’s Online Transaction Processing (OLTP) database team is responsible for technology implementation. Independent innovation will be conducted based on Huawei’s experience, and infrastructures will be integrated based on new hardware capabilities. In this way, the project will achieve the overall objective of a high availability cloud-based deployment with high security, high performance, low cost, and differentiated competitiveness.
A three-layer product architecture will be adopted. The top level is the distributed extension layer. In the middle level, the enterprise core layer will support the high performance and general database capabilities for enterprise-level services. At the bottom level is the distributed storage and cloud storage layer. This architecture supports the vertical integration of software and hardware to deliver high-performance, high-availability, and cloud-native database capabilities.
I firmly believe that the distributed financial database jointly developed by Huawei and CMB will contribute to CMB’s digital transformation and help CMB become a successful FinTech bank.