China Mobile Liaoning Becoming a Paradigm in Transforming Storage Using FusionStorage

Background

Liaoning Mobile, a fully-owned subsidiary of China Mobile Limited (China Mobile), is responsible for the network construction and maintenance, and telecommunications operations of China Mobile in Liaoning. China Mobile was founded in July 1999 and listed in Hong Kong and New York in November 2000. After more than ten years of unremitting efforts, Liaoning Mobile has developed a comprehensive telecommunications network featuring wide coverage, quality communications, extensive services, and a leading service level.

Challenges

Liaoning Mobile is currently witnessing exponential data growth every year as the number of subscribers increases along with a service shift from traditional voice to new value-added services. Liaoning Mobile’s core Business Support System (BSS), Operations Support System (OSS), and Management Support System (MSS) adopted UNIX servers combined with traditional storage on the live network. Traditional storage systems could no longer accommodate new service requirements due to high service risks, high maintenance costs, and low resource utilization. In addition, the systems took a long time to recover data and failed to provide external services, which inconvenienced consumers and increased O&M workloads.

A Unique Road to Transformation

Many provincial branches of China Mobile faced three problems in their service support systems: slow response, expensive maintenance, and underutilized resources.

The fierce competition in the telecom industry has transformed many carrier business models, shifting from their traditional resource-intensive models to user demand-centric ones. This improves the significance of BSS and OSS, and the core data assets of these systems are stored on databases. In the past, most carriers in China deployed their core databases on traditional architectures composed of UNIX servers and centralized storage (midrange and high-end storage arrays), which required and often depleted large investments. Liaoning Mobile was no exception. It has adopted more than 300 UNIX servers from various vendors on its BSS and bears more than 100 systems comprising billing, network management, management information, value-added data, and location services. Each of the systems were configured with independent storage, leading to storage silos. With an increasing growth of service data, traditional storage systems were gradually considered high service risks due to their soaring maintenance costs and low resource utilization. The databases ran slowly and required more time to process services. Generating daily or monthly business reports took more than 10 hours. Capacity expansion for high-end legacy storage systems like the EMC DMX series required cabinet-level replacement, costing more than newly purchased devices. Loads on different UNIX servers varied dramatically. Loads for some devices were high at peak hours, but low on average. As a result, these devices ran under light loads for a long time.

To solve the preceding problems, Liaoning Mobile had two options: One was to use open source databases, which required great changes on traditional applications. Further, the open source databases failed to support some complex relational data models, making this option unsuitable. The other option was to use distributed databases, which required no modification to databases. This solution replaces UNIX servers and traditional storage with new servers and distributed storage. According to tests conducted by Liaoning Mobile, open source non-relational databases are better for Internet companies dominated by unstructured and semi-structured data. Liaoning Mobile mainly processes structured data, so it is difficult to reconstruct their legacy service system. 

Solution

Transformation

Huawei deployed its cloud-based FusionStorage to transform Liaoning Mobile’s core storage architecture and build a new distributed storage system. FusionStorage improved Liaoning Mobile’s overall database system performance. For example, the Business Analysis Support System (BASS) needed 10 hours to process 100 terabytes of data, and now it takes only 2 hours. In addition, the maximum number of concurrent BASS users increased from 400 to 1,000. Liaoning Mobile also enjoyed smooth capacity expansion and linear cost investment. Compared with traditional storage, FusionStorage saved nearly RMB 20,000 (US$3,027) per year on maintenance fees when replacing storage systems with 1 terabyte of capacity. Total investment needed to migrate data in the future is estimated at 30 percent less than previous models.

When using traditional storage to build cloud resource pools after the hardware warranty expired, Liaoning Mobile needed higher-level, larger-capacity, and next-generation products. In addition, the data migration service fees reached up to millions of RMB. In contrast, FusionStorage solutions only require the customer to replace the hardware that has expired, with data automatically synchronized between the legacy and new devices. FusionStorage supports service-unaware device replacement in batches and automated data synchronization, saving data migration costs and preventing service interruption.

Superb Performance and High Scalability

Huawei provided Liaoning Mobile with a storage resource pool solution powered by FusionStorage cloud storage. Featuring massive, on-demand scale-out capabilities, the solution removes storage silos and provides efficient block storage services for databases, placing Huawei ahead of its competitors.

Database applications require data block service capabilities featuring high bandwidth and low latency. Liaoning Mobile’s requirements can be summarized into two aspects: hybrid database types and robust system scalability. Liaoning Mobile has a variety of databases including a 4A (Authentication, Authorization, Accounting, and Audits) security management platform database, a Value-added service General Operation Platform (VGOP) database, a charging database, a Business and Operation Support System (BOSS) database, and a BASS database. Each database hosts dozens or even hundreds of services with diverse read/write models and user access habits. For example, the peak IOPS values of the OSS database (> 20,000), BASS database (> 20,000), and BOSS database (> 40,000) demonstrate Liaoning Mobile’s business scale.

Liaoning Mobile also requires robust system scalability. More than 10 application systems with nearly 1 petabyte of data can be shared over a converged resource pool that remains available, even when the data volume grows. The core difference between Huawei’s cloud-based FusionStorage and traditional SAN storage is scalability. Traditional SAN storage uses controller stacking for expansion, that is, a dual- or multi-controller stack can be expanded to dozens of controllers. The capacity and performance can be enhanced by adding disk enclosures to back-end controllers (scale-up). However, bottlenecks may occur when the disk enclosure volume reaches a certain scale. Adding disks can increase the total capacity, but the performance cannot be linearly improved, limited by the controller architecture.

It is no wonder that many customers are switching to Huawei. FusionStorage is designed with a fully distributed architecture. It virtualizes hundreds of servers into a virtualized and scaled-out storage pool using related software. Each server provides both computing and storage capabilities, and the storage resource pool can be expanded linearly as the number of servers increases.

In this project, FusionStorage groups SSDs on storage servers into a shared distributed cache resource pool which is shared by all service systems. The pool adopts a two-layer read cache mechanism (Layer 1: memory cache; Layer 2: SSD cache) to shorten the data access time. The average latency of common 4 kilobyte data block reads and writes is about 1 millisecond, and the Distributed Hash Table (DHT) algorithm is compatible with high-performance hardware (100 percent NVMe SSD configuration), which helps shorten the data read/write time and increase the number of concurrent users from 400 to 1,000. Together, all these features satisfy the most demanding performance requirements of front-end services.

Moreover, FusionStorage uses InfiniBand switches with large throughput and low latency in internal networking, preventing Fibre Channel switching performance bottlenecks common in traditional storage systems. The internal data exchange bandwidth is expanded from 10 Gbit/s to 128 Gbit/s, greatly improving I/O performance and preventing I/O bottlenecks when hundreds of customers concurrently access a database. According to the BASS test results, FusionStorage takes only 2 hours to process and analyze 100 terabytes of data (traditional storage needs 10 hours for the same process).

All-Around Data Protection

In terms of data protection, FusionStorage uses cluster management and provides multi-level high availability designs at disk, node, cabinet, and system levels. Each disk stores multiple data blocks (partitions). Copies of these data blocks are distributed to other nodes in the system based on pre-set policies. When detecting a disk or server fault, FusionStorage automatically repairs data in the background. Data block copies are distributed to different nodes. When data reconstruction is performed on multiple nodes simultaneously, each node rebuilds only a small amount of data. When reconstructing a large amount of data on a single node, multiple nodes work in parallel, avoiding performance bottlenecks without interrupting database services.

Traditional storage systems on Liaoning Mobile's live network adopt RAID technologies, and only 10 or more disks are involved in data reduction. In addition, due to the bottleneck in independent disk data parity, the data protection efficiency of traditional storage systems is lower than FusionStorage. FusionStorage uses synchronous data write/read recovery technologies to ensure data consistency and provide a more reliability than traditional high-end storage systems.

Liaoning Mobile also benefits from the distributed mechanism and high performance of FusionStorage for its data backup and Disaster Recovery (DR). Compared with traditional backup systems, FusionStorage improves the backup efficiency by more than 50 percent, and ensures data consistency through backup software. In the future, FusionStorage will be able to construct data lakes specific to demands in block, file, object, and backup scenarios in a comprehensive data protection ecosystem.

Liaoning Mobile utilizes distributed technologies to implement smooth system reconstruction at three layers: the application layer, the platform layer, and the data layer. Liaoning Mobile has become a paradigm in the industry for its active exploration and continuous innovation, especially with regards to storage.

What’s more impressive, Liaoning Mobile’s innovations have paid off. Since the reconstruction of its servers and distributed storage systems, nearly 60 percent of Liaoning Mobile’s UNIX servers and traditional high-end storage systems on the live network have been replaced. The total data volume of the new distributed block storage systems has exceeded 10 petabytes.

Liaoning Mobile’s distributed storage systems have reduced IT system investments and maintenance costs by an estimated 30 percent in total (based on the volume of data to be migrated). Similarly, the new database platform helps Liaoning Mobile shorten the time to output daily and monthly reports, greatly improving user satisfaction. Liaoning Mobile’s new solution helps reduce the procurement quantities of UNIX servers and high-end storage devices, facilitating energy saving and emission reduction to achieve sustainable development goals. Thanks to Huawei and FusionStorage, Liaoning Mobile now has an outstanding operational system to build its business support network for today, and tomorrow.

Benefits