Constructing the World's First SKA Prototype with Huawei OceanStor Mass Data Storage
Enterprise products, solutions & services
Shanghai Astronomical Observatory has deployed Huawei OceanStor mass data storage to build prototype core service storage architecture for the Square Kilometer Array (SKA), an international scientific project that aims to build the world's largest radio telescope, with the aim of eventually having a collection area of over a square kilometer in total.
Conceived in the early 1990s and initiated by more than 10 countries, the telescopes that form SKA will be installed in areas with little radio interference, across sites located in eight countries including Australia and South Africa. Shanghai Astronomical Observatory released its SKA regional center prototype at the SKA Shanghai Meeting in November 2019 and it was recognized by the SKA Organization as the first to be developed, As such, it was selected as one of the top 10 national astronomical science and technology advances in China in 2019.
SKA will produce the largest data flow ever seen in astronomy's history. Just the first phase of SKA — which accounts for just 10% of the entire project — will create data at a growth rate of approximately 106 PB each year once operational. By 2027, the data volume generated by SKA is expected to reach 760 PB.
Clearly, the construction of supporting storage systems is critical for the project, given such data volumes as well as associated High-Performance Computing (HPC)clusters and a global SKA user base.
These storage systems are going to have to offer extremely high bandwidth and low latency, to support those HPC clusters and process literally astronomical big data. Dynamic capacity expansion and a modular design, to maximize Return On Investment (ROI), are also key, with the ability to smoothly expand critical to minimizing Total Cost of Ownership (TCO). And, of course, total reliability with zero tolerance for data loss is a given.
Huawei OceanStor mass data storage, including its Distributed Hash Table (DHT) algorithm and compatibility with high-performance hardware, with all Non-Volatile Memory express (NVMe) Solid-State Drives (SSDs) configured, proved that it was more than capable of supporting the largest SKA workflow test ever conducted.
Indeed, OceanStor's innovative, layered read cache mechanism considerably shortens data access times. In the event of a 4 KB data read/write, average latency is approximately 1 ms. In terms of internal networking, high throughput and low latency InfiniBand switches are used to expand the internal data exchange bandwidth, from 10 Gbit/s to 128 Gbit/s, accelerating data access and improving Input/Output (I/O) performance to support the processing of astronomical big data.
With an innovative Erasure Code algorithm ensuring data reliability, fully distributed architecture allows for dynamic and flexible capacity expansion. Performance increases linearly as more nodes are added, meeting the data growth needs of the SKA regional center over the next few years.
