Huawei Releases Hyperion Research White Paper on the Trend of Data-Intensive HPC

[Shenzhen, China, October 9, 2021] Huawei released the White Paper on the Trend of Data-Intensive HPC, partnership with Hyperion Research. The White Paper shows that while the overall High-performance computing (HPC) market is projected to grow at a 6.8% CAGR over the from 2019 to 2024, the HPDA portion of the market, inclusive of HPC-enabled AI, is expected to grow at a much stronger 17% 5-year CAGR, and the AI portion at an even faster rate of 33% 5-year CAGR. The future HPC market has shifted its focus from compute-intensive to data-intensive workloads.

HPC was mainly used in research institutes. This situation has completely changed with the democratization of HPC brought about by commercial off-the-shelf technologies that have made HPC affordable for a variety of fields. These innovations have also accelerated the adoption and development of big data analytics and AI in many industries.

From computing-centric to data center-centric, HPC is undergoing a game-changing evolution. This is driven by the demands of data-intensive applications and is a result of cutting-edge techniques such as big data and AI. It's worth noting that three trends are gaining momentum: ever-growing data volumes, higher data reliability demands, and varying data workloads.

Meanwhile, data-intensive HPC poses higher requirements on storage, which is the key to improving data analysis efficiency.

The report recognizes the rapidly evolving requirements being placed on the HPC ecosystem by the growth in adoption of data-intensive applications and workloads. This is in addition to continued demands of the traditional HPC mod/sim applications, which are also being augmented by HPDA/AI. Faster generation of more data by HPDA/AI/ML/DL techniques is straining the existing HPC storage ecosystem. Addressing and optimizing for both types of workloads will require acute attention be paid to the storage portion of the HPC infrastructure.

HPDA is not a new concept. IDC proposed a similar idea back in 2014. High-performance data analytics (HPDA) refers broadly to data-intensive workloads utilizing HPC resources, including big data and AI workloads. Certain vertical applications, such as autonomous driving, genome sequencing, movie rendering, precise weather forecasting, and financial fraud detection, have demonstrated a greater propensity to adopt and leverage HPDA and HPC-enabled AI techniques than others. This further necessitates the evolution from HPC to HPDA.

To succeed in the data-intensive HPC market, storage providers need to fully understand and meet the demands placed on systems by HPDA loads that go beyond traditional HPC modeling/simulation requirements. For example, Huawei OceanStor Pacific series, the next-generation HPDA storage accelerates the evolution of HPC to data-intensive with ultra-high-density design, hybrid workloads-oriented, and multi-protocol interworking. Currently, the OceanStor Pacific series has been successfully applied to high-performance computing scenarios such as weather prediction and satellite remote sensing.