长时间序列格点数据管理平台的设计与实践
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:


Design and Practice of Long-Term Sequential Grid Data Management Platform
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    随着数值模式时空分辨率的提升,数据量急剧增加,长序列数据很难直接通过文件拷贝或者网络传输方式为用户提供数据服务。为此,笔者设计实现了一种分布式管理平台,该平台根据用户定制的数据需求,运用预报要素、空间范围、时间尺度等约束条件,抽取或根据区域参数裁剪指定气象要素,生成精简数据进行用户服务。该平台集成了搜索引擎、格点数据解码、内存数据库技术以及分布式框架,实现跨操作系统的统一接口调用和数据快速获取,有效解决用户访问长时间序列历史资料的难题。实验测试显示,该平台在格点数据管理规模和访问效率方面均表现出色。特别是在北京2022年冬季奥运会和冬残奥会气象保障服务中,该平台发挥了重要作用,展现了其实际应用的价值和潜力。

    Abstract:

    With the rapid development of numerical weather prediction services, the resolution and forecasting lead time of meteorological models have significantly improved, leading to an exponential growth in the volume of forecast data output. As a national meteorological model research and operational centre, CMA Earth System Modeling and Prediction Center (CEMC) currently produces daily gridded data outputs of 0.76 TB, with an annual output reaching 155.12 TB. Given the enormous data volumes, researchers’ preferences for data access are evolving. Wagemann predicts that future scientific users increasingly prefer cloud platforms or other interfaces for data access rather than solely relying on downloads. To address these issues, this paper proposes a lightweight distributed parallel processing framework for gridded data management, aiming to streamline data management processes and enhance data access speed. The core design philosophy revolves around leveraging search engine technology for rapid metadata retrieval and gridded data decoding techniques for efficient data acquisition. To mitigate performance penalties from repetitive decoding, the framework decodes gridded data files once and supports multiple retrievals and extractions, significantly accelerating data access. Additionally, it supports cross-platform data access, facilitating easier data acquisition for researchers. The framework adopts a three-tier architecture: the data layer stores data, the algorithm layer implements core search and cataloguing algorithms, and the business layer interfaces directly with user needs. The framework implements crucial functions such as gridded data cataloguing, extraction, and clipping. During cataloguing, users invoke the cataloguing interface and input parameters (e.g., original data file paths, index names, index types), and the system automatically parses file metadata and generates indexes. For data extraction, users call the retrieval interface with specific parameters to obtain designated data. Moreover, the framework supports precise extraction of specified latitudinal and longitudinal data segments by configuring cropping parameters. It reduces decoding time by creating indexes based on binary storage characteristics, utilises an inverted index value-id model for rapid data location retrieval, enhances processing performance through GlusterFS shared storage and Celery distributed message queues, and ensures efficient and stable data transmission using gRPC technology for C/S communication. Practical tests and applications demonstrate the framework’s exceptional performance in handling massive meteorological data. Notably, it successfully processes petabyte-scale gridded data during the Beijing Winter Olympics meteorological support services, significantly improving data access efficiency. Additionally, the framework supports flexible processing and scalable upgrades for various file formats to meet diverse user needs. By integrating advanced search engine technology, gridded data decoding methods, and a distributed cluster framework, the platform not only enables rapid data retrieval and efficient access but also satisfies researchers’ urgent demand for cross-platform data access. As meteorological data continues to grow, this platform holds significant potential to play a pivotal role in various fields, offering more robust data support for weather forecasting, scientific research, and operational applications.

    参考文献
    相似文献
    引证文献
引用本文

贾晓振,胡江凯,王大鹏,梁晨.长时间序列格点数据管理平台的设计与实践[J].气象科技,2024,52(6):797~806

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2023-12-12
  • 定稿日期:2024-10-09
  • 录用日期:
  • 在线发布日期: 2024-12-25
  • 出版日期:
您是第位访问者
技术支持:北京勤云科技发展有限公司