pipsCloud: High performance cloud computing for remote sensing big data management and processing

Massive, large-region coverage, multi-temporal, multi-spectral remote sensing (RS) datasets are employed widely due to the increasing requirements for accurate and up-to-date information about resources and the environment for regional and global monitoring. In general, RS data processing involves a complex multi-stage processing sequence, which comprises several independent processing steps according to the type of RS application. RS data processing for regional environmental and disaster monitoring is recognized as being computationally intensive and data intensive. We propose pipsCloud to address these issues in an efficient manner, which combines recent cloud computing and HPC techniques to obtain a large-scale RS data processing system that is suitable for on-demand real-time services. Due to the ubiquity, elasticity, and high-level transparency of the cloud computing model, massive RS data management and data processing for dynamic environmental monitoring can all be performed on the cloud via Web interfaces. A Hilbert-R⁺-based data indexing method is employed for the optimal querying and access of RS images, RS data products, and interim data. In the core platform beneath the cloud services, we provide a parallel file system for massive high-dimensional RS data, as well as interfaces for accessing irregular RS data to improve data locality and optimize the I/O performance. Moreover, we use an adaptive RS data analysis workflow management system for on-demand workflow construction and the collaborative processing of a distributed complex chain of RS data, e.g., for forest fire detection, mineral resources detection, and coastline monitoring. Our experimental analysis demonstrated the efficiency of the pipsCloud platform.