Swanepoel, DerickVan den Bergh, Frans2021-08-092021-08-092017-03Swanepoel, D. & Van den Bergh, F. 2017. Multi-threaded compression of Earth observation time-series data. <i>International Journal of Digital Earth, 10(12).</i> http://hdl.handle.net/10204/120771753-8947http://dx.doi.org/10.1080/17538947.2017.1301580http://hdl.handle.net/10204/12077Earth observation data are typically compressed using general-purpose single-threaded compression algorithms that operate at a fraction of the bandwidth of modern storage and processing systems. We present evidence that recently developed multi-threaded compression codecs offer substantial benefits over widely used single-threaded codecs in terms of compression efficiency when applied to a selection of moderate resolution imaging spectroradiometer (MODIS) datasets stored in the HDF5 format. Compression codecs from the LZ77 and Rice families are shown to vary in efficacy when applied to different MODIS data products, highlighting the need for compression strategies to be tailored to different classes of data. We also introduce LPC-Rice, a new multi-threaded codec, that performs particularly well when applied to time-series data.AbstractenData compressionMulti-threadingTime-seriesHierarchical Data FormatHDFHDF5Moderate resolution imaging spectroradiometerMODISMulti-threaded compression of Earth observation time-series dataArticleSwanepoel, D., & Van den Bergh, F. (2017). Multi-threaded compression of Earth observation time-series data. <i>International Journal of Digital Earth, 10(12)</i>, http://hdl.handle.net/10204/12077Swanepoel, Derick, and Frans Van den Bergh "Multi-threaded compression of Earth observation time-series data." <i>International Journal of Digital Earth, 10(12)</i> (2017) http://hdl.handle.net/10204/12077Swanepoel D, Van den Bergh F. Multi-threaded compression of Earth observation time-series data. International Journal of Digital Earth, 10(12). 2017; http://hdl.handle.net/10204/12077.TY - Article AU - Swanepoel, Derick AU - Van den Bergh, Frans AB - Earth observation data are typically compressed using general-purpose single-threaded compression algorithms that operate at a fraction of the bandwidth of modern storage and processing systems. We present evidence that recently developed multi-threaded compression codecs offer substantial benefits over widely used single-threaded codecs in terms of compression efficiency when applied to a selection of moderate resolution imaging spectroradiometer (MODIS) datasets stored in the HDF5 format. Compression codecs from the LZ77 and Rice families are shown to vary in efficacy when applied to different MODIS data products, highlighting the need for compression strategies to be tailored to different classes of data. We also introduce LPC-Rice, a new multi-threaded codec, that performs particularly well when applied to time-series data. DA - 2017-03 DB - ResearchSpace DO - 10.1080/17538947.2017.1301580 DP - CSIR J1 - International Journal of Digital Earth, 10(12) KW - Data compression KW - Multi-threading KW - Time-series KW - Hierarchical Data Format KW - HDF KW - HDF5 KW - Moderate resolution imaging spectroradiometer KW - MODIS LK - https://researchspace.csir.co.za PY - 2017 SM - 1753-8947 T1 - Multi-threaded compression of Earth observation time-series data TI - Multi-threaded compression of Earth observation time-series data UR - http://hdl.handle.net/10204/12077 ER -19631