Abstract: Compression of large datasets in S-NPP, JPSS and other HDF5 data files may substantially reduce their size, in turn reducing disk space requirements and file download time. However, mismatches between the layout of the files' datasets, the HDF5 instance's cache settings, choice of compression algorithm, and the access pattern of applications using the data can sometimes result in poor performance or exhausting machine resources when running the application. Whether designed in advance or modi...

Source: Overcoming Pitfalls When Using HDF5 Compression