Boost read and write of numpy array combined with high compressive ratio

[tjk9501]

Hello,

I have a very large 2d numpy (cupy) array, Rt, whose dimension is 512512/32 by 240300*200/40, and now I want to achieve fast reading and writing of the array between local disk and memory. I have tried many methods, including cupy.save, cupy.savez, etc. It turns out that all the functions do not have enough compressive ratio, which result in 24GB disk space consumption. Only using cupy.savez_compressed and

import cupy as cp
Rt_cpu = pandas.DataFrame(data=cp.asnumpy(Rt_gpu))
Rt_cpu.to_pickle(filename,compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1})

can compress Rt down to hundreds of MBs. But reading and writing Rt from and to local drive costs 60 seconds, which is very slow. We want to achieve 10 times speed up of reading and writing between RAM and local disk compared with pandas.to_pickle() function and maintain same degree of compressive ratio as cupy.savez_compressed. Does anyone have a solution or provide zarr code examples showing how to read/write the Rt between local drive and memory?

Thanks!!!!

[jakirkham]

Would suggest checking out KvikIO, which can enable GDS backed IO:

• https://github.com/rapidsai/kvikio
• https://docs.rapids.ai/api/kvikio/stable/api.html#kvikio.zarr.GDSStore
• https://zarr.readthedocs.io/en/stable/tutorial.html#storage-alternatives

Also here's a blogpost using KvikIO with XArray:

• https://xarray.dev/blog/xarray-kvikio

cc @madsbk