Safetensors

[ncassereau]

The Huggingface library Safetensors seems to be able to load data from storage much faster than the native torch.load. The writing process is not faster (could even be 10% slower from my preliminary tests) but the loading process is very fast. Considering that you are using very large tensor, that could prevent the dataloader from being the bottleneck of the training process.

Tests should be carried out to see how good (or bad) it is with a standard training loop but that could be an option for ClinicaDL. Because data would be stored in a different format, it would probably be only a flag but still.

It comes from Huggingface so the dependency is somewhat reliable. The only requirement is that tensors are contiguous.

https://github.com/huggingface/safetensors