CI error: Unexpected segmentation fault encountered in worker

[deependujha]

🐛 Bug

A non-deterministic CI bug has recently started coming on mac (github action), where tests/streaming/test_dataset.py::test_dataset_resume_on_future_chunks[True] fails with segfault.

Check action: https://github.com/Lightning-AI/litdata/actions/runs/13076309775/job/36489260607?pr=443

https://github.com/Lightning-AI/litdata/actions/runs/12948719378/job/36117940606

Additional context

logs:

Progress:  25%|██▌       | 2/8 [00:13<00:39,  6.57s/it]
Progress: 100%|██████████| 8/8 [00:13<00:00,  1.73s/it]
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
  warnings.warn(
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
  warnings.warn(
ERROR: Unexpected segmentation fault encountered in worker.
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
  warnings.warn(
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
  warnings.warn(
ERROR: Unexpected segmentation fault encountered in worker.

=========================== short test summary info ============================
FAILED tests/streaming/test_dataset.py::test_dataset_resume_on_future_chunks[True] - RuntimeError: DataLoader worker (pid(s) 24332) exited unexpectedly
===== 1 failed, 213 passed, 11 skipped, 20 warnings in 1317.66s (0:21:57) ======
Error: Process completed with exit code 1.