No space left on device regression on ubuntu-latest since February 5th

[tomaarsen]

Description

(See also #9329)

As of February 5th, I started noticing that my ubuntu-latest tests would start failing due to System.IO.IOException: No space left on device, while my windows-latest tests still consistently pass.

See here the first test failure with that error. I'm unsure how much space my CI tests actually use up, but I suspect that it's between 18 and 20GB. According to the GitHub docs here, the ubuntu-runner should have 150GB of space. I recognize that this is reduced by all of the installed tools, but one way or another, something changed on ~February 5th that seemingly caused the runners to have less space available.

Could it be that I am accidentally being given runners for private repositories? Those have 14GB of space, which would indeed not be enough.

It seems like there was indeed a release on February 5th, which may have caused this discrepancy. I would love some help to get my CI to pass again!

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240201.1.0

https://github.com/UKPLab/sentence-transformers/actions/runs/7780698356

Is it regression?

20240126.1.0

Expected behavior

(I can't provide a link to the latest successful build in the last question, so I'll add it here: https://github.com/UKPLab/sentence-transformers/actions/runs/7752303635/job/21141574130)

I would expect for the CI to run the Python tests as instructed.

Actual behavior

When downloading ML models for testing, the runner encounters System.IO.IOException: No space left on device. I'm using the Standard GitHub hosted runners, and my repository is public, so according to the docs I should have ~150GB of disk space, just like on Windows. However, on Windows the runner works just fine.

I think the CI likely uses 18-20GB of disk space. Note that this started happening without changes to the code - all PRs & branches just started failing when the runner images updated.

Repro steps

Any CI triggered on the Sentence Transformers repository will trigger the failure.

[mikhailkoliada]

Hey! I guess we always guaranteed only 14GB of the space, there were some mistakes I the docs which are now fixed, screenshot taken from the link you provided. Sorry for the confusion.

[tomaarsen]

Oh, I see! That clears it up. I'll try to experiment with some solutions on my side, then.

• Tom Aarsen

[tomaarsen]

But you did reduce the disk size of the Ubuntu runners, right?

• Tom Aarsen

[t1m0thyj]

Hey! I guess we always guaranteed only 14GB of the space, there were some mistakes I the docs which are now fixed, screenshot taken from the link you provided. Sorry for the confusion.

@mikhailkoliada Thanks for clarification and updating the docs! Should this blog from a month ago be updated? Currently it is misleading because it says GitHub is offering "10 times the storage for their builds, for free".