ubuntu-latest runner disk space reduced as of roughly February 5th

[tomaarsen]

Hello!

As of February 5th, I started noticing that my ubuntu-latest tests would start failing due to System.IO.IOException: No space left on device, while my windows-latest tests still consistently pass.

See here the first test failure with that error. I'm unsure how much space my CI tests actually use up, but I suspect that it's between 18 and 20GB. According to the GitHub docs here, the ubuntu-runner should have 150GB of space. I recognize that this is reduced by all of the installed tools, but one way or another, something changed on ~February 5th that seemingly caused the runners to have less space available.

Could it be that I am accidentally being given runners for private repositories? Those have 14GB of space, which would indeed not be enough.

It seems like there was indeed a release 4 days ago, which may have caused this discrepancy. I would love some help to get my CI to pass again!

• Tom Aarsen

[apeltzer]

Similar issue here: https://github.com/nf-core/smrnaseq/actions/runs/7904801386/job/21575967539?pr=316#step:4:429

[jasmezz]

Or here: https://github.com/nf-core/funcscan/actions/runs/7830752238/job/21366288111#step:4:128

[tomaarsen]

I've opened a related issue here: #9344

The origin of this regression seems to be this PR: https://github.com/actions/runner-images/pull/9251/files, which has reduced the free space from 31GB+ to 17GB+ for ubuntu runners.

cc @apeltzer @jasmezz

• Tom Aarsen

[apeltzer]

I also did some tests, @tomaarsen and added a custom action that "cleans up" the action runner first by reducing the packages installed, etc pp. See here:

                                                          Filesystem      Size  Used Avail Use% Mounted on
Before doing anything, no cleanup performed:              /dev/root        73G   54G   20G  74% /                                                                           
Before running the pipeline, after initial cleanup:       /dev/root        73G   30G   43G  41% /
After running the pipeline, before final cleanup:         /dev/root        73G   48G   25G  66% /

As you can see, without the cleanup, there is ~20GB of free space on the runner, in our case during pipeline excution this gets clogged up and the pipeline fails. If I run the custom runner jlumbroso/free-disk-space@v1.3.1 to cleanup, its running well (2nd case, made it so that 43GB are available).

[tomaarsen]

Awesome! I tried something similar using easimon/maximize-build-space@master, but did not get any luck there. Thanks for sharing!

[mikhailkoliada]

Answered to the issue filed :) We indeed guarantee only 14GB of free space for both private & public repos