CI: caching: add early termination & run check on schedule #2506
Conversation
Anton-Latukha
force-pushed
the
2021-12-18-keep-main-cache-by-schedule
branch
from
4b9f50e to
a7a2e89
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Yep. I became aware of the Windows caching problem & its cache somehow does not work properly. I think it is better to devote attention to it in #2507 |
Anton-Latukha
force-pushed
the
2021-12-18-keep-main-cache-by-schedule
branch
3 times, most recently
from
338b3ed to
779fc5b
Compare
Anton-Latukha
changed the title
|
Early termination is seen in: https://github.com/haskell/haskell-language-server/actions/runs/1598559261 Now removing the WIP PR hook. |
Anton-Latukha
force-pushed
the
2021-12-18-keep-main-cache-by-schedule
branch
from
779fc5b to
913a895
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Anton-Latukha
force-pushed
the
2021-12-18-keep-main-cache-by-schedule
branch
from
abb61bb to
f8312b4
Compare
Recently there is a lot of PRs/builds from internal repo branches. Repo cache has 10G pool limit & PR caches are not shared between PRs, but main branch cache is. Checking & when missing rebuilding it ensures that if internal branch PRs pushed main state out of the pool - it gets reinstituted. *afaik* there is no way to prioritize main branch cache better, since GitHub CI FIFO caches out of the pool by build time.
Anton-Latukha
force-pushed
the
2021-12-18-keep-main-cache-by-schedule
branch
from
f8312b4 to
bab8157
Compare
|
Yep. 3 checks a day may seem excessive. Until some hackathon 8), or community growth. Thank you 8) |
|
Restarted the checks to see caching at work. We shortened the CI cycle by 50 minutes: https://github.com/haskell/haskell-language-server/runs/4591646270?check_suite_focus=true 🌀 🥳 |
|
really great news, love to see again a windows build of 6 min |
|
Yes, >2 hours cycles were very painful. 1:15 is much tolerable (these cycles feel very tolerable to me), now the person knows that basically in an hour testing would be done. Would've been great to sneak favors from GitHub 😉. |
I am on it 😄 |
This PR allows to forget about cache pool exhaustion through internal branches.
Despite
cachingworkflow runs onmasteronly, GitHub caching is designed such that every workflow run that uses the cache(test, bench)can also save cache into the pool of own git repo. Which means the changes to the project configuration files inside PRs would produce caches for PR, which when there is enough internal branches PRs activity would pushmastercache out of the 10G FIFO pool because GitHub removes from the pool by the creation time. While PR caches that would stay - have a sharing & access scope limited to the PRs they are built for.Removal of the
mastercache out of the pool means that cache that was shared for all PRs - no longer there, which⇒ ∀PRs to build & save their own cache - which creates the pool exhaustion & congestion problem.For example: in #2502 https://github.com/haskell/haskell-language-server/runs/4571293708?check_suite_focus=true we see that master cache got pushed-out & see this situation.
afaik there is no way (documented one or more KISS/elegant one) to prioritize main branch cache better.