Cache NUMBA kernels between CI runs

[sjperkins]

Closes #278

[JSKenyon]

This looks awesome! I will probably move this to the dev branch before merging.

[sjperkins]

This looks awesome! I will probably move this to the dev branch before merging.

Cool. Need to prod it a bit to see if it works.

[sjperkins]

Constructing the key out of the date may be overkill. I suspect we could just use numba-cache and it would propagate and be updated between runs.

I guess the downside is that it might accumulate a bunch of crufty old kernels. Note AFAICT there's a 10GB cache limit per repo and cache entries expire weekly so it may not be a big deal.

@bennahugo suggested we add the numba version to the cache key. I wonder if numba is clever enough to trigger recompiles on new numba versions.

The python version may also be relevant given a codex __pycache__ dir looks as follows

__init__.cpython-310.pyc
__init__.cpython-39.pyc
bda_avg.cpython-310.pyc
bda_avg.cpython-39.pyc
bda_avg.row_average-23.py310.1.nbc
bda_avg.row_average-23.py310.2.nbc
bda_avg.row_average-23.py310.nbi
bda_avg.row_average-23.py39.1.nbc
bda_avg.row_average-23.py39.2.nbc
bda_avg.row_average-23.py39.nbi
bda_avg.row_chan_average-313.py310.1.nbc
bda_avg.row_chan_average-313.py310.2.nbc
bda_avg.row_chan_average-313.py310.3.nbc
bda_avg.row_chan_average-313.py310.4.nbc
bda_avg.row_chan_average-313.py310.5.nbc
bda_avg.row_chan_average-313.py310.nbi
bda_avg.row_chan_average-313.py39.1.nbc
bda_avg.row_chan_average-313.py39.2.nbc
bda_avg.row_chan_average-313.py39.3.nbc
bda_avg.row_chan_average-313.py39.4.nbc
bda_avg.row_chan_average-313.py39.nbi
bda_mapping.bda_mapper-341.py310.1.nbc
bda_mapping.bda_mapper-341.py310.nbi
bda_mapping.bda_mapper-341.py39.1.nbc
bda_mapping.bda_mapper-341.py39.2.nbc
bda_mapping.bda_mapper-341.py39.nbi
bda_mapping.cpython-310.pyc
bda_mapping.cpython-39.pyc

[sjperkins]

Hmmm, I was fairly sure I mkdir'd that

[sjperkins]

So the kernel caching does not seem to be improving the test suite run time. even though kernel caches are created: https://github.com/ratt-ru/QuartiCal/actions/caches. This would also seem to suggest NUMBA_CACHE_DIR is respected.

[JSKenyon]

So the kernel caching does not seem to be improving the test suite run time. even though kernel caches are created: https://github.com/ratt-ru/QuartiCal/actions/caches. This would also seem to suggest NUMBA_CACHE_DIR is respected.

Is respected, or isn't? Might need to rerun the tests a few times - I have muddied the waters by merging in main. I do think that there is probably something which can be done - will take a closer look at the end of the week.

[sjperkins]

So the kernel caching does not seem to be improving the test suite run time. even though kernel caches are created: https://github.com/ratt-ru/QuartiCal/actions/caches. This would also seem to suggest NUMBA_CACHE_DIR is respected.

Is respected, or isn't?

I think it is respected -- The caches are about 11MB.

Another thought occurred, the cached kernel modification times are probably earlier than the checked out python code -- this might trigger recompilation: https://numba.readthedocs.io/en/stable/developer/caching.html

The cache is invalidated when the corresponding source file is modified.

Edit: Referenced the main article on caching, rather than the cuda article.

[sjperkins]

Another thought occurred, the cached kernel modification times are probably earlier than the checked out python code -- this might trigger recompilation: https://numba.readthedocs.io/en/stable/developer/caching.html

The cache is invalidated when the corresponding source file is modified.

Unfortunately it looks like it is the case that the timestamp is only the input to the cache key (at least as of Aug 22): https://numba.discourse.group/t/cache-behaviour/1520

I would like to propose to move away from invalidating the cache index based on the timestamp of the file, and use only the code+closure signature of the function itself. Would anyone see a problem with that? When I say code+closure signature I mean the exact same information that is used to select the overload from within the cache.

So this approach doesn't seem viable.

[JSKenyon]

So this approach doesn't seem viable.

Ah unfortunate. Perhaps there will be progress upstream at some point.