Restore pre 22.3 wheel cache pathing #11538
Conversation
|
Please feel free to edit/update/rewrite this PR as necessary. |
|
I believe these test failures are caused by an update to git to fix a security issue and are not related to my change. I don't see an issue or PR for this yet. Does anyone know if this is being addressed or is this something that needs to be done? FWIW my understanding of the issue is that you are only exposed if you don't trust the git repo that is being added as a submodule. If you do trust it you can do something like: It looks like pip's tests may build the git repo from scratch in which case it would be trusted. But I'm not familiar enough with pip's test suite to be confident of that. |
|
I think you're right in your analysis. Would you be interested in creating a PR to fix the test? @sbidoul do you know why we have a dedicated test for submodules? It feels like we're going into too much detail on "git stuff" - the test feels like it's simply checking that git works the way it should, not testing any pip specific functionality... |
|
@pfmoore actually, pip does have dedicated code to handle git submodules and I think that test is to exercise that part. |
src/pip/_internal/models/link.py
Outdated
| for hash_name, hash_value in hashes.items(): | ||
| if hash_name in _SUPPORTED_HASHES: | ||
| link_hash = LinkHash(name=hash_name, value=hash_value) | ||
| break |
There was a problem hiding this comment.
Perhaps we should get the first hash found by iterating _SUPPORTED_HASHES so as to get a deterministic result that does not depend on the order of hashes provided by the index?
There was a problem hiding this comment.
That makes sense. If the index starts to support newer hashes we'll end up orphaning cache entries, but we'll only do so when the hashes become stronger (due to the order of _SUPPORTED_HASHES entries ) and not randomly due to index changes.
For updates like this do you prefer a follow on commit or a squash update and force push?
There was a problem hiding this comment.
I tend to find squashes cleaner (you can click force-pushed to see what changed), but I don't think we have any strong enforced preferences in the code base.
|
Thanks for this PR, @cboylan. I'm not super familiar with pip's hash checking code but, echoing what you said in #11527 (comment), I'm a bit worried that this bug may have affected something worse than the cache and that our test suite did not catch it. |
cboylan
force-pushed
the
stable-wheel-cache
branch
from
4ea36e6 to
f3747bf
Compare
Prior to bad03ef wheel cache paths incorporated source material hashes in their paths. This commit which ended up in 22.3 stopped including that information. This is problematic for two reasons. First our cache is no longer encoding data integrity information that was previously intentionally included. Second it means anyone upgrading from < 22.3 to 22.3 will have orphaned wheel cache entries. The fix here is to update the Link object to set Link.link_hash in the Link.from_json method. Otherwise the hash information is simply missing. This will cause anyone upgrading from 22.3 to newer to have orphaned wheels, but that seems worthwile considering 22.3 hasn't been around as long as the previous implementation and we get stronger data integrity controls out of it. This fixes pypa#11527
cboylan
force-pushed
the
stable-wheel-cache
branch
from
f3747bf to
a08a278
Compare
|
@pfmoore it would be good to include this in the bugfix release as well. Any concerns (other than the news fragment)? |
|
As long as someone reviews, approves and commits it before the weekend, I'm fine with it. But I note @sbidoul's comment above:
Are we sure that the fix here is correct? I do not want to merge this, release 22.3.1, and then face calls that we need a 22.3.2 because there's a bigger problem here... I will note that we've had basically nobody else complaining that #11527 is a significant issue, nor have we had any other bug reports that I'm aware of that are related to this. So I don't consider this a serious issue that needs a bugfix release in its own right. So personally, I'd rather we ensure that we get it right rather that rush a fix just to get into 22.3.1. If people with more expertise in the cache and hash checking mode than I have are comfortable it's OK, though, I'm happy to accept their word on the matter. |
|
#11557 was reported as a hash checking regression. But I've not investigated myself. I don't have much time at hand at the moment, unfortunately. |
For whatever it is worth, addressing this isn't urgent for me either. I do think this is an improvement and addresses a regression, but I'm happy to take the time to make sure it is correct as well. |
This was suggested in code review. Co-authored-by: Stéphane Bidoul <stephane.bidoul@acsone.eu>
This came up in code review. It was pointed out that the news entry appears in the changelog so needs to be written for that audience. Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
|
Fixes #11692 as well! |
|
This one would be superseded by #11696, but we probably want keep the new test from this PR. |
Prior to bad03ef wheel cache paths incorporated source material hashes in their paths. This commit which ended up in 22.3 stopped including that information. This is problematic for two reasons. First our cache is no longer encoding data integrity information that was previously intentionally included. Second it means anyone upgrading from < 22.3 to 22.3 will have orphaned wheel cache entries.
The fix here is to update the Link object to set Link.link_hash in the Link.from_json method. Otherwise the hash information is simply missing.
This will cause anyone upgrading from 22.3 to newer to have orphaned wheels, but that seems worthwhile considering 22.3 hasn't been around as long as the previous implementation and we get stronger data integrity controls out of it.
This fixes #11527