pip_requirements with private index and hashes raises hashin.PackageNotFoundError

[HarrySky]

How are you running Renovate?

Self-hosted

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

renovate-runner 3.14.3

Describe the bug

Hi, thank you for this great tool 😄

I was using Renovate for updating pip requirements with private index and recently added hashes to them.
When it comes to versions - it checks and updates them properly, but it seems that hashin ignores registryUrls completely and it results in "Artifact update problem".

I put relevant Python traceback below.

PR that implemented hashes feature: #6460

To reproduce you need following renovate.json file:

{
  "extends": [
    "config:base"
  ],
  "packageRules": [
    {
      "groupName": "software",
      "matchManagers": ["pip_requirements"],
      "matchPaths": ["requirements.txt"],
      "registryUrls": ["https://test.pypi.org"],
      "schedule": ["before 5pm"]
    }
  ]
}

With following requirements.txt file (test package does not exist on pypi.org):

test==1.07 \
    --hash=sha256:322d9e33339f2124a9dc7826c198d3c57cd24b58295fb7146f9f464b25017889

Relevant debug logs

Logs

Traceback (most recent call last):
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 89, in _download
    r = urlopen(url)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/python/3.9.7/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python/3.9.7/bin/hashin", line 8, in <module>
    sys.exit(main())
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 836, in main
    return run(
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 135, in run
    return run_packages(specs, requirements_file, *args, **kwargs)
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 190, in run_packages
    data = get_package_hashes(
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 680, in get_package_hashes
    data = get_package_data(package, index_url, verbose)
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 608, in get_package_data
    content = json.loads(_download(url))
  File "/usr/local/python/3.9.7/lib/python3.9/site-packages/hashin.py", line 93, in _download
    raise PackageNotFoundError(url)
hashin.PackageNotFoundError: https://pypi.org/pypi/my-private-package/json

Have you created a minimal reproduction repository?

No reproduction repository

[rarkins]

Does the private registry need authentication and if so, did you configure that using hostRules?

[HarrySky]

@rarkins yes, it needs authentication, but I just add auth in registryUrls schema: "https://user:pass@my.private.registry/simple"

[rarkins]

registryURls are not passed to hashin. We first need to understand how to configure hashin to support private modules in general. Have you been able to locate any docs? e.g. it's presumably the same as how you authenticate when you run hashin directly

[HarrySky]

It seems that hashin does not have docs, but according to hashin --help you can use --index-url flag (peterbe/hashin#107). Seems that you can run hashin --index-url https://PRIVATE_INDEX -r requirements.txt

[rarkins]

Couple of challenges:

• We allow more than one registryUrls entry, but I guess in this case we'll need to pick the first one
• If will work if someone embeds the credentials in the URL like you've done, but others might use hostRules and expect it to work also

[andreadelfante]

Hi, any news about this issue?