feature request: container URIs that require auth #564
Comments
|
It looks like this: docker: nvcr.io/nvidia/hpc-benchmarks
latest:
'crane ls nvcr.io/nvidia/hpc-benchmarks: unsupported status code 401; body: <html>': 'crane
digest nvcr.io/nvidia/hpc-benchmarks:crane ls nvcr.io/nvidia/hpc-benchmarks: unsupported
status code 401; body: <html>: parsing reference "nvcr.io/nvidia/hpc-benchmarks:crane
ls nvcr.io/nvidia/hpc-benchmarks: unsupported status code 401; body: <html>":
could not parse reference'
tags:
20.10-hpl: "crane digest nvcr.io/nvidia/hpc-benchmarks:20.10-hpl: unsupported status\
\ code 401; body: <html>\r\n<head><title>401 Authorization Required</title></head>\r\
\n<body bgcolor=\"white\">\r\n<center><h1>401 Authorization Required</h1></center>\r\
\n<hr><center>nginx/1.14.2</center>\r\n</body>\r\n</html>\r\n"
'crane ls nvcr.io/nvidia/hpc-benchmarks: unsupported status code 401; body: <html>': 'crane
digest nvcr.io/nvidia/hpc-benchmarks:crane ls nvcr.io/nvidia/hpc-benchmarks: unsupported
status code 401; body: <html>: parsing reference "nvcr.io/nvidia/hpc-benchmarks:crane
ls nvcr.io/nvidia/hpc-benchmarks: unsupported status code 401; body: <html>":
could not parse reference'
<body bgcolor="white">: 'crane digest nvcr.io/nvidia/hpc-benchmarks:<body bgcolor="white">:
parsing reference "nvcr.io/nvidia/hpc-benchmarks:<body bgcolor=\"white\">": could
not parse reference'
maintainer: '@vsoch'
url: https://ngc.nvidia.com/catalog/containers/nvidia:hpc-benchmarks/tags
description: 'The NVIDIA HPC-Benchmarks collection provides three NVIDIA accelerated
HPC benchmarks: HPL-NVIDIA, HPL-AI-NVIDIA, and HPCG-NVIDIA.' |
|
I don't know if they have "locked down" their registry, but this doesn't work for me: $ docker pull nvcr.io/nvidia/hpc-benchmarks
Using default tag: latest
Error response from daemon: unauthorized: authentication requiredJust to double check I found a known tag on https://catalog.ngc.nvidia.com/orgs/nvidia/containers/hpc-benchmarks $ docker pull nvcr.io/nvidia/hpc-benchmarks:21.4-hpl
Error response from daemon: unauthorized: authentication requiredSo it seems they have locked down this image, which is unfortunate - I don't think we can provide a community recipe that is not available for that. In terms of the recipe generated above, that indeed is a bug - we have some basic parsing for error states (it doesn't return a proper error code) but haven't seen this one yet - I can add a fix tonight. For your use case, if you want to add this container module, since it requires auth you'll likely need to generate the recipe file, log i to their cloud, and then copy pasta the hashes. If you have another idea please share to talk about - it's pretty lame they do that, imho. |
|
We have a similar problem in easybuild. In our case, the solution lies in the search order for the source tarball. It looks first in the same directory as the software recipe file, then in the easybuild sources directory, and only then it tries to fetch the file. In case of things like cuDNN from nvidia, we download it manually and out in one of those directories, as there’s no good universal solution and it’s a tiny number of recipes which suffer from this. |
|
This is an example of such a problem: https://github.com/easybuilders/easybuild-easyconfigs/blob/766d895a61376c86c99c2f3e3bd8e53a3caf8bcf/easybuild/easyconfigs/t/TensorRT/TensorRT-4.0.1.6-fosscuda-2018b-Python-2.7.15.eb#L10 usually the url would be there |
|
What do you means source tarball in this context? |
|
The file(s) that need to be downloaded. In their case, it’s the source code, as easybuild compiles and installs software from source. On your case, it would be the container file(s). Meaning that one should download the container by hand first and leave it somewhere so shpc finds it for installation. It’s far from ideal, of course. |
|
So - the point of an OCI registry is to provide these .tar.gz blobs that eventually are extracted into the container. The request to get a manifest or config is the first step to get the hashes of those files. This OCI registry provided by NVIDIA has essentially put a paywall in that auth is required, so we can’t do that. There is no other place you can just find those files. Shpc is also not a package manager - we “install” from registries. So there is no means to “download by hand” - that’s what the registry is for. |
|
The way we could get this to work is to require the user to provide auth. I’m just not sure I want to maintain recipes that require it! |
vsoch
changed the title
|
Updated title to be about this feature - please further comment future readers if this is something you'd like (and if so, what the ideal interaction would be). |
https://github.com/singularityhub/singularity-hpc/blob/main/registry/nvcr.io/nvidia/hpc-benchmarks/container.yaml
It seems like there was an error during the automated creation of this file, or am I misinterpreting the
latestandtags?The text was updated successfully, but these errors were encountered: