[ECR]: Pull through cache for authenticated registries

[srrengar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
ECR launched pull through cache recently which reliably and anonymously retrieves public images from ECR Public and Quay.io, caches them in a customer's private ECR registry, and keeps the images in sync. Customers also want the same functionality for other private registries that require authentication, and registries need accounts to have higher pull limits
https://aws.amazon.com/blogs/aws/announcing-pull-through-cache-repositories-for-amazon-elastic-container-registry/

Which service(s) is this request for?
ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
To cache images from private registries or images from public registries that require authentication for higher pull limits. Customers need to pass credentials through ECR pull through cache for ECR to access the images from private registries or to let ECR have access to higher pull limits.

Are you currently working around this issue?
Customers need to build tooling to log in to private registries, copy those images into ECR, and keep those images in sync with the upstream private registry, and maintain this entire system.

[benjimin]

Would this also enable anonymous access, proxied through the customer's VPC, and some configuration for how frequently image tags (such as latest) are polled for updates?

Maintaining our own ECR sync lambda system (driven supply-side rather than demand-side) has a tendency to counter-productively retrieve even more image versions than we actually need (e.g. skipped increments, alternate architectures).

[afirth]

Working around - dockerhub

1. adding image-pull-secret to every namespace and every service account as required. Usually after they break once, e.g. during a cluster upgrade when we have to redeploy all the managed nodes at once. Outages caused in the past by spot-termination-handler, aws-alb-ingress-controller, kube2iam (that was bad because downstream services didn't crashloop), and probably others. This exists, but ends up either patching all serviceAccounts including the AWS and K8s ones like kube-system/statefulset and kubesystem/aws or needing manual config for every service anyway.
2. Moving images to ECR. This is mostly ok except that
   1. we build with skaffold, and the ECR login system doesn't work with the docker-api, so need to override that to use docker-cli
   2. devs adding services from helm/upstream are typically unaware of the imagePullSecret requirement (or alternatively image registry, if separately exposed in the chart) << biggest problem
   3. dependabot setup is not so straightforward

2.ii. would be solved if EKS/ECR supported setting something like

{
  "registry-mirrors": ["https://mirror.gcr.io"]
}

ideally 1) without editing bootstrap data (just slap it in here ) and 2) without using gcr

maybe it does and I just haven't found the docs.

[joebowbeer]

Being able to configure our ECR to pull-through our private GitHub Container Registry (ghcr) would be super.

[mwos-sl]

One more use case we have is several deployments across different aws regions. We need a pull through cache in each each region for images produced by CI into one of the regions, for services like ECS and EKS. Why not ECR replication? Costs. We don't need all of the images in each region to be replicated, only used one.

-> #2208

[wosiu]

Excited to see "Work in Progress" label!
@coultn is there any ETA for this to be GA? Asking because we've got very hot discussion about alternatives in our company. ETA could help to judge whether we should wait for a native aws solution or invest in developing some workaround in-house.

[mwos-sl]

Bumping question if there's maybe any ETA for pull through caching of private registries hosted on AWS?

[blakepettersson]

Now ACR has support for pull-through caches (Azure/acr#599), at least for Dockerhub (which still is a whole load more images than ECR pull-through supports). Given the rate things are going it's likely that ACR will build out its support for other registries way before ECR...

[jwenz723]

I would like to be able to configure a pull through cache rule in my private ECR registry in one of my AWS accounts to pull images from a private ECR registry in another one of my AWS accounts.

It would be nice if this allowed cross region pulling as well.

My goal is to have a single registry where all my CI workflows push images to, lets call it build-us-west-2. Then in each of my accounts where I deploy the image I want to store a local copy of the image, lets call these accounts (the AWS region for each account is stated in the name) dev-us-west-2, dev-us-east-1, prod-us-west-2, and prod-us-east-1.

I would like to configure a pull through cache rule for each of the accounts where the image is deployed like this:

dev-us-west-2 -> build-us-west-2
dev-us-east-1 -> build-us-west-2
prod-us-west-2 -> build-us-west-2
prod-us-east-1 -> build-us-west-2

[seumassoltysik]

Hard to believe this is not an existing capability. We use artifactory on the corporate build level but it would be great to mirror that docker registry with ECR running in account in which we have our Jenkins and EKS infra running.

[tigercxx]

Just tried pull through repository for a private docker repository in GAR, and it worked smoothly. Surprised to see that it's not supported in ECR yet.

[rfulwell]

@rnene100 I see you added the Coming Soon label. Can you provide any more details?

[rnene100]

Amazon Elastic Container Registry (ECR) now includes Docker Hub, Azure Container Registry, and GitHub Container Registry as supported upstream registries for ECR’s pull through cache feature.
See announcement- https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ecr-pull-through-cache-additional-upstream-registries/

[nxf5025]

Amazon Elastic Container Registry (ECR) now includes Docker Hub, Azure Container Registry, and GitHub Container Registry as supported upstream registries for ECR’s pull through cache feature. See announcement- https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ecr-pull-through-cache-additional-upstream-registries/

Are there plans to support other private registries such as Artifactory? I also noticed this in the doc:

AWS Lambda doesn't support pulling container images from Amazon ECR using a pull through cache rule.

That's our main use case so wondering if that will change in the future.

[Sodki]

Amazon Elastic Container Registry (ECR) now includes Docker Hub, Azure Container Registry, and GitHub Container Registry as supported upstream registries for ECR’s pull through cache feature.

I'm confused about this decision. Why couldn't it be a generic solution since the vast majority of Docker registries work in the same way and authenticate in the same way? We're on GitLab, so we can't take advantage of this.

[wosiu]

@rnene100 does the new functionality allows ECR pull-though cache for ECR in another region?
So basically these 2 use-cases mentioned in this thread:

1. [ECR]: Pull through cache for authenticated registries  #1584 (comment)
2. [ECR]: Pull through cache for authenticated registries  #1584 (comment)

After reading the announcement it sounds like it is still not supported :(

[rnene100]

@wosiu ECR to ECR pull through cache is not part of this but something we are actively considering. I don't have a timeline on that. Thanks for checking.

@Sodki Our goal is to continue to add support for additional upstream registries that authenticate in the same way (will add GitLab to the list).

@nxf5025 AWS Lambda is looking at adding this support. I don't have timelines on that but I have reached out to the team. Thanks for checking

[rnene100]

Support for authenticated upstream with pull through cache has been launched. We will incrementally add more authenticated upstream registries. Resolving issue.

[wosiu]

@rnene100 I don't this this ticket should be marked as closed. Many people (including me) have been voting for this one because of ECR to ECR pull-through. Now we need to start from 0 when collecting votes, which means tens of months before it gets a critical mass and start being considered by AWS.

[mmerickel]

Is there a different open issue for tracking that? I’m here for the ecr to ecr as well as ecr from self-hosted-gitlab-in-private-vpc features. But since the latter feels very far off - the ecr to ecr is really important for folks doing regional deployments on standard aws offerings.

[rnene100]

@mmerickel Please open a separate issue for ECR to ECR pull through cache so that it's easier to track. As mentioned earlier, we are actively considering that feature. Thank you!

[wosiu]

Opened: #2208
Anyone here, please vote for this one

fyi @mmerickel

[karaluh]

@rnene100 Any update on Lambda support for this?

[karaluh]

And also on ECS support? ATM cache rules cannot be used directly there, because ECS requires the ECR repository to exist when running tasks, so manual docker pull command is required beforehand, so the repository is initialized first.

[thejosephstevens]

@rnene100 , I'm seeing that Quay.io is not among the providers with support for auth, is it best to open a separate Github Issue for that? Along the lines of Sodki's comment, it's not clear why there isn't a generic provider for Docker-compatible registries supporting basic auth (that's basically what ACR and GCP Artifact Registry did). ACR allows you to override the hostname of registries, so you could coerce an unsupported registry to fit, and Artifact Registry just gives you a full Custom option where you provide the hostname and auth. Since all the ECR options lock the hostname config I'm stuck having to move to a different registry in order to do auth'd caching.