Skip to content

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images #2318

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
42 tasks done
spiffxp opened this issue Jul 9, 2021 · 9 comments
Closed
42 tasks done

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images #2318

spiffxp opened this issue Jul 9, 2021 · 9 comments
Assignees
@spiffxp
Labels
area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/release-eng Issues or PRs related to the Release Engineering subproject kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Milestone
v1.22

Comments

@spiffxp
Copy link
Member

spiffxp commented Jul 9, 2021 •
edited
Loading

Part of umbrella issue to migrate the kubernetes project away from use of GCP project google-containers: #1571

This issue covers the deprecation of and migration away from two google.com assets:

  • the google.com-owned GCS bucket gs://kubernetes-release-dev living in GCP project google-containers, in favor of the community-owned GCS bucket gs://k8s-release-dev living in GCP project k8s-release
  • the google.com-owned GCR repo gcr.io/kubernetes-ci-images, in favor of the community-owned GCR repo gcr.io/k8s-staging-ci-images

Step 1: Set and announce deprecation window

The first part is setting and announcing a deprecation window for CI artifacts placed in gs://kubernetes-release-dev, and their companion images in gcr.io/kubernetes-ci-images e.g.

  • v1.17 was the first release that had CI builds of kubernetes land in gs://k8s-release-dev/ci and gs://kubernetes-release-dev/ci simultaneously
  • v1.18 was the first release that cluster-api and kubeadm started defaulting to gcr.io/k8s-staging-ci-images instead of gcr.io/kubernetes-ci-images
  • v1.20 was the last release that had CI builds land in gs://kubernetes-release-dev/ci-cross
  • v1.22 is the last release to have CI builds of kubernetes land in gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images
  • we will stop using gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images completely once older CI jobs have aged out according to the kubernetes version support policy, or when we see no remaining traffic to gs://kubernetes-release-dev for a period of N weeks

Step 2: Setup community-owned alternatives, and use them for release-blocking / merge-blocking CI

This is a rough breakdown of the work necessary to complete the heavy lifting before we can shard out work to all repos across the project

Step 3: Ensure no references to kubernetes-release-dev within project

The next part of this is ensuring there are no remaining references to kubernetes-release-dev outside of vendor/ directories (except those necessary for the ci-kubernetes-build-.*deprecated.* jobs to satisfy the deprecation window)

  • Migrate documentation to refer to k8s-release-dev instead of kubernetes-release-dev
  • Migrate any remaining scripts / tooling across the project to use k8s-release-dev

This is pretty easily verifiable (with human inspection of results) using https://cs.k8s.io

$ curl -s -X POST   \
  -F "q=kubernetes-release-dev" \
  -F "repos=*" \
  -F "excludeFiles=vendor/"  \
  "https://cs.k8s.io/api/v1/search" \
  | jq -r '.Results | keys | .[]' \
  | sort \
  | sed -e 's/^/- [ ] /g'

Step 4: Ensure no references to kubernetes-ci-images within project

  • Migrate documentation to refer to k8s-staging-ci-images instead of kubernetes-ci-images
  • Migrate any remaining scripts / tooling across the project to use k8s-staging-ci-images
$ curl -s -X POST   \
>   -F "q=kubernetes-ci-images" \
>   -F "repos=*" \
>   -F "excludeFiles=vendor/"  \
>   "https://cs.k8s.io/api/v1/search" \
>   | jq -r '.Results | keys | .[]' \
>   | sort \
>   | sed -e 's/^/- [ ] /g'

Step 5: Complete deprecation

Related issues

/wg k8s-infra

/sig testing
/area prow
FYI @kubernetes/sig-testing-leads

/sig release
/area artifacts
/area release-eng
FYI @kubernetes/release-engineering

/kind cleanup
/kind deprecation
/priority important-soon
/milestone v1.22

/assign
@k8s-ci-robot k8s-ci-robot added wg/k8s-infra sig/testing Categorizes an issue or PR as relevant to SIG Testing. area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters labels Jul 9, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Jul 9, 2021
@k8s-ci-robot k8s-ci-robot added sig/release Categorizes an issue or PR as relevant to SIG Release. area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/release-eng Issues or PRs related to the Release Engineering subproject kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jul 9, 2021
@spiffxp spiffxp changed the title Migrate kubernetes project away from gs://kubernetes-release-dev to gs://k8s-release-dev Migrate kubernetes project away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images Jul 9, 2021
@spiffxp spiffxp mentioned this issue Jul 9, 2021
Closed
6 tasks
@spiffxp spiffxp changed the title Migrate kubernetes project away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images Jul 9, 2021
This was referenced Jul 9, 2021
Merged
Closed
Merged
Merged
This was referenced Jul 12, 2021
Closed
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 12, 2021 •
edited
Loading

Timeline:

This was referenced Jul 12, 2021
Merged
Merged
Merged
Closed
Merged
Merged
Merged
Merged
Merged
This was referenced Jul 17, 2021
Merged
Merged
Merged
Merged
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 17, 2021

Went on a bit of a spree today and opened the following:

Where possible I tried to converge toward dl.k8.io instead of a simple s/kubernetes-release-dev/k8s-release-dev, since this is somewhat of a dry-run for the work to migrate gs://kubernetes-release

@spiffxp spiffxp mentioned this issue Jul 19, 2021
Merged
spiffxp added a commit to spiffxp/cloud-provider-gcp that referenced this issue Jul 19, 2021
@spiffxp
use dl.k8s.io where possible
cf2335c 
The only time a kubernetes GCS bucket name should be showing up in a
hardcoded URI is if gsutil is being used (e.g. gsutil cp gs://foo/bar .)

Otherwise, for tools like curl or wget, dl.k8s.io is much nicer for us
as a project, since we can transparently change where that redirects to
without having to change code everywhere

These changes will mean very few changes will be necessary to
accommodate a gs://kubernetes-release -> gs://k8s-release migration
equivalent of the CI migration we're going through right now

These changes also address the gs://kubernetes-release-dev references
currently used by this repo (ref:
kubernetes/k8s.io#2318)
@spiffxp spiffxp mentioned this issue Jul 19, 2021
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 22, 2021

Two more PRs opened / merged:

Need to do another pass and make sure documentation is wrapped up

@CecileRobertMichon CecileRobertMichon mentioned this issue Jul 22, 2021
Closed
@spiffxp spiffxp mentioned this issue Jul 23, 2021
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 23, 2021

I missed that the release-branch cut caused v1.23 builds to start getting pushed:

This was referenced Jul 26, 2021
Closed
Closed
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 29, 2021

This should take care of the last of the non-necessary/historically-archived kubernetes-relesase-dev references:

This was referenced Jul 29, 2021
Merged
Merged
Merged
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Aug 4, 2021 •
edited
Loading

kubernetes-release-dev traffic (sent bytes)
Screen Shot 2021-08-04 at 9 13 24 AM

@spiffxp
Copy link
Member Author

spiffxp commented Aug 4, 2021

kubernetes-ci-images traffic (sent bytes)
Screen Shot 2021-08-04 at 9 15 21 AM

@spiffxp
Copy link
Member Author

spiffxp commented Aug 5, 2021

/close
Announcement: https://groups.google.com/g/kubernetes-dev/c/58lP-CIP2zg/m/dS8TTmM4BwAJ

@k8s-ci-robot
Copy link
Contributor

@spiffxp: Closing this issue.

In response to this:

/close
Announcement: https://groups.google.com/g/kubernetes-dev/c/58lP-CIP2zg/m/dS8TTmM4BwAJ

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@spiffxp spiffxp mentioned this issue Aug 5, 2021
Merged
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@spiffxp spiffxp

Labels
area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/release-eng Issues or PRs related to the Release Engineering subproject kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
None yet
Milestone
v1.22
Development

No branches or pull requests
2 participants
@spiffxp @k8s-ci-robot