1.9.0 manifests use LATEST tag for Kubeflow Notebooks!

[thesuperzapper]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

For some reason the Kubeflow Notebooks spawner_ui_config.yaml was not correctly synced from the kubeflow/kubeflow repo, resulting in the default spawner using the :latest tag, rather than the correct :1.9.0 tag.

https://github.com/kubeflow/manifests/blob/v1.9.0/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml

This is a CRITICAL issue which must be rectified by a 1.9.1 release almost immediately, as it will inevitably result in many users using unstable Notebook images without realizing it, as we use latest for all dev builds.

Steps to reproduce the issue

N/A

Put here any screenshots or videos (optional)

No response

[thesuperzapper]

@juliusvonkohout @kubeflow/release-managers @kubeflow/release-team this is a critical issue for 1.9.1, and its so serious that we should consider releasing a fix with only this.

The only other things which we probably need in that emergency 1.9.1 release would be fixing the regressions which make the Kubeflow Pipelines SDK authentication no longer work:

• "outside the cluster":
  • ISSUE: 400: Bad request; unsupported request method. Can't connect the Pipelines SDK to Kubeflow Pipelines with Kind for Kubeflow 1.9 #2844
• "inside the cluster"
  • ISSUE: [bug] Jwt issuer is not configured #2840
  • NOTE: this one seems to be related to the non-removal of AuthorizationPolicies like I found in RBAC: access denied on central dashboard #2832 (comment), so does not technically require a patch release, but rather better upgrade docs.

[juliusvonkohout]

Yes, this was discussed a few weeks ago in the release team meetings and we also have #2804

I also talked about it with @kimwnasptd and hope that he can fix it more easily since he has access to both repositories to adjust the synchronization scripts and structure.

[juliusvonkohout]

Please check #2853