Unable to work with HelmRelease with v2.2.1

[hanpeter]

Describe the bug

Starting with v2.2.0 and continuing in v2.2.1, I am not able to interact with HelmReleases.

Steps to reproduce

$ flux get hr -A
✗ failed to get API group resources: unable to retrieve the complete list of server APIs: helm.toolkit.fluxcd.io/v2beta2: the server could not find the requested resource

Expected behavior

Same behaviour as v2.1.2

$ flux get hr -A
NAMESPACE	NAME           REVISION	SUSPENDED	READY	MESSAGE
foo     	foo         	1.2.3  	False    	True 	Release reconciliation succeeded
bar     	bar           	2.3.4 	False    	True 	Release reconciliation succeeded

Screenshots and recordings

No response

OS / Distro

macOS 14.1.2

Flux version

v2.2.1

Flux check

► checking prerequisites
✗ Kubernetes version v1.24.17-eks-8cb36c9 does not match >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.0.1
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.35.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.0.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.0.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.0.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✗ check failed

Git provider

No response

Container Registry provider

No response

Additional context

This is similar to Issue 4474.

Code of Conduct

• I agree to follow this project's Code of Conduct

[kingdonb]

The HelmRelease API Version changed in 2.2, it looks like there is a mismatch between your Flux version (cli) and your flux version (distribution in the cluster):

Flux version
v2.2.1

Flux check
► checking prerequisites
✗ Kubernetes version v1.24.17-eks-8cb36c9 does not match >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.0.1

Did you upgrade Flux on the cluster with flux bootstrap? The version of your CLI must match the version of the Flux distro installed into the cluster. Please check out the release notes for v2.1 and v2.2 for any of the breaking changes you may need to be aware of in this release.

[hanpeter]

So does that mean CLI v2.2 is a backward incompatible version from v2.0 or v2.1?

[stefanprodan]

So does that mean CLI v2.2 is a backward incompatible version from v2.0 or v2.1?

Yes, the CLI ensures backwards compatibility only for GA APIs, for beta versions you need a CLI that matches the cluster version.

[0xStarcat]

how does one go about re-installing the v2.1.2 cli?

brew install fluxcd/tap/flux@2.1.2 produces:

No available formula or cask with the name "fluxcd/tap/flux@2.1.2". Did you mean fluxcd/tap/flux or fluxcd/tap/flux@0.41?

[hanpeter]

@0xStarcat, you will have to do brew uninstall and install v2.1.2 manually using the official bash script.

[hanpeter]

@stefanprodan, by "GA APIs", do you mean helm.toolkit.fluxcd.io/v1? How does one go about using that API version with Flux v2? Am I to do a find & replace the output of the flux bootstrap command? Does that even work with Flux v2? I cannot find any mention of the v1 in this repository. Did that version even exist?

I am, by no means, saying it is impossible to work around this issue. I have downloaded v2.1.2 CLI, and that version works. I am, however, trying to point out that this is a poor user experience. I updated a CLI, trusting the semantic versioning, and now I am faced with a backward incompatible issue, which is exactly the problem that the semantic versioning is trying to fix.

[hiddeco]

Flux v1 was https://github.com/fluxcd/flux, before Flux v2 we used to have a v0.x range in this repository.

We actively advertise instability for a range of commands in their --help section if breaking changes may occur because the APIs they build on have not reached a stable version yet.

$ flux get helmrelease --help
The get helmreleases command prints the statuses of the resources.

⚠️  Please note that this command is in preview and under development.
While we try our best to not introduce breaking changes, they may occur when
we adapt to new features and/or find better ways to facilitate what it does.

...

[jack1902]

Going to chime in here about the "user-experience"

If i am upgrading Flux across multiple environments say from version 2.1.0 to 2.2.0, i am now expected to have multiple versions of the CLI installed so that i can interact with different environments. This is tedious, as rolling out across environments or checking in on those environments requires me to have multiple differing versions.

To add to this, kubectl works with older versions of kubernetes such that i can have one environment on 1.27 and another on 1.26 whilst my cli is on 1.27. This enables me to quickly switch between clusters without having to use different binaries.

I would not expect 2.2.0 of the CLI to not function against a 2.1.0 cluster as Semver dictates that a breaking change is a major version. I understand that v2beta2 etc are CRD deployed to the cluster, but i would not expect that 2.2.0 no longer works with a 2.1.0 flux cluster.

[mujak27]

Encountered the same issue with Flux 2.0.0, but successfully resolved it by upgrading to 2.2.3. The latest patch addresses the Helm breaking change and I think we can now consider closing this issue and related ones.

[Mo0rBy]

I've just encountered this issue.
I am the latest version:

❯ flux --version
flux version 2.2.3

But I get an error when attempting to get HelmReleases:

❯ flux get hr -A
✗ failed to get API group resources: unable to retrieve the complete list of server APIs: helm.toolkit.fluxcd.io/v2beta2: the server could not find the requested resource

Did a brew uninstall flux and installed using the bash script to downgrade to 2.1.2.

I want to upgrade the Flux version in my cluster, is there anything I need to worry about when doing this? Other than blocking other members of my team from interacting with Flux until they update their CLI version.

[souleb]

@Mo0rBy this is usually a connection issue. You should check you kubeconfig and make sure that you have access to your cluster.

[Mo0rBy]

@souleb I double checked the context I was using and it was the correct one. As I said, I downgraded my Flux CLI version to 2.1.2 and then my command worked as expected.

[stefanprodan]

Flux v2.2 comes with HelmRelease API v2beta2, you need upgrade your clusters to be able to use v2.2 CLI.

[Mo0rBy]

@stefanprodan Is that just running the bootstrap command to upgrade the version used in the cluster?
And will that automatically update the apiVersions defined on all my HelmReleases yaml files in my repos? Or is that a manual task? And by manual, I mean using a quickfix list and cdo a substitute command.

[gruberdev]

@stefanprodan Is that just running the bootstrap command to upgrade the version used in the cluster?
And will that automatically update the apiVersions defined on all my HelmReleases yaml files in my repos? Or is that a manual task? And by manual, I mean using a quickfix list and cdo a substitute command.

@Mo0rBy

I think the upgrade process depends on how it was initially created. That means you should use the CLI if you used it to initialize Flux, instead of Terraform or plain manifests. And yes, re-running the boostrap command would upgrade Flux on the server-side.

With that said, the upgrade process of the HelmReleases is manual AFAIK, and for a good reason, some of the behavior of how the helm-controller changed, and thus, features like drift detection are now configurable on a resource level.

I would advise following gradual changes by manually replacing the apiVersion of less important resources and then upgrade the remaining resources fully.

[Mo0rBy]

Awesome, thanks @gruberdev, something I'll try later, other important things to do for now.