Upgrade from v2.11 to 2.12 Breaks Git Secret

[IdoOzeri]

I've upgraded my Argocd installation via the official chart.
From chart version 7.3.11 (app version v2.11.7) to 7.4.0 (app version v2.12.0).

I have a Secret resource in my cluster with the following label:

argocd.argoproj.io/secret-type: repository

Once I upgraded to v2.12.0, this repository connection still appears as "Successful" in the user interface, but my applications can no longer sync, complaining of the following error:

SSH agent requested but SSH_AUTH_SOCK not-specified

I've done some digging and found this issue, which suggested modifying the secret-type label to repo-creds. Once I did that, the problem was gone.

My question is, is the secret-type: repository label deprecated in > v2.12.0?
I haven't seen any deprecation notice in the docs.

[rickardk-se]

Are you using the same secret across projects?

[rhaex]

I experienced the same issue when upgrading from v2.11.2 to v.2.12.3 using the manifest https://raw.githubusercontent.com/argoproj/argo-cd/v2.12.3/manifests/install.yaml.

Everything upgrades fine, but after the upgrade the SSH connection to the repo is broken and gives the errors as seen above and in the linked issue.

And yes, we use the same repo & secret for several projects.

Modifying the label worked:
kubectl -n argocd label --overwrite secret repo-xxxxx argocd.argoproj.io/secret-type=repo-creds

So indeed it looks like the old value has been deprecated and was removed without mentioning it in the release notes or upgrade documentation.

[IdoOzeri]

Are you using the same secret across projects?

Yes, I only have 1 project and 1 secret. Nothing changed but the Argo version.

[rickardk-se]

Sharing secrets across projects stopped working in 2.12.

[rhaex]

Sharing secrets across projects stopped working in 2.12.

Yeah I know, but our secret is not bound to any project. So all projects use the same repo secret, but the secret itself is not bound to a project (so without project field as described here https://argo-cd.readthedocs.io/en/latest/operator-manual/upgrading/2.11-2.12/).

The upgrade breaks the repo connection, until you change secret-type: repository into secret-type: repo-creds, after everything works fine again. And that breaking change is not mentioned anywhere (was it even intentional?).

[blakepettersson]

Sharing secrets across projects stopped working in 2.12.

That is not the case.

And that breaking change is not mentioned anywhere (was it even intentional?).

secret-type: repository definitely works. What did change in 2.12 was that if a secret had a project value set, it can only be used by applications within that same project.

The upgrade breaks the repo connection, until you change secret-type: repository into secret-type: repo-creds, after everything works fine again. And that breaking change is not mentioned anywhere (was it even intentional?).

I haven't seen that one at least, nor have I heard of this elsewhere.

[CubicFeat]

We also ran into this issue.

Changing the label to repo-creds did solve the issue, but I noticed afterward that the k8s secret does contain a project field, and it was set to default, which is exactly what the upgrade notes say will be an issue. We may have created it this way years ago, I'm not sure, but from what I can see there is no way to see or set that via the UI.

I've since changed the label back to just repository and completely removed the project field from the k8s secret, and things still work.

@blakepettersson any chance there is a feature coming where a person can see what projects a repository is scoped to? There are no "view" or "edit" options under Settings->repositories, just a list.

[blakepettersson]

@CubicFeat do you mean in the UI? Since 2.12 there is a PROJECT column to see which project a repository can be scoped to. If it is empty it's globally scoped (it can be used by any project).

[Dutchy-]

Which upgrade notes mention the default project? Because it is not on https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.11-2.12/ and I think it should be.

[blakepettersson]

Hmm I guess the upgrade notes have not been propagated to readthedocs: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/upgrading/2.11-2.12.md

[andrii-korotkov-verkada]

Can you share the secret yaml with masked data, please? Also, the code that references the secret.

[wyardley]

We are seeing these sync errors on one of our apps, since updating from, I believe, 2.10.x, to 2.13.x via the latest helm chart. We don't have a project field set. The projects in question are syncing fine; and the specific app shows up as "synced" and "healthy" -- we just see (consistent) errors in the web UI.

I did note that there are url and type fields set (the url is set to a vendor's repo, so maybe something from the original setup), though don't think that is an issue? I did try adding the label (argocd.argoproj.io/secret-type: repo-creds) which was not previously set, and restarting argocd-server and argocd-repo-server deployments, and it didn't resolve the issue. I'm not sure what the value in type is.

Here's the value of the github-repo-secret secret, if it helps.

apiVersion: v1
data:
  sshPrivateKey: XXXXX
  type: XXXX
  url: XXXXXXXXXX
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2024-02-21T16:43:59Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    argocd.argoproj.io/secret-type: repo-creds
  name: github-repo-secret
  namespace: argocd
  resourceVersion: "393638178"
  uid: daca6d32-636f-42f9-bf96-5ef5b2a388be
type: Opaque

Spec:
  Destination:
    Name:       in-cluster
    Namespace:  xxxx
  Project:      xxxx
  Source:
    Path:             xxxx
    Repo URL:         git@github.com:privaterepo/reponame.git
    Target Revision:  HEAD

Note: the repo, reponame, and path within the repo all exist and are correct.

Edit I confirmed that the cred in this secret does not currently work, but each repo has its own repo-specific secret (like repo-NNNNNNNNN), which is presumably the one the sync project is actually using. So this may explain the weird behavior in the UI. Does argo boostrap the repo-NNNNNNN ones itself using deploy keys?