fix(applicationset): use requeue after if generate app errors out

[MrFreezeex]

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

The GenerateApplications can call to external resources like Github API for instance which might be rate limited or fail. If those requests somehow fail we should requeue them after some time like (same reason as

argo-cd/applicationset/controllers/applicationset_controller.go

Line 154 in e98d3b2

// While some generators may return an error that requires user intervention,

).

For instance, in our environments (argocd 2.11.3) we were rate limited by Github and the ArgoCD applicationset controller was logging the following error about every second or less for every application set using the pull request generator that we have:

time="2024-06-21T14:17:15Z" level=error msg="error generating params" error="error listing repos: error listing pull requests for LedgerHQ/xxx: GET https://api.github.com/repos/LedgerHQ/xxx/pulls?per_page=100: 403 API rate limit exceeded for installation ID xxx. If you reach out to GitHub Support for help, please include the request ID xxx and timestamp 2024-06-21 xxx UTC. [rate reset in 8m18s]" generator="&{0xc000d652c0 0x289a100 {0xc00087bdd0}  [] true}"
time="2024-06-21T14:17:15Z" level=error msg="error generating application from params" applicationset=argocd/xxx error="error listing repos: error listing pull requests for LedgerHQ/xxxx: GET https://api.github.com/repos/LedgerHQ/xxx/pulls?per_page=100: 403 API rate limit exceeded for installation ID xxx. If you reach out to GitHub Support for help, please include the request ID xxx and timestamp 2024-06-21 xxx UTC. [rate reset in 8m18s]" generator="{nil nil nil nil nil &PullRequestGenerator{Github:&PullRequestGeneratorGithub{Owner:LedgerHQ,Repo:xxx,API:,TokenRef:nil,AppSecretName:xxxx,Labels:[argocd/preview],},GitLab:nil,Gitea:nil,BitbucketServer:nil,Filters:[]PullRequestGeneratorFilter{},RequeueAfterSeconds:*1800,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},Bitbucket:nil,AzureDevOps:nil,} nil nil nil nil}"

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.60%. Comparing base (1aa898c) to head (5625c19).
Report is 607 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18761      +/-   ##
==========================================
+ Coverage   50.36%   50.60%   +0.23%     
==========================================
  Files         315      315              
  Lines       43190    43205      +15     
==========================================
+ Hits        21752    21862     +110     
+ Misses      18954    18841     -113     
- Partials     2484     2502      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[crenshaw-dev]

I think this change makes sense. Would you mind adding a test? I see that there are already tests to verify that some kinds of errors cause the reconciliation requeue time to be set to that constant. You can build on those existing tests.

[MrFreezeex]

I think this change makes sense. Would you mind adding a test? I see that there are already tests to verify that some kinds of errors cause the reconciliation requeue time to be set to that constant. You can build on those existing tests.

@crenshaw-dev Sure, I just added a test like you suggested 👍.

[terrytangyuan]

+1 This would be very useful

[ishitasequeira]

Thanks @MrFreezeex for the PR. The change makes sense and LGTM!

[ishitasequeira]

@crenshaw-dev do you have any other concerns/comments?

test added

[crenshaw-dev]

No concerns!

[blakepettersson]

/cherry-pick release-2.12

[blakepettersson]

/cherry-pick release-2.11

[ishitasequeira]

@blakepettersson , I see that you cherry-picked the PR, but the cherrypick PRs seem to be failing tests due to lint failures. Can you raise separate PRs if this PR needs to be cherry-picked?