Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

CI is running parallel tests with undersized key buffers #2362

Open
mflendrich opened this issue Feb 4, 2025 · 1 comment
Open

CI is running parallel tests with undersized key buffers #2362

mflendrich opened this issue Feb 4, 2025 · 1 comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/from-migration Indicates that this issue is a copy of a corresponding issue mentioned in the description. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@mflendrich
Copy link
Collaborator

Issue originally authored by tnozicka as #2274

We are running 32 test in parallel in which a single ScyllaCluster needs several keys to be generated, yet we configure the buffers to:

- --crypto-key-buffer-size-min=3
- --crypto-key-buffer-size-max=6

https://gcsweb.scylla-operator.scylladb.com/gcs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1955/pull-scylla-operator-master-e2e-openshift-aws-parallel/1868673877073203200/artifacts/deploy/0/operator/50_operator.deployment.yaml

This is barely enough to use the cache for one cluster and given how underprovisioned the operator is in the CI we can't afford to generate all keys on demand, so we should use the cache as much as possible to utilize the free cycles we have.

This is an example timing from a test that failed because cert were generated on demand for multiple test and too slow

Dec 16 16:10:05.479: INFO: basic-kb7wb-auth-token: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-client-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-user-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-cql-connection-configs-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretUpdated

This likely cause many e2e tests to flake.
@mflendrich mflendrich added kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Feb 4, 2025
@mflendrich mflendrich mentioned this issue Feb 4, 2025
Closed
@mflendrich mflendrich added the lifecycle/from-migration Indicates that this issue is a copy of a corresponding issue mentioned in the description. label Feb 4, 2025
@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out

/lifecycle stale

@scylla-operator-bot scylla-operator-bot bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 7, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/from-migration Indicates that this issue is a copy of a corresponding issue mentioned in the description. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mflendrich