Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[ARCHIVED] CI is running parallel tests with undersized key buffers #2274

Closed
tnozicka opened this issue Dec 17, 2024 · 1 comment
Closed

[ARCHIVED] CI is running parallel tests with undersized key buffers #2274

tnozicka opened this issue Dec 17, 2024 · 1 comment
Labels
lifecycle/migrated-away Indicates that this issue is archived and tracking continues in a copy. priority/archived Lowest priority possible - means that the issue is archived.

Comments

@tnozicka
Copy link
Contributor

tnozicka commented Dec 17, 2024 •
edited by mflendrich
Loading

We are running 32 test in parallel in which a single ScyllaCluster needs several keys to be generated, yet we configure the buffers to:

- --crypto-key-buffer-size-min=3
- --crypto-key-buffer-size-max=6

https://gcsweb.scylla-operator.scylladb.com/gcs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1955/pull-scylla-operator-master-e2e-openshift-aws-parallel/1868673877073203200/artifacts/deploy/0/operator/50_operator.deployment.yaml

This is barely enough to use the cache for one cluster and given how underprovisioned the operator is in the CI we can't afford to generate all keys on demand, so we should use the cache as much as possible to utilize the free cycles we have.

This is an example timing from a test that failed because cert were generated on demand for multiple test and too slow

Dec 16 16:10:05.479: INFO: basic-kb7wb-auth-token: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-client-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-user-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-ca: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-cql-connection-configs-admin: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-local-serving-certs: SecretCreated
Dec 16 16:10:05.479: INFO: basic-kb7wb-alternator-local-serving-certs: SecretUpdated

This likely cause many e2e tests to flake.
@tnozicka tnozicka added kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Dec 17, 2024
@tnozicka tnozicka self-assigned this Dec 17, 2024
@tnozicka tnozicka changed the title CI is running parallel test with undersized key buffers CI is running parallel testd with undersized key buffers Dec 17, 2024
@tnozicka tnozicka changed the title CI is running parallel testd with undersized key buffers CI is running parallel tests with undersized key buffers Dec 17, 2024
@tnozicka tnozicka mentioned this issue Dec 17, 2024
Closed
@tnozicka tnozicka added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Dec 17, 2024
@rzetelskik rzetelskik mentioned this issue Dec 17, 2024
Merged
@tnozicka tnozicka closed this as not planned Won't fix, can't repro, duplicate, stale Jan 2, 2025
@rzetelskik rzetelskik mentioned this issue Jan 9, 2025
Closed
@mflendrich mflendrich mentioned this issue Feb 4, 2025
Open
@mflendrich
Copy link
Collaborator

Will be tracked in #2362.

@scylladb scylladb locked and limited conversation to collaborators Feb 4, 2025
@mflendrich mflendrich added lifecycle/migrated-away Indicates that this issue is archived and tracking continues in a copy. priority/archived Lowest priority possible - means that the issue is archived. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/flake Categorizes issue or PR as related to a flaky test. labels Feb 4, 2025
@mflendrich mflendrich changed the title CI is running parallel tests with undersized key buffers [ARCHIVED] CI is running parallel tests with undersized key buffers Feb 4, 2025
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees
No one assigned
Labels
lifecycle/migrated-away Indicates that this issue is archived and tracking continues in a copy. priority/archived Lowest priority possible - means that the issue is archived.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adjust key buffer sizes in CI based on parallelism tnozicka/scylla-operator
2 participants
@mflendrich @tnozicka