Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Intermittent warning/skips on Composer rules #72

Open
qgallet opened this issue Jun 30, 2023 · 1 comment
Open

Intermittent warning/skips on Composer rules #72

qgallet opened this issue Jun 30, 2023 · 1 comment

Comments

@qgallet
Copy link

qgallet commented Jun 30, 2023

This was working fine on 0.59, now there seems to be some lock issue on Composer API queries. Here's the command used:

gcpdiag lint --project=my-project --include=composer

At the start of the command, the following log will be temporarily visible on the console, before the results are printed.

... still fetching logs (project: my-project, resource type: cloud_composer_environment, max wait: 117s)

After two minutes, some of the checks are skipped because of (I guess) a timeout:

🔎  composer/BP/2023_001: Cloud Composer logging level is set to INFO
   - my-project/europe-west1/my-composer               [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_003 
🔎  composer/ERR/2022_001: Composer Service Agent permissions
   - my-project                                        [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2022_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2023_001 
🔎  composer/WARN/2022_001: Composer Service Agent permissions for Composer 2.x
   - my-project                                        [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_003 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_001 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_003 
🔎  composer/WARN/2023_004: Cloud Composer database CPU usage does not exceed 80%
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_005: Cloud Composer is consistently in healthy state
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_006: Airflow schedulers are healthy for the last hour
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_007: Cloud Composer Scheduler CPU limit exceeded.
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_008: Cloud Composer Airflow database is in healthy state
   - my-project/europe-west1/my-composer               [ OK ]

Rules summary: 9 skipped, 8 ok, 0 failed

If I launch the command with a more restrictive parameter, like --include=composer/WARN, this still happens although less frequently.

@ropeck
Copy link
Collaborator

ropeck commented Jul 31, 2023

Thanks for the issue report. Can you check if this still occurs with v0.63 recently released?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants