-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: compute alerts on systematic failure (#4)
Stop showing the ignoreFail as a code error. Create a short summary for cases where the full summary would be truncated by GH. Compute an Alerts section and make it available standalone for use in Slackops. Add unit tests. Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Tao Li <tao.li@enterprisedb.com> Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com> Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
- Loading branch information
1 parent
8b2bd1b
commit d4314a5
Showing
27 changed files
with
641 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,67 @@ | ||
on: [push] | ||
|
||
defaults: | ||
run: | ||
# default failure handling for shell scripts in 'run' steps | ||
shell: 'bash -Eeuo pipefail -x {0}' | ||
|
||
jobs: | ||
overflow_test: | ||
runs-on: ubuntu-latest | ||
name: Test CIclops with summary overflow | ||
steps: | ||
- uses: actions/checkout@v3 | ||
|
||
- name: Generate Test Summary | ||
uses: ./ | ||
with: | ||
artifact_directory: example-artifacts/ | ||
output_file: test-summary.md | ||
short_file: short.md | ||
alerts_file: alerts.txt | ||
# NOTE: it does not seem possible to pass $GITHUB_STEP_SUMMARY as a | ||
# regular file to the underlying script. Hence, we create a file and | ||
# in a later step write with >> $GITHUB_STEP_SUMMARY | ||
|
||
- name: Create local file that is bigger than GH limit | ||
run: | | ||
dd if=/dev/zero of=big-test-summary.md bs=1M count=2 | ||
- name: Check full summary fits within GH limit | ||
# $GITHUB_STEP_SUMMARY will reject content over 1024 bytes | ||
# on exceeding, the workflow WILL FAIL and still count as success() | ||
# With this step, we do proper error flow, and fail if the limit would be | ||
# exceeded. | ||
id: check-overflow | ||
run: | | ||
size=$(stat -c '%s' big-test-summary.md) | ||
if [ "$size" -gt 1024 ]; then | ||
echo "overflow=true" >> $GITHUB_OUTPUT | ||
fi | ||
# Here we force the "big test summary" to overflow GH limits, and we | ||
# create an output that further steps can leverage: steps.check-overflow.outputs.overflow | ||
|
||
- name: If the full summary fits within GH limits, use it | ||
# This step should be skipped, we expect | ||
if: ${{!steps.check-overflow.outputs.overflow}} | ||
run: | | ||
cat big-test-summary.md >> $GITHUB_STEP_SUMMARY | ||
- name: If the full summary is too big, use short version | ||
if: ${{steps.check-overflow.outputs.overflow}} | ||
run: | | ||
cat short.md >> $GITHUB_STEP_SUMMARY | ||
- name: If full summary is too big, archive it | ||
if: ${{steps.check-overflow.outputs.overflow}} | ||
uses: actions/upload-artifact@v3 | ||
with: | ||
name: test-summary.md | ||
path: test-summary.md | ||
retention-days: 7 | ||
|
||
- name: Create slack body with alerts | ||
run: | | ||
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT | ||
cat alerts.txt >> $GITHUB_OUTPUT | ||
echo 'EOF' >> $GITHUB_OUTPUT |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
on: [push] | ||
|
||
jobs: | ||
plain_test: | ||
runs-on: ubuntu-latest | ||
name: Unit test | ||
steps: | ||
- uses: actions/checkout@v3 | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v4 | ||
with: | ||
python-version: '3.x' | ||
|
||
- name: Install dependencies | ||
run: python -m pip install --upgrade pip prettytable | ||
|
||
- name: Run suite | ||
run: python test_summary.py -v |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,3 @@ | ||
# editor and IDE paraphernalia | ||
.idea | ||
__pycache__/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,4 +14,4 @@ | |
"workflow_id": 12, | ||
"repo": "my-repo", | ||
"branch": "my-branch" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,4 +14,4 @@ | |
"workflow_id": 12, | ||
"repo": "my-repo", | ||
"branch": "my-branch" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,4 +14,4 @@ | |
"workflow_id": 12, | ||
"repo": "my-repo", | ||
"branch": "my-branch" | ||
} | ||
} |
19 changes: 18 additions & 1 deletion
19
...-0.0git256.g2a1b38b.1.dev-1_86b346ab9958ad86dc1829a4b9f509415ce8da7d997e5ea40bb24af8.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,18 @@ | ||
{"name": "Update operand -- upgrades operand version", "state": "failed", "start_time": "2023-01-25T10:26:03.771385171Z", "end_time": "2023-01-25T10:38:44.769651724Z", "error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc0003c4c20>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1", "error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", "error_line": 268, "platform": "local", "postgres_kind": "PostgreSQL", "redwood": false, "matrix_id": "local-v1.21.14-PGD-PostgreSQL-12.13-5.0.0-0.0git256.g2a1b38b.1.dev-1", "postgres_version": "12.13-5.0.0-0.0git256.g2a1b38b.1.dev-1", "k8s_version": "v1.21.14", "workflow_id": 4004501040, "repo": "EnterpriseDB/pg4k-pgd", "branch": "dev/cnp-3285-2"} | ||
{ | ||
"name": "Update operand -- upgrades operand version", | ||
"state": "failed", | ||
"start_time": "2023-01-25T10:26:03.771385171Z", | ||
"end_time": "2023-01-25T10:38:44.769651724Z", | ||
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc0003c4c20>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1", | ||
"error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", | ||
"error_line": 268, | ||
"platform": "local", | ||
"postgres_kind": "PostgreSQL", | ||
"redwood": false, | ||
"matrix_id": "local-v1.21.14-PGD-PostgreSQL-12.13-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"postgres_version": "12.13-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"k8s_version": "v1.21.14", | ||
"workflow_id": 4004501040, | ||
"repo": "EnterpriseDB/pg4k-pgd", | ||
"branch": "dev/cnp-3285-2" | ||
} |
19 changes: 18 additions & 1 deletion
19
...-0.0git256.g2a1b38b.1.dev-1_86b346ab9958ad86dc1829a4b9f509415ce8da7d997e5ea40bb24af8.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,18 @@ | ||
{"name": "Update operand -- upgrades operand version", "state": "failed", "start_time": "2023-01-25T11:08:20.179946291Z", "end_time": "2023-01-25T11:20:54.376679592Z", "error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc000a41e40>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1", "error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", "error_line": 268, "platform": "local", "postgres_kind": "PostgreSQL", "redwood": false, "matrix_id": "local-v1.21.14-PGD-PostgreSQL-13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", "postgres_version": "13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", "k8s_version": "v1.21.14", "workflow_id": 4004501040, "repo": "EnterpriseDB/pg4k-pgd", "branch": "dev/cnp-3285-2"} | ||
{ | ||
"name": "Update operand -- upgrades operand version", | ||
"state": "failed", | ||
"start_time": "2023-01-25T11:08:20.179946291Z", | ||
"end_time": "2023-01-25T11:20:54.376679592Z", | ||
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc000a41e40>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1", | ||
"error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", | ||
"error_line": 268, | ||
"platform": "local", | ||
"postgres_kind": "PostgreSQL", | ||
"redwood": false, | ||
"matrix_id": "local-v1.21.14-PGD-PostgreSQL-13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"postgres_version": "13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"k8s_version": "v1.21.14", | ||
"workflow_id": 4004501040, | ||
"repo": "EnterpriseDB/pg4k-pgd", | ||
"branch": "dev/cnp-3285-2" | ||
} |
19 changes: 18 additions & 1 deletion
19
...-0.0git256.g2a1b38b.1.dev-1_bc078a3b7d0af245efb2c1a040e8d51bbeb3f218e998a035ad896d22.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,18 @@ | ||
{"name": "Smoke test -- sets up a PGD Group according to flexible three regions architecture", "state": "failed", "start_time": "2023-01-25T10:38:50.421338678Z", "end_time": "2023-01-25T10:52:23.675119439Z", "error": "Timed out after 602.794s.\nError: Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:463 failed:\nExpected\n <*fmt.wrapError | 0xc00007d8e0>: {\n msg: \"context deadline exceeded - \",\n err: <context.deadlineExceededError>{},\n }\nto be nil\n <*errors.errorString | 0xc0003b2650>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:463 failed:\\nExpected\\n <*fmt.wrapError | 0xc00007d8e0>: {\\n msg: \\\"context deadline exceeded - \\\",\\n err: <context.deadlineExceededError>{},\\n }\\nto be nil\",\n }", "error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", "error_line": 466, "platform": "local", "postgres_kind": "PostgreSQL", "redwood": false, "matrix_id": "local-v1.21.14-PGD-PostgreSQL-13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", "postgres_version": "13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", "k8s_version": "v1.21.14", "workflow_id": 4004501040, "repo": "EnterpriseDB/pg4k-pgd", "branch": "dev/cnp-3285-2"} | ||
{ | ||
"name": "Smoke test -- sets up a PGD Group according to flexible three regions architecture", | ||
"state": "failed", | ||
"start_time": "2023-01-25T10:38:50.421338678Z", | ||
"end_time": "2023-01-25T10:52:23.675119439Z", | ||
"error": "Timed out after 602.794s.\nError: Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:463 failed:\nExpected\n <*fmt.wrapError | 0xc00007d8e0>: {\n msg: \"context deadline exceeded - \",\n err: <context.deadlineExceededError>{},\n }\nto be nil\n <*errors.errorString | 0xc0003b2650>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:463 failed:\\nExpected\\n <*fmt.wrapError | 0xc00007d8e0>: {\\n msg: \\\"context deadline exceeded - \\\",\\n err: <context.deadlineExceededError>{},\\n }\\nto be nil\",\n }", | ||
"error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", | ||
"error_line": 466, | ||
"platform": "local", | ||
"postgres_kind": "PostgreSQL", | ||
"redwood": false, | ||
"matrix_id": "local-v1.21.14-PGD-PostgreSQL-13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"postgres_version": "13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"k8s_version": "v1.21.14", | ||
"workflow_id": 4004501040, | ||
"repo": "EnterpriseDB/pg4k-pgd", | ||
"branch": "dev/cnp-3285-2" | ||
} |
19 changes: 18 additions & 1 deletion
19
...-0.0git256.g2a1b38b.1.dev-1_86b346ab9958ad86dc1829a4b9f509415ce8da7d997e5ea40bb24af8.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,18 @@ | ||
{"name": "Update operand -- upgrades operand version", "state": "failed", "start_time": "2023-01-25T10:42:32.45376546Z", "end_time": "2023-01-25T10:54:59.12104098Z", "error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc0000b3ce0>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1", "error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", "error_line": 268, "platform": "local", "postgres_kind": "PostgreSQL", "redwood": false, "matrix_id": "local-v1.21.14-PGD-PostgreSQL-14.6-5.0.0-0.0git256.g2a1b38b.1.dev-1", "postgres_version": "14.6-5.0.0-0.0git256.g2a1b38b.1.dev-1", "k8s_version": "v1.21.14", "workflow_id": 4004501040, "repo": "EnterpriseDB/pg4k-pgd", "branch": "dev/cnp-3285-2"} | ||
{ | ||
"name": "Update operand -- upgrades operand version", | ||
"state": "failed", | ||
"start_time": "2023-01-25T10:42:32.45376546Z", | ||
"end_time": "2023-01-25T10:54:59.12104098Z", | ||
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc0000b3ce0>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1", | ||
"error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go", | ||
"error_line": 268, | ||
"platform": "local", | ||
"postgres_kind": "PostgreSQL", | ||
"redwood": false, | ||
"matrix_id": "local-v1.21.14-PGD-PostgreSQL-14.6-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"postgres_version": "14.6-5.0.0-0.0git256.g2a1b38b.1.dev-1", | ||
"k8s_version": "v1.21.14", | ||
"workflow_id": 4004501040, | ||
"repo": "EnterpriseDB/pg4k-pgd", | ||
"branch": "dev/cnp-3285-2" | ||
} |
Oops, something went wrong.