tests: add more tests

[lorenzofelletti]

In light of the recent issue with the API server availability rules, which was hard to catch by simply looking at the git diff, I think that adding more thorough testing is due to make repeating such a mistake harder in the future.

In this PR I'm adding more tests around the API server rules, and I'm also moving the tests to a dedicated folder, allowing to break the giant tests.yam file in multiple, more readable ones. The Makefile is updated accordingly, so to run all test files in the tests folder on make test.

[skl]

Thank you for doing this, @lorenzofelletti! I was thinking the same.

One thing to watch out for - there's a few dependabot PRs open which seem to depend on go.mod being bumped to 1.23, as well as the prometheus version being bumped from 0.53 to 0.300, for example:

• chore(deps): bump github.com/cloudflare/pint from 0.62.2 to 0.69.1 in /scripts #994

Note that the prometheus version bump appears to change the behaviour of the tests (understandable given how out of date the deps were).

I can fix these on #994 but you'll need to make sure to merge/rebase latest master branch, just wanted to give you a heads up in case your test behaviour is affected.

[lorenzofelletti]

Hi @skl, thanks for letting me know. I'll rebase/merge master to keep this in sync.

[skl]

@lorenzofelletti deps updated including prometheus, fixed tests to match latest behaviour in a2e2d9a. Please pull latest master and apologies for the conflicts!

[github-actions]

This PR has been automatically marked as stale because it has not
had any activity in the past 30 days.

The next time this stale check runs, the stale label will be
removed if there is new activity. The issue will be closed in 7
days if there is no new activity.

Thank you for your contributions!

[lorenzofelletti]

Hey @skl, I just didn't have bandwidth to progress on this for the last month or so!
Now I'm willing to continue with this one, could you please reopen this PR or should I create a new one?

[skl]

Reopened! Make sure to merge or rebase latest main, thanks!

[lorenzofelletti]

@skl I have a question:

Why is it that if I teak the named tests to fail, I get the failed test name reported in the pre-existing tests, but not in the ones I added?

E.g.:

Name shown:

tests:
# ...
- name: KubeletTooManyPods alert (single-node with instance as IP address)

  FAILED:
    name: KubeletTooManyPods alert (single-node with instance as IP address),
    alertname: KubeletTooManyPods, time: 1m, 
        exp:[
            0:
              Labels:{alertname="KubeletTooManyPods", cluster="kubernetes", instance="10.0.0.0", node="node0", severity="info"}
              Annotations:{description="Kubelet 'node0' is running at 100% of its Pod capacity.", runbook_url="https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods", summary="Kubelet is running at capacity."}
            ], 
        got:[]

Name not shown:

tests:
# ...
- name: calculate apiserver request total increase 30d rate

  FAILED:
    expr: "code_verb:apiserver_request_total:increase30d{verb=\"GET\"}", time: 1m,
        exp: {__name__="code_verb:apiserver_request_total:increase30d", code="200", verb="GET"} 3.96E+04, {__name__="code_verb:apiserver_request_total:increase30d", code="500", verb="GET"} 3.24E+03
        got: {__name__="code_verb:apiserver_request_total:increase30d", code="200", verb="GET"} 1.08E+04, {__name__="code_verb:apiserver_request_total:increase30d", code="500", verb="GET"} 3.6E+02

I can't understand why this behaviour...

[skl]

@lorenzofelletti maybe it's a bug in promtool - there seems to be an inconsistency between the failure messages shown for promql_expr_test definitions, vs those shown for alert_rule_test definitions (the test name seems to only appear on the latter).