Setup a job to automatically run and PR the results of audit/audit-gcp.sh

[spiffxp]

/assign @hh
/wg k8s-infra

EDIT: redoing description entirely, things have changed since this issue was created

We want a prowjob that does the following:

• runs on k8s-infra-prow-build-trusted
• uses the k8s-infra-gcp-auditor GCP service account via workload identity
• runs the audit/audit-gcp.sh script
• if there are changes, opens or updates an existing PR to k8s.io (similar to how prow opens/updates autobump PRs)
• (optionally) fails if the existing PR has been open for too long

I am super open to suggestions about whether there are better or more-actionable ways to do auditing. But we need to start with something.

I sketched out what such a job would look like here: https://github.com/bashfire/prow-config/blob/435a8039bc9cf496690ad572884a72e9608ebb4e/config/jobs/bashfire/k8s-io.yaml
This is one of the first things we want to setup on a freshly created k8s-infra cluster to be sure we actually have the cluster and all of the IAM policies / roles created properly

First run as Job, next run as CronJob

[spiffxp]

creating the cluster ref: #243

[spiffxp]

CronJob would open a PR (akin to how prow maintains a bump PR?)

[hh]

Depends on script in this PR: #213

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[scottilee]

@hh is this issue still relevant? If so, it seems like there should be a CronJob that periodically checks that a specified cluster has the required IAM policies and when it doesn't open a PR in this repo to fix? Let me know if I'm missing anything.

[cblecker]

/remove-lifecycle stale

I think it is. I think the point is to run this regularly and then output the details

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[spiffxp]

/remove-lifecycle rotten

[spiffxp]

I am noodling on something over in https://github.com/bashfire/prow-config/blob/master/config/jobs/bashfire/k8s-io.yaml

The job will output the diff at the end. What I'd like to do next is create a PR if there are any changes, and continue to update that PR until it's been merged

[spiffxp]

/assign @spiffxp @bartsmykla
bart to help with refactoring pr-creator

[bartsmykla]

I will sit with @munnerz next week and we'll move this thing forward together (as discussed at today's call)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/priority important-soon
/milestone v1.21

[spiffxp]

/remove-priority backlog

[rikatz]

/lifecycle active
So bot doesn't close this :P
Will see at least if I can find someone to take a look into this and help :)

[spiffxp]

Chatted with @hh about this

[spiffxp]

/uncc @bartsmykla
/priority critical-urgent
During yesterday's meeting @thockin suggested we should be making no more big changes to infra until we have this taken care of. I don't like blocking everything for too long, but I am inclined to agree.

[hh]

/assign @hh

[hh]

I didn't get a chance to dive deep into this, but we have a somewhat working similar script that does currently create or update a PR when changes have occurred in conformance data underpinning apisnoop.cncf.io :

https://github.com/cncf/apisnoop/blob/gcb-snoodb-pr-gater/cloudbuild.yaml

[bernokl]

For this job I would suggest using the working pr-creator from the yaml hh posted above,
The alternative is to use golang-pr creator, I am reluctant to incur the overhead of the golang-pr creator, when the solution is so short.
Is there a reason to use the entire complexity of golang when our entire job that currently works is only this?

cp -r coverage/[artifact] path/to/repo/[artifact]
cd repo
git diff > ../GIT_DIFF
if [ -s ../GIT_DIFF ]
then
git config --global user.email "cncf-ci@ii.coop"
git config --global user.name "CNCF CI Bot"
git checkout -b snoop-auto-updates
git add .
git commit -m "Changes in coverage diff $(date)"
git push --force https://$(cat ../github-token.txt)@github.com/cncf/apisnoop snoop-auto-updates
GIT_EDITOR=cat GITHUB_TOKEN=$(cat ../github-token.txt) ../hub pr list | grep "Changes in coverage diff" > ../PR_EXISTS
if [ -s ../PR_EXISTS ]
then
echo "PR exists already"
else
GIT_EDITOR=cat GITHUB_TOKEN=$(cat ../github-token.txt) ../hub pull-request -p
fi

[bernokl]

Currently we use gcb for our build which means our credentials/secrets are in the gcb project.
We saw that your mockup for this job uses google secrets in trusted cluster.
How do you see the job getting access to those secrets?
Do we want to run this as a gcb job with its own secrets or should this job run in the trusted cluster? it is currently a multi-step job

[spiffxp]

/remove-help
Since I think this is being worked by some of @hh's team.

Discussed in meetings but to close the loop here:

• this should run as a prowjob on k8s-infra-prow-build-trusted, the gcp-auditor service account is available there
• the github token could either be stored in secret manager or could be added to k8s-infra-prow-build-trusted as a secret (similar to how secrets are manually added to the default cluster used by prow.k8s.io)

[spiffxp]

/reopen
/area infra/auditing

[k8s-ci-robot]

@spiffxp: Reopened this issue.

In response to this:

/reopen
/area infra/auditing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

#1648 - auditor account didn't have access to secretmanager, add a one-off script for this for now

Sample audit PR's manually opened by @hh

• Audit Update for kubernetes-public #1650
• Audit Update for k8s-infra-* (non-boskos) #1651
• Audit Update for k8s-infra-boskos 64/160 #1652

I have /held these PRs because I would like to merge the PR I opened last month that annotated many of the changes that have happened in the past 7 months commit by commit. I am in the midst of updating it right now, and suggest I leave out some of the questionable changes I can't account for, for the automation to PR instead (#1534 (comment))

[hh]

We now have a set of partial-audits PR created via a prow job.
They are smaller, though some touch 450 files (boskos-*).
Let me know if some. need broken down into smaller PRs.

It was created by using a gist of a slightly modify audit-gcp.sh over a few runs, using a different branch each time.

• https://gist.github.com/hh/eefe79116d11b7f4aa412b2b7fd02df8/revisions

The changes to the job are here:

• kubernetes/test-infra@master...ii:ci-k8sio-audit-3

And the resulting list of 10 PRs:

• https://github.com/kubernetes/k8s.io/pulls?q=is%3Apr+is%3Aopen+label%3Aarea%2Finfra%2Fauditing+%22partial-audit%3A%22+

[spiffxp]

Copy-pasting from #1676 (comment)

The partial-audit PR's opened by cncf-ci caught almost everything. I reviewed and approved them, but I can't do anything further to merge them until the account has signed the CLA. Use /check-cla to refresh once you've gotten it signed, and they should merge (hopefully without conflict)

• partial-audit: k8s-infra-* update as of 2021-02-24 #1717
• partial-audit: prow-boskos-* batch (4/4) update as of 2021-02-24 #1716
• partial-audit: prow-boskos-* batch (3/4) update as of 2021-02-24  #1715
• partial-audit: prow-boskos-* batch (1/4) update as of 2021-02-24  #1714
• partial-audit: prow-boskos-* batch (2/4) update as of 2021-02-24 #1713
• partial-audit: k8s-staging-* update as of 2021-02-24 #1712
• partial-audit: k8s-infra-e2e-boskos-scale-* update as of 2021-02-24 #1711
• partial-audit: k8s-infra-boskos-gpu-* update as of 2021-02-24 #1710

The outstanding changes I didn't see covered in them are:

• audit: add psharma to org admins - a9efe55
• audit: add psharma as owner to kubernetes-public - f8a9628
• audit: update gcp-auditor org-scoped roles - f0d0b63

[spiffxp]

/close
Thank you for your help in getting this over the line @hh!

I think we can call this done now:

• job config in kubernetes/test-infra
• shows up on testgrid: https://testgrid.k8s.io/wg-k8s-infra-k8sio#ci-k8sio-audit
• sample PRs created by the job:
  • audit: update as of 2021-03-03 #1748
    • PR was not updated-in-place over time as nothing had changed between runs
  • audit: update as of 2021-03-11 #1755
    • PR was updated-in-place three times
    • PR was changed to reflect the new date over time

I have some quality of life suggestions for improvements, but I will make those elsewhere.

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Thank you for your help in getting this over the line @hh!

I think we can call this done now:

• job config in kubernetes/test-infra
• shows up on testgrid: https://testgrid.k8s.io/wg-k8s-infra-k8sio#ci-k8sio-audit
• sample PRs created by the job:
• audit: update as of 2021-03-03 #1748
  • PR was not updated-in-place over time as nothing had changed between runs
• audit: update as of 2021-03-11 #1755
  • PR was updated-in-place three times
  • PR was changed to reflect the new date over time

I have some quality of life suggestions for improvements, but I will make those elsewhere.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.