Let artifact phase and post-command run in grace period

[DrJosh9000]

Description

Artifact phase (the built-in artifact upload performed by the agent) and post-command hooks are now allowed to run in the signal grace period (a configurable length of time after job cancellation).

The signal grace period is also now enforced within the job executor (as well as outside of it when run normally as a subprocess of the agent).

Context

Artifacts uploaded this way can be relevant to user debugging job timeouts, so we should try a bit harder to upload them.

A similar argument applies to post-command hooks, which can be used for post-job cleanup.

Both of these (and usual job teardown) are still subject to the executor being SIGKILL-ed by the agent parent process. However, some use-cases run buildkite-agent bootstrap separately to the agent ("standalone bootstrap", which happens for example in agent-stack-k8s). So the executor needs to be able to kill itself in case there's nothing else to do so, hence graceCtx instead of nonCancelCtx.

The executor (a.k.a. bootstrap) is also normally passed the (computed) signal grace period as a flag/env var from the parent agent process, but the flag shares the same definition as agent start (with default -1). If there is no parent agent process, the -1 signal grace period will cause graceCtx to be cancelled immediately after the main context is cancelled. An existing integration test shows this. So, for standalone bootstrap, the same flag computation that happens in agent start to obtain a positive signalGracePeriod should be done in bootstrap as well.

Changes

• Add withGracePeriod helper method, to create a new context that is cancelled some period of time after another context is cancelled.
• Use withGracePeriod to replace the existing detached nonCancelCtx with a new graceCtx. The difference is that graceCtx will still be cancelled after the signal grace period.
• Use graceCtx for tearDown, artifactPhase, and create a grace context for runPostCommandHooks.
• Refactor the computation with cancel-grace-period and signal-grace-period-seconds from agent start into a function shared with bootstrap.

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go fmt ./...)

[wolfeidau]

@DrJosh9000 how does this effect --signal-grace-period-seconds, will it like go over now that artifact uploads are outside this period?

So this change resolves issues with this flag breaking uploads 🎉

[wolfeidau]

I think we should also replace the context for the post command hook to avoid this error.

[wolfeidau]

Great work chasing down all these loose ends 🎉

Appreciate the extra internal documentation as well. ⭐