diff --git a/keps/prod-readiness/sig-node/5307.yaml b/keps/prod-readiness/sig-node/5307.yaml new file mode 100644 index 00000000000..1e5f9b1fd94 --- /dev/null +++ b/keps/prod-readiness/sig-node/5307.yaml @@ -0,0 +1,3 @@ +kep-number: 5307 +alpha: + approver: "johnbelamaric" diff --git a/keps/sig-node/5307-container-restart-policy/README.md b/keps/sig-node/5307-container-restart-policy/README.md new file mode 100644 index 00000000000..b4d88628f1f --- /dev/null +++ b/keps/sig-node/5307-container-restart-policy/README.md @@ -0,0 +1,1101 @@ + +# KEP-5307: Container Restart Policy + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Container Restart Count](#container-restart-count) + - [Job PodFailurePolicy](#job-podfailurepolicy) + - [Risks and Mitigations](#risks-and-mitigations) + - [Unintended Restart Loops](#unintended-restart-loops) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + - [Wrapping entrypoint](#wrapping-entrypoint) + - [Non-declarative (callbacks based) restart policy](#non-declarative-callbacks-based-restart-policy) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +This KEP introduces restart policies for a container so kubelet can apply +those rules on container failure. It extends the existing Never, OnFailure, +and Always restart policies. This will allow users to configure special exit +codes of the container, representing the restart of a task, to be treated as +non-failure even if the Pod has a restartPolicy=Never. This scenario is +important for use cases, when rescheduling of a task is very expensive, +and restarting in-place is preferable. + +## Motivation + + + +With the proliferation of AI/ML training jobs where each job takes hundreds +of Pods, each using expensive hardware and very expensive to schedule, +in-place restarts are becoming more and more important. + +Consider the example, the Pod is a part of a large training job. The progress +of each training “step” is only made when all Pods completed the calculation +for this step. Each Pod starts from a checkpoint, they all make progress +together, and write a new checkpoint. If any of Pods failed, the fastest +way to restart the calculation is to interrupt all Pods by restarting them, +so they all will start from the previous checkpoint. Thus, a special handling +of this restart is required. + +There are a few reasons why the OnFailure restart policy will not work: + +1) The cases of failed hardware must result in a Pod failure and rescheduling. +There needs to be a differentiation of these two failures - caused by hardware +issue and caused by a in-place restart request. + +2) Pods are often parts of JobSets with the Job failure policy configured +(see https://kubernetes.io/docs/tasks/job/pod-failure-policy/). The pod +failure policy is a server-side policy and is not compatible with the Pods +with restartPolicy OnFailure. + +### Goals + + + +- Introduce API which allows to keep restarting a container on specified exit codes. +- Allow extensibility of an API to support more scenarios in future. + +### Non-Goals + + + +- Implement the ​​maxRestartTimes https://github.com/kubernetes/enhancements/issues/3322 +- Support all possible restart policy rules in this KEP - some may be ideas for the future + + +## Proposal + + + +### User Stories (Optional) + + + +#### Story 1 + +As a ML researcher, I'm orchestrating a large number of long-running AI/ML training +worklaods. Workload failures in such workloads are unavoidable due to various reasons. +When a workload fails (with a retriable exit code), I would like the container to be +restarted quickly and avoid re-scheduling the pod because this consumes significant +amount of time and resource. Restarting the failed container "in-place" is critical +for a better utilization of compute resources. + +Since AI/ML training workloads are often declared as Job, with [PodFailurePolicy](https://kubernetes.io/docs/tasks/job/pod-failure-policy/), +some errors should be treated as retriable and restart the container in-place +by the kubelet, without re-creating and re-scheduling the Pod. + +See https://github.com/kubernetes-sigs/jobset/issues/876 for the detailed description. + +#### Story 2 + +As a Kubernetes orchestrator, I would like to differentiate between retriable errors +via container exit codes. I would like to restart the container in-place if the error +is retriable to avoid re-scheduling the Pod for shorter pod/container restart time, +and I would like to reschedule the Pod if the error is not retriable. + +### Notes/Constraints/Caveats (Optional) + + + +#### Container Restart Count + +The container level restart policy may restart the container differently from +the existing assumptions on pod level restart policy. Pod with `restartPolicy=Never` +may have containers restarted and have the restart count higher than 0. + +#### Job PodFailurePolicy + +This will not affect how Job `podFailurePolicy` interacts with pod failures, +because container-level restart will not be considered as Pod termination. +The Job controller only checks and evaluates `podFailurePolicy` after a Pod +is terminated, as mentioned in the [KEP-3329](https://github.com/kubernetes/enhancements/blob/fc234c10fe886bb3dcdf3499ae822e0650fb1179/keps/sig-apps/3329-retriable-and-non-retriable-failures/README.md?plain=1#L902-L903). + +This KEP is informed from the discussion about some future improvements +we may need to implement as described here for JobSet: +https://github.com/kubernetes/enhancements/issues/3329#issuecomment-1571643421. +Instead of making Job controller to handle container-level restart and failures, +kubelet if more suitable to handle the container restart policy. This aligns +with the current implementation that Job controller only restart / reschedule +the Pod after it is terminated, and delegate the rest to the kubelet. + +This enables efficient setups to accelerate container restart and to improve +resource utilization. For example,Jobs configured with `podFailurePolicy` for +hardware failures (Pod needs to be rescheduled to other nodes), and containers +configured with `restartPolicy` to restart in-place for training errors. + +### Risks and Mitigations + + + +#### Unintended Restart Loops + +A container might persistently exit with an "Restart" exit code due to +an unresolvable underlying issue, leading to frequent restarts that consume +node resources and potentially mask the problem. + +The container restart will still follow the exponential backoff to avoid +excessive resource consumption due to restarts. + +## Design Details + + + +The proposal is to extend the Pod's [ContainerAPI](https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go#L2528), +especially the ContainerRestartPolicy with a new possible value of "RuleBased", and +introduce a new type of ContainerRestartRules with only one allowed Action, "Restart. + +The proposed API is as following: + +```go +type ContainerRestartPolicy string + +const ( + ContainerRestartPolicyAlways ContainerRestartPolicy = "Always" + // New "RuleBased" ContainerRestartPolicy + ContainerRestartPolicyRuleBased ContainerRestartPolicy = "RuleBased" +) + +type Container struct { + // Omitting irrelevant fields... + + // Currently only "Always" is allowed for init containers; no value + // is allowed for normal containers. With this KEP, "RuleBased" will + // be allowed for both normal and init containers. The Container + // Restart Policy will override the Pod's restart policy. + RestartPolicy *ContainerRestartPolicy + + // Can only be specified if the RestartPolicy is not "RuleBased". + // Represents a list of rules to be checked to determine if the + // container should be restarted on exit. The rules are evaluated in + // order. Once a rule matches a container exit condition, the remaining + // rules are ignored. If no rule matches the container exit condition, + // the Pod-level restart policy determines the whether the container + // is restarted or not. + RestartRules []ContainerRestartRule +} + +// ContainerRestartRule describes how a container exit is handled. +type ContainerRestartRule struct { + // Specifies the action taken on a container exit if the requirements + // are satisfied. The only possible value is "Restart" to restart the + // container. + Action *ContainerRestartRuleAction + + // Represents the requirement on the container exit codes. + OnExitCodes ContainerRestartRuleOnExitCodesRequirement +} + +type ContainerRestartRuleAction string + +const ( + ContainerRestartRuleActionRestart ContainerRestartRuleAction = "Restart" +) + +// ContainerRestartRuleOnExitCodesRequirement describes the requirement +// for handling an exited container based on its exit codes. +type ContainerRestartRuleOnExitCodesRequirement struct { + // Represents the relationship between the container exit code(s) and the + // specified values. Possible values are: + // + // - In: the requirement is satisfied if the container exit code is in the + // set of specified values. + // - NotIn: the requirement is satisfied if the container exit code is + // not in the set of specified values. + Operator ContainerRestartRuleOnExitCodesOperator + + // Specifies the set of values to check for container exit codes. + // At most 255 elements are allowed. + Values []int32 +} + +type ContainerRestartRuleOnExitCodesOperator string + +const ( + ContainerRestartRuleOnExitCodesOpIn ContainerRestartRuleOnExitCodesOperator = "In" + ContainerRestartRuleOnExitCodesOpNotIn ContainerRestartRuleOnExitCodesOperator = "NotIn" +) +``` + +To specify a container to only restart with an exit code of 42, it +can be specified as following in a Pod manifest: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: my-pod +spec: + containers: + - name: my-container + image: nginx:latest + ports: + - containerPort: 80 + restartPolicy: RuleBased + restartRules: + - action: Restart + onExitCodes: + operator: In + values: [42] + restartPolicy: Never + +``` + +The proposal is to support the following combinations: + +- The action can only be `Restart`; +- Only `onExitCodes` rules are allowed, no other conditions; +- The `operator` can be either `In` or `NotIn`; +- Values only support an array of integers and no wildcard. + +With the limitations above, an API will do nothing for pods with +pod-level `restartPolicy=Always`, as the only container-level restart +rules action is `Restart`. Same for the pods with pod-level +`restartPolicy=OnFailure`. Except that exit code 0 can be configured +to be restartable, which is effectively the same as `restartPolicy=Always`. + +For the pod with the `restartPolicy=Never`, it will allow restarting +the container for the subset of exit codes. The sync and restart logic +will be implemented in k8s.io/kubelet/container. + +This API change is only intended to restart the container if the container +itself exited with the given list of exit codes. It is not intended to change +the behavior of other means that lead to container being restarted, for +example, pod resize or pod restart. + +See more discussion on how this API interacts with other components like Job +controller in [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional). + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- `k8s.io/apis/core`: `` - `` +- `k8s.io/apis/core/v1/validations`: `` - `` +- `k8s.io/features`: +- `k8s.io/kubelet`: +- `k8s.io/kubelet/container`: + +##### Integration tests + + + + + +Unit and E2E tests provide sufficient coverage for the feature. Integration tests may be added to cover any gaps that are discovered in the future. + +##### e2e tests + + + +- Verify that containers can specify restartPolicy. +- Verify that containers exited with exit codes specified in the restart +policy are restarted and the pod keeps running. +- Verify that containers exited with exit codes not specified in the restart +policy are not restarted and the pod fails. + +### Graduation Criteria + + + +#### Alpha + +- Container restart policy added to the API. +- Container restart policy implemented behind a feature flag. +- Initial e2e tests completed and enabled. + +#### Beta + +- Container restart policy functionality running behind feature flag +for at least one release. +- Container restart policy runs well with Job controller. + +#### GA + +- No major bugs reported for three months. +- User feedback (ideally from at least two distinct users) is green. + +### Upgrade / Downgrade Strategy + + + +API server should be upgraded before Kubelets. Kubelets should be downgraded before the API server. + +### Version Skew Strategy + + + +Previous kubelet client unaware of the container restart policy will ignore +this field and keep the existing behavior determined by pod's restart policy. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [ ] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - Components depending on the feature gate: +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? + +###### Does enabling the feature change any default behavior? + + + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +###### What happens if we reenable the feature if it was previously rolled back? + +###### Are there any tests for feature enablement/disablement? + + + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +###### Will enabling / using this feature result in introducing new API types? + + + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + + +### Wrapping entrypoint + +One way to implement this KEP as a DIY solution is to wrap the entrypoint +of the container with the program that will implement this exit code handling +policy. This solution does not scale well as it needs to be working on multiple +Operating Systems across many images. So it is hard to implement universally. + +### Non-declarative (callbacks based) restart policy + +An alternative to the declarative failure policy is an approach that allows +containers to dynamically decide their faith. For example, a callback is called +on an “orchestration container” in a Pod when any other container has failed. +And the “orchestration container” may decide the fate of this container - restart +or keep as failed. + +This may be a possibility long term, but even then, both approaches can work +in conjunction. + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-node/5307-container-restart-policy/kep.yaml b/keps/sig-node/5307-container-restart-policy/kep.yaml new file mode 100644 index 00000000000..417c0c6a6dd --- /dev/null +++ b/keps/sig-node/5307-container-restart-policy/kep.yaml @@ -0,0 +1,44 @@ +title: Container Restart Policy +kep-number: 5307 +authors: + - "@yuanwang04" + - "@SergeyKanzhelev" +owning-sig: sig-node +participating-sigs: +status: provisional +creation-date: 2025-05-16 +reviewers: + - "@SergeyKanzhelev" +approvers: + - "@dchen1107" # SIG Node approval + +see-also: + - "/keps/prod-readiness/sig-apps/3329" + +# The target maturity stage in the current dev cycle for this KEP. +# If the purpose of this KEP is to deprecate a user-visible feature +# and a Deprecated feature gates are added, they should be deprecated|disabled|removed. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.34" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.34" + beta: "v1.35" + stable: "v1.36" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: ContainerRestartPolicy + components: + - kubelet +disable-supported: true + +# The following PRR answers are required at beta release +metrics: +