Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Feature Request: Configurable minimumHealthyPercent and maximumHealthyPercent for ECS Service Hotswap #29618

Closed
1 of 2 tasks
rexrafa opened this issue Mar 26, 2024 · 3 comments
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2

Comments

@rexrafa
Copy link

rexrafa commented Mar 26, 2024

Describe the feature

Currently, when using the --hotswap flag for deploying ECS services, the minimumHealthyPercent is hardcoded to 0, which can be disruptive for services with interdependencies. This can result in downtime as the current tasks are stopped before the new ones are fully up and running. To improve the flexibility of hotswap deployments, allowing users to set the minimumHealthyPercent and maximumHealthyPercent parameters would be beneficial.

Use Case

We aim to speed up deployment times in our development environment by using the hotswap feature. However, our services have dependencies on each other, and the current hotswap behavior is too disruptive. We want to keep the current tasks running until the new ones are fully operational. Configuring minimumHealthyPercent and maximumHealthyPercent would give us the control needed to ensure smoother transitions during deployments.

Proposed Solution

Modify the ECS service hotswap implementation to accept minimumHealthyPercent and maximumHealthyPercent as configurable parameters, with default values of 0 and 200, respectively. These parameters should be exposed in the CDK, allowing users to override them as needed.

For example, the current implementation at:

It could be modified to accept these parameters from the user's CDK stack configuration.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.117.0

Environment details (OS name and version, etc.)

Darwin Kernel Version 23.3.0

@rexrafa rexrafa added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Mar 26, 2024
@github-actions github-actions bot added the @aws-cdk/aws-ecs Related to Amazon Elastic Container label Mar 26, 2024
@tim-finnigan tim-finnigan self-assigned this Mar 26, 2024
@tim-finnigan tim-finnigan added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Mar 26, 2024
@tim-finnigan
Copy link

Thanks for the feature request. I saw some related discussion on this topic and @pahud's comment here: #22241 (comment)

I am not sure if this is a good default value but I believe if you run an ecs service with just single task with maximumHealthyPercent=100 and you need force the task update, you probably have to set the minimumHealthyPercent to 0 otherwise the only task might not be replaced unless maximumHealthyPercent is set to 200. However, with maximumHealthyPercent=200 the new deployment will have to bring up an additional healthy task before it can terminate the original one. If the additional one is not in healthy state min>0 max>100 might not be able to replace the only task. With min=0, the only task will be immediately tore down and replaced.

As cdk hot swap is designed for development only I believe setting min=0 makes sense here.

I think this request needs some further review and discussion, would be interested to get your thoughts on that comment.

@tim-finnigan tim-finnigan added p2 response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. effort/medium Medium work item – several days of effort and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Mar 26, 2024
@tim-finnigan tim-finnigan removed their assignment Mar 26, 2024
@dysosmus
Copy link

@tim-finnigan , answering for my co-worker @rexrafa as he is OoO.

We mostly agree with @pahud's comment that min=0 is a good default and shouldn't be changed as the default value.
Our opinion is that allowing the user to customize that value will enable them to determine the exact trade-off between speed, safety, and cost they desire in their development process. It also helps to better account for development environments where ECS Services run more than a single task.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 28, 2024
mergify bot pushed a commit that referenced this issue Oct 23, 2024
### Issue #29618

### Reason for this change

We aim to speed up deployment times in our development environment by using the hotswap feature. However, our services have dependencies on each other, and the current hotswap behavior is too disruptive. 

### Description of changes

We modified the hotswap implementation for ECS services to pass the `minimumHealthyPercent` and `maximumHealthyPercent` configurable parameters. These parameters are exposed to the cli and can be passed as `--hotswap-ecs-minimum-healthy-percent <number>` and `--hotswap-ecs-maximum-healthy-percent <number>`

The implementation is careful to maintain the existing behaviour. That is, if none of the new flags is used, the current `minimumHealthyPercent = 0` and  `maximumHealthyPercent = undefined` values are used.

### Description of how you validated changes

We added a unit test validating that the correct values are passed to the task definition. We also executed using the locally built version of cdk validating that the behavior is as expected: the parameters are respected during hotswap deployments, and the existing API is maintained.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@rexrafa rexrafa closed this as completed Nov 18, 2024
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 18, 2024
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2
Projects
None yet
Development

No branches or pull requests

3 participants