ECS: Task fails to start

[Smotrov]

Describe the bug

I used to deploy my AWS ECS code with CDK with no problem.
All of a sudden yesterday all tasks couldn't start. It shows Stopped status with following message on hover CannotPullContainerError: unsupported media type application/vnd.in-toto+json.

Is it fixable somehow? Now could not deploy anything.

Expected Behavior

Proper deployment

Current Behavior

Can't deploy outdates on ECS

Reproduction Steps

Hard to reproduce

Possible Solution

No response

Additional Information/Context

Before this deployment cdk deploy --all randomly fails with following error

failed commit on ref "manifest-sha256:851e5bb7dcd97948bd32720606df510feebee3ee9404e4db5e1ed00529803920": unexpected status from PUT request to https://408064982279.dkr.ecr.eu-west-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-408064982279-eu-west-1/manifests/2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024: 400 Bad Request
EStreamingProcessor:  fail: docker push 408064982279.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-408064982279-eu-west-1:2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024 exited with error code 1: failed commit on ref "manifest-sha256:851e5bb7dcd97948bd32720606df510feebee3ee9404e4db5e1ed00529803920": unexpected status from PUT request to https://408064982279.dkr.ecr.eu-west-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-408064982279-eu-west-1/manifests/2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024: 400 Bad Request

 ❌ Deployment failed: Error: Failed to publish asset 2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024:408064982279-eu-west-1
    at Deployments.publishSingleAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:11645)
    at async Object.publishAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:197644)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181545
[18:47:26] Reading cached notices from /Users/smotrov/.cdk/cache/notices.json

Failed to publish asset 2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024:408064982279-eu-west-1
[18:47:26] Error: Failed to publish asset 2974a6651b65171af5a4efae98ae091a3faf4ef209d848d6a5db6ec4af5c0024:408064982279-eu-west-1
    at Deployments.publishSingleAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:11645)
    at async Object.publishAsset (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:197644)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181545

Right after this cdk deploy --all has no errors except not starting tasks.

CDK CLI Version

2.147.1 (build d3695d4)

Framework Version

No response

Node.js Version

v18.20.2

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[khushail]

@Smotrov , thanks for reporting this. Did anything change from your side between successful deployment to failed one ? I mean any packages update or in the code ? Also you mentioned that it is not reproducible so that means it occurred one time ? Could you please share the synthesized template and logs ??

[khushail]

normally this error "CannotPullContainerError" would occur in these circumstances as mentioned here in this doc -
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_cannot_pull_image.html

This sage post might be helpful in further understanding as to why you are getting the error - https://repost.aws/knowledge-center/ecs-task-stopped

Let me know if this is helpful.

[Smotrov]

Thank you fixed with Docker desktop reset / reinstall.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.