Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(step-functions): add bucketNamePath in item reader (#31619)
### Issue # (if applicable) Closes #29409. ### Reason for this change - For `DistributedMap` state of StepFunctions, `IItemReader` currently only allows S3 bucket as input source to be declared statically in CDK. - In other words, current CDK implementation only caters to static use-case where we know either `bucket` or `bucketName` (from which we can create `IBucket`) and pass it to `IItemReader`. - Whereas via AWS management console, if we create `DistributedMap` manually, then we can also convey S3 source dynamically using State Input / JsonPath. - In other words, for dynamic use-case, we will neither have `bucket` nor `bucketName` i.e. we only know state input variable which will convey `bucketName` e.g. `$.bucketName`. - So, if we want to use `IItemReader` for dynamic use case also, then, to avoid making breaking change (e.g. changing type of `bucket` from `IBucket` to `string`), we will: - (1) need to make `bucket: IBucket` an optional prop in `ItemReaderProps` (refer [Making properties optional](https://github.com/aws/jsii/blob/main/packages/jsii-diff/BREAKING_CHANGES.md#making-properties-optional)) - (2) and add another optional field `bucketNamePath: string` to convey state input variable name (e.g. $.bucketName) ### Description of changes - Add `bucketNamePath` as optional prop in `ItemReaderProps`. - Make `bucket` an optional prop instead of required prop in `ItemReaderProps`. - Adapt implementing classes of `IItemReader` to handle `bucket` being optional (refer [Making properties optional](https://github.com/aws/jsii/blob/main/packages/jsii-diff/BREAKING_CHANGES.md#making-properties-optional)). - Add `validateItemReader` in `IItemReader` which implementing classes will implement to handle mutual exclusivity of `bucket` and `bucketNamePath`. - Modify `DistributedMap:validateState` to validate `IItemReader` if present. - Modify README to explain and add examples relevant to changes. - Add new unit-tests for `DistributedMap`. - Add new integration tests to validate deployment and expected result. ### Description of how you validated changes - Build changes: `cd ./packages/aws-cdk-lib/ && yarn build aws-stepfunctions && yarn watch` - Add and run new unit-tests for `DistributedMap`: `yarn test aws-stepfunctions/test` - Run `yarn rossetta:extract -d aws-stepfunctions -v` to confirm validity of README changes - Add new integration test and run (with snapshot created): - Build once and watch: `npx lerna run build --scope=@aws-cdk-testing/framework-integ && yarn watch` - Test: `yarn build --directory test/aws-stepfunctions/test && yarn integ test/aws-stepfunctions/test/integ.item-reader-path-s3-object.js --update-on-failed` - Verified expected step function execution result during snapshot creation ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
- Loading branch information