Skip to content

Commit

Permalink
feat(step-functions): add bucketNamePath in item reader (#31619)
Browse files Browse the repository at this point in the history
### Issue # (if applicable)

Closes #29409.

### Reason for this change

- For `DistributedMap` state of StepFunctions, `IItemReader` currently only allows S3 bucket as input source to be declared statically in CDK.
- In other words, current CDK implementation only caters to static use-case where we know either `bucket` or `bucketName` (from which we can create `IBucket`) and pass it to `IItemReader`.
- Whereas via AWS management console, if we create `DistributedMap` manually, then we can also convey S3 source dynamically using State Input / JsonPath.
- In other words, for dynamic use-case, we will neither have `bucket` nor `bucketName` i.e. we only know state input variable which will convey `bucketName` e.g. `$.bucketName`.
- So, if we want to use `IItemReader` for dynamic use case also, then, to avoid making breaking change (e.g. changing type of `bucket` from `IBucket` to `string`), we will:
  - (1) need to make `bucket: IBucket` an optional prop in `ItemReaderProps` (refer [Making properties optional](https://github.com/aws/jsii/blob/main/packages/jsii-diff/BREAKING_CHANGES.md#making-properties-optional))
  - (2) and add another optional field `bucketNamePath: string` to convey state input variable name (e.g. $.bucketName)

### Description of changes

- Add `bucketNamePath` as optional prop in `ItemReaderProps`.
- Make `bucket` an optional prop instead of required prop in `ItemReaderProps`.
- Adapt implementing classes of `IItemReader` to handle `bucket` being optional (refer [Making properties optional](https://github.com/aws/jsii/blob/main/packages/jsii-diff/BREAKING_CHANGES.md#making-properties-optional)).
- Add `validateItemReader` in `IItemReader` which implementing classes will implement to handle mutual exclusivity of `bucket` and `bucketNamePath`.
- Modify `DistributedMap:validateState` to validate `IItemReader` if present.
- Modify README to explain and add examples relevant to changes.
- Add new unit-tests for `DistributedMap`.
- Add new integration tests to validate deployment and expected result.

### Description of how you validated changes

- Build changes: `cd ./packages/aws-cdk-lib/ && yarn build aws-stepfunctions && yarn watch`
- Add and run new unit-tests for `DistributedMap`: `yarn test aws-stepfunctions/test`
- Run `yarn rossetta:extract -d aws-stepfunctions -v` to confirm validity of README changes
- Add new integration test and run (with snapshot created):
  - Build once and watch: `npx lerna run build --scope=@aws-cdk-testing/framework-integ && yarn watch`
  - Test: `yarn build --directory test/aws-stepfunctions/test && yarn integ test/aws-stepfunctions/test/integ.item-reader-path-s3-object.js --update-on-failed`
  - Verified expected step function execution result during snapshot creation

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
ChakshuGupta13 authored Oct 14, 2024
1 parent f4f8abc commit 97130d8
Show file tree
Hide file tree
Showing 26 changed files with 76,786 additions and 19 deletions.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 97130d8

Please # to comment.