state machine : Dynamic passing of bucket and key to distributed map

[AmylDV]

Describe the feature

Currently there is no means of passing a bucket and key dynamically to a distributed Map state using CDK. This functionality is available in the states language using JSONPath along the lines of

"ItemReader": {
"Resource": "arn:aws:states:::s3:getObject",
"ReaderConfig": {
"InputType": "JSON"
},
"Parameters": {
"Bucket.$": "$.Payload.bucket",
"Key.$": "$.Payload.key"
}
}

Use Case

I want to define my state machine using CDK rather than the states language

Proposed Solution

Create an IItemReader that can take a string in place of a bucket.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.128.0

Environment details (OS name and version, etc.)

MacOs Sonoma

[pahud]

Thank you for the feature request. We welcome and appreciate your PR for this.

[JannikWempe]

I faced the same issue and still wanted to use the DistributedMap class instead of using CustomState.

I came up with a dirty, hacky workaround and extended the S3CsvItemReader.

export interface ExtendedS3CsvItemReaderProps extends sfn.S3CsvItemReaderProps {
	readonly "key.$"?: string;
}

class ExtendedS3CsvItemReader extends sfn.S3CsvItemReader {
	readonly dynamicKey: string | undefined;

	constructor(props: ExtendedS3CsvItemReaderProps) {
		super(props);
		this.dynamicKey = props["key.$"];
	}

	public render() {
		const rendered = super.render();

		if (this.dynamicKey) {
			// ignores the provided `key` and uses `key.$` instead
			rendered.Parameters = {
				Bucket: rendered.Parameters.Bucket,
				"Key.$": this.dynamicKey,
			};
		}

		return rendered;
	}
}

You can use it just like the existing S3CsvItemReader but pass in a "key.$" prop (you still have to pass a key but that will be ignored). You could do the same with Bucket.$.

I may dig deeper and provide a PR (with a polished implementation of course 😜).

[ChakshuGupta13]

Hi @pahud, can you please assign this issue to me?
(Discussed one possible solution with @GavinZZ and raised above PR for proposed initial changes.)

[GavinZZ]

Assigned this issue to @ChakshuGupta13, I will try to review your PR by end of this month. If I don't, feel free to ping me directly.

[ChakshuGupta13]

FYI: We can still currently utilise dynamic parameters except bucket:

const myBucket = new Bucket(stack, 'MyBucket', ...);
const distributedMap = new sfn.DistributedMap(this, 'DistributedMap', {
    itemReader: new sfn.S3ObjectsItemReader({
      bucket: myBucket,
      prefix: JsonPath.stringAt('$.prefix'),
    }),
  });
  distributedMap.itemProcessor(new sfn.Pass(this, 'Pass'));

This code snippet produces following cdk.out template:

...
"Parameters\":{\"Bucket\":\"",
       {
        "Ref": "MyBucket..."
       },
       "\",\"Prefix.$\":\"$.prefix\"}}}}}"
      ]
...

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.