aws-stepfunctions: add keyPath parameter to the S3JsonItemReader constructor

[Lykan9999]

Describe the feature

add an optional keyPath parameter to the S3JsonItemReader constructor, that would allow for more dynamic S3 object fetching. Right now the key field only accepts a hard-coded string values. and we can't specify an object key that might be generated dynamically from a previous step

Use Case

DistributedMaps are great for orchestrating iterable tasks on big data, imagine a flow where the first step is a data fetching task that reads a huge dataset from a data vendor and writes a JSON dump of the responses in a S3 bucket, then you want to iterate over each of the responses in a DistributedMap, the data fetching service can generate the objectKey dynamically and set it to for example $.data.objectKey the the DistributedMap can reference it as follows:-

...
itemReader: new sfn.S3JsonItemReader({
    bucket: bucket,
    keyPath: `$.data.objectKey`
})
...

Proposed Solution

Add akeyPath parameter to the S3JsonItemReader constructor

Other Information

I am not sure if there is an existing way we can do this, if yes then please enlighten me and mind my ignorance.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.131.0

Environment details (OS name and version, etc.)

MacOS Sonoma 14.4.1 (23E224)

[khushail]

Thanks @Lykan9999 for reaching out. Correct me if I understood it wrong. You want to make the keys dynamic so I think best way to implement this would be to make 'key' field accept dynamic values rather than hardcoded string.
However I would leave it upto the community and team to decide on implementation.

[dreessan]

This would be useful to us as well. We'r trying to iterate over a fetched SQL table. The simplest way for Athena to iterate over rows is to store the SELECT as a .csv whose output location we cannot specify ahead of time.

[svleeuwen]

This is an essential feature. We use this in all our workflows (fetch from db -> write to S3 -> use map to iterate and process)

The key should have a unique name to prevent getting overwritten when several processes run in parallel. So we can't hard code it.

When editing state machine via AWS console you can do this and also specify bucket path.

Resulting in the configuration:

"ItemReader": {
    "Resource": "arn:aws:states:::s3:getObject",
    "ReaderConfig": {
      "InputType": "JSON"
    },
    "Parameters": {
      "Bucket.$": "$.bucketName",
      "Key.$": "$.fileKey"
    }
  },

A hard coded bucket name and dynamic key is supported as well.

[svleeuwen]

I still would like to be able to use a dynamic key.

I have a StateMachine with a lot of dynamic steps that follow each other and I have to get the file_key from the previous_step:

"ItemReader": {
    "Resource": "arn:aws:states:::s3:getObject",
    "ReaderConfig": {
        "InputType": "JSON",
    },
    "Parameters": {
        "Bucket": intermediate_bucket.bucket_name,
        "Key.$": "$.state.previous_step.result.file_key",
    },
}

Currently I'm not able to switch to this construct and have to keep using a CustomState object.