(stepfunctions): itemSelector keys with null value are omitted

[anentropic]

Describe the bug

when configuring a Map state, the key with null value is omitted from the input data passed to the item processor

                    sfn.Map(
                        self,
                        "MyMap",
                        max_concurrency=self._max_concurrency,
                        items_path="$.partitioned.job_names",
                        item_selector={
                            "batch_size.$": "$.batch_size",
                            "job_name.$": "$$.Map.Item.Value",
                            "max_batches": None,
                        },
                        result_path="$.partition_results",
                    )

I am confused if this is intended or not

On the one hand sfn.Condition.is_null(...) exists, so it seems intended to be possible to deal with keys having null values?

On the other hand the "Data flow simulator" tool returns an error (No results for path: '[object Object]') if key has a null value:

unclear if that means it's not supported or it's just a bug in the simulator

it is quite a common issue in JSON de/serializer code that the case of "key present but value is null" is conflated with "key not present" and null keys are omitted by default

however if it's not possible to pass keys with null values this complicates things in Step Functions particularly because many of the various methods with JSONPath args will error if the key is not present

leaving only the less satisfactory option of using some not-null value to mean null

Expected Behavior

all explicitly specified keys are propagated to the input

Current Behavior

keys with null value are omitted

Reproduction Steps

see above

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.133.0 (build dcc1e75)

Framework Version

No response

Node.js Version

v18.18.0

OS

macOS 14.3.1

Language

Python

Language Version

3.11.5

Other information

No response

[anentropic]

I see now this bug has been turned into a mis-feature and I should use a workaround like sfn.JsonPath.string_to_json("null") where I want null values

[pahud]

Yes. Does this workraound work for you?

[anentropic]

yes it does

[pahud]

Closing this issue as the workaround is valid.

Feel free to reopen if there's still a concern.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.