AwsCustomResource: Return response field from lambda backed custom resource

[manosntoulias]

Describe the bug

I'am trying to create a lambda backed custom resource and get a field from its response but I get an error that the Vendor response doesn't contain such key

I thought that the keys in the response of the lambda function must match the string parameter of custom_resource::.get_response_field which in that case would be "info.current". What am I doing wrong? You can find the a simple version of the code below

Expected Behavior

Get "info.current" as an output after deployment finishes

Current Behavior

CustomResource attribute error: Vendor response doesn't contain info.current key in object arn:aws:cloudformation:eu-west-1:238091120948:stack/First/25356620-b1bb-11ec-b3e5-0299088d230d|DemoCustomResourceDemo0225F457|9f3
a0e9b-52f9-45ab-9cba-0823e5a4301b in S3 bucket cloudformation-custom-resource-storage-euwest1. Rollback requested by user.

Reproduction Steps

from aws_cdk.aws_logs import RetentionDays
from constructs import Construct
from aws_cdk import (Stack,
                     aws_iam as iam,
                     aws_lambda as l,
                     CfnOutput,
                     Duration,
                     custom_resources as cr
                     )


class DemoConstruct(Construct):

    def __init__(self, construct: Construct, construct_id: str):
        super().__init__(construct, construct_id)
        with open(f"./resources/lambdas/pathsLambda.py") as lambda_path:
            code = lambda_path.read()
        demo_lambda_function = l.Function(
            scope=self,
            id=f"DemoLambda-{construct_id}",
            code=l.Code.from_inline(code),
            environment={
                "SOME_ARG": "argVal"
            },
            handler="index.lambda_handler",
            runtime=l.Runtime.PYTHON_3_7,
            timeout=Duration.seconds(60),
            memory_size=2048
        )
        c_resource = cr.AwsCustomResource(
            self,
            f"CustomResource-{construct_id}",
            log_retention=RetentionDays.ONE_WEEK,
            on_update=cr.AwsSdkCall(
                service="Lambda",
                action="invoke",
                physical_resource_id=cr.PhysicalResourceId.of("Trigger"),
                parameters={
                    "FunctionName": demo_lambda_function.function_name,
                    "InvocationType": "RequestResponse",
                    "Payload": "{\"Request\":{\"Method\": \"method1\"}}"
                },
                # output_paths=["info.current"]
            ),
            policy=cr.AwsCustomResourcePolicy.from_statements([
                iam.PolicyStatement(
                    actions=["lambda:InvokeFunction"],
                    resources=[demo_lambda_function.function_arn]
                )
            ])
        )
        CfnOutput(self, id="InfoCurrent", value=c_resource.get_response_field("info.current"), export_name="InfoCurrent")


class Demo(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
        DemoConstruct(self, "Demo")

And the lambda code:

def lambda_handler(event, context):

    print(event)
    print(context)

    myStr = "q"
    #for i in range(4096):
    #    myStr += 'q'
    my_dict = {
        "info": {
            "current": {
                    "version": 3
            },
            "applied": {
                "count": 23
            }
        },
        "other": myStr
    }
    print(my_dict)
    return my_dict

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.10

Framework Version

No response

Node.js Version

v16.3.0

OS

Mac

Language

Python

Language Version

3.7

Other information

No response

[NGL321]

Hey @manosntoulias,

I have converted this to a discussion since there probably isn't a bug here (since I have been able to successfully create custom resources with responses), but rather a complication/error in the process. If this does turn out to be a bug, I will convert back to an issue.

In the interim, I have verified this behavior and am investigating the probably cause and will update with an answer ASAP (best guess atm is that it has to do with a different structure for custom resource responses vs basic lambda responses)!

[peterwoodworth]

I'm copying my answer from the last thread since this seems to be a duplicate

The output_paths parameter and getResponseField method both work with the Api response.
For example, this custom resource is using the S3 listobjectsv2 API call. If we take a look at that call's response syntax, we can see how the response data will be returned. In this case, the example wants to only return the data seen in the first two objects in Contents, which is an array of Objects.

So for your case, you're using Lambda Invoke. I'm not too sure what info.current is, but it's not in any of the response elements seen here 🙂

Hope this helps guide you in the right direction!

[manosntoulias]

Hello again @peterwoodworth, I looked carefully at your answer and you are right when specifying "Payload" I get the response in the output:

However it seems that the value of 'Payload' is stringified and cannot access paths like "Payload.info" as I get the same error.

Tried changing my lambda response to this without success:

my_dict = { "statusCode": 200, "headers": { "Content-Type": "application/json" }, "body": { "info": { "current": { "version": 3 }, "applied": { "count": 23 } }, "other": myStr } }

Im also trying to use CustomResource instead of AwsCustomResource, but since it does not use lambda invoke even just specifying "Payload" fails. Maybe the response structure for this as well will help.

Thanks again.

[manosntoulias]

@peterwoodworth this does not work, because the json.loads gets called before deployment without the actual response (i.e., I tried it and fails at synthesis time)

But event if it did work, it would not help me as I need a way to filter the custom response and make it less than the 4096Byte limit. I just thought that if I could find the correct arg for the c_resource.get_response_field I could use the same arg for the output_paths and solve my issue.

In any case, maybe you have another idea for the 'CustomResource' class that I am using instead in my other comment

[peterwoodworth]

Are you trying to parse the string as JSON in the same stack? I believe you should be able to CfnInput the string which you're already outputting in the CfnOutput in another stack and be able to handle the parsing there

[manosntoulias]

Yes in the same stack. I could try this but It would require the first stack to fail if the second fails during deployment due to not having the correct response in the parsed json

[peterwoodworth]

Ah I see the issue. If you need the same stack to fail on deployment, based on the response from the custom resource (which you won't be able to parse in your construct code, since that is run at synth time), then I think the best solution here is to use a Trigger, which allows you to execute code during deployment time.

This is a pretty new module, and I've never messed around with it personally. But it seems pretty straightforward to me based on the readme! Let me know if you have any questions or concerns about this

[manosntoulias]

Sorry didn't have the time to work on this earlier.

Apart from the Trigger, I could use a second custom Resource that does the parsing and checks for the contents of the json.

But, it seems to me that both of those ways are way too complicated for something this simple. Am I right in saying that the custom Resources are implemented by having flat json responses in mind and not having nested json like mine? Because it is fairly simple to access strings or integers at the top level of the json (through Token.asString/Token.asInteger) and not so simple to access inner strings/integers inside nested json (I though that Token.asAny would work but without success).

Due to this, I will alter the response of my lambda function and make it flat json. It was an already build version of someone else's public repo and that is why I reserved this as a last option

[manosntoulias]

I have more success with CustomResource. Changed the lambda response to this:

`

{

    "PhysicalResourceId": "AAAAAAAAAABBBBBBBBBB",
    "Data": {
        "info": {
            "current": {
                    "version": 3
            },
            "applied": {
                "count": 23
            }
        },
        "other": myStr
    }
}`

and now I am able to do c_resource.get_att_string("other"). However still fail to get "info" as its value is not a string.

I have a hard time using c_resource.get_att instead. I cannot find a way to turn this to a string and have it in the output

[peterwoodworth]

Under the hood, AwsCustomResource actually just creates a CustomResource

aws-cdk/packages/@aws-cdk/custom-resources/lib/aws-custom-resource/aws-custom-resource.ts

Line 414 in 9487b39

this.customResource = new cdk.CustomResource(this, 'Resource', {

And get_response_field actually just uses that custom resource's get_att_string() method.

aws-cdk/packages/@aws-cdk/custom-resources/lib/aws-custom-resource/aws-custom-resource.ts

Lines 461 to 463 in 9487b39

	public getResponseField(dataPath: string): string {
	AwsCustomResource.breakIgnoreErrorsCircuit([this.props.onCreate, this.props.onUpdate], 'getDataString');
	return this.customResource.getAttString(dataPath);

So, which one you end up using doesn't actually matter 🙂

[manosntoulias]

Ahh ok, that makes sense. Thanks!

I think it is only a matter of figuring out how to parse the return value of getResponseFieldReference/GetAtt then. The docs you linked say to use the 'Token.asXxx' methods but I have not managed to do it. I suppose I should use Token.asAny but I have not figured out how yet

[peterwoodworth]

I don't think you'd want to use tokens, because you still need to run the code and interpret the response due to your requirement of needing the same stack to fail. This would need to happen at synth time, because that's when the code runs. After that, it's up to CloudFormation.

But not entirely, see my comment in the other thread on Triggers for a way to execute code while your stack is deploying

[muhammadali2022skipq]

I'm also trying to do something like this and faced with a similar error let me know if you have found a solution.

[daknhh]

The only thing you can do right now is using Cfn constructs sometimes. I also have the problem with a CustomResource which is calculating the Cidr for VPC and Subnets. For VPC i have to use the ec2nCfnVPC construct but for my Subnets i can use the ec2.PrivateSubnet construct

[manosntoulias]

@muhammadali2022skipq and everyone else who might be interested,

I have:

• alternated the source code of the lambda function to return a flattened response instead of having nested json objects. Also made the response having a smaller (< 4096) fixed size
• switched to CustomResource and provider framework instead of AwsCustomResource. You can use get_att_string to get the first level of the response (hence the flattening). With AwsCustomResource it is not possible because the first level are the RESPONSE ELEMENTS of the lambda invoke api. This means you can only get the whole Payload field in string format and not the individual fields inside payload.

Note: All these were done because with the accessor methods (e.g. get_att_string) I can only access the first level of the response in integer or string format. Nested json objects are returned as string and I can not access their fields. If someone knows how to access nested fields, feel free to comment

[itsmearunsgit]

@manosntoulias you have any solution for the nested json object access ? I am facing the same issue when using AwsCustomResources and failed to get netsted object using "custom_resource.get_response_field('Response["ReplicaCount"]')"

I am getting below error:

MyEksClusterStack | 1:04:59 pm | CREATE_FAILED | Custom::AWSCDK-EKS-HelmChart | EKSWelcome/chart-NginxIngress/Resource/Default (EKSWelcomechartNginxIngress8BF6ECAB) CustomResource attribute error: Vendor response doesn't contain Response.ReplicaCount attribute in object

Let me know if you have solutions

[BwL1289]

Another option is to use CustomResource without a Provider:

self._create_thumbprint_custom_resource = CustomResource(
            self,
            f"{id}CustomResource",
            service_token=self._thumbprint_lambda_function_arn,
            properties=payload,
        )

# Get the thumbprint from the response
self._thumbprint = self._create_thumbprint_custom_resource.get_att(
    attribute_name="thumbprint"
).to_string()

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.

[jacido]

Hello, I'm trying to provide custom data as input to the event so that the Lambda can function differently based on that input.
Eventbridge rule will be watching for matching events and trigger the lambda function when the rule is matched.

metricsRule.addTarget(new LambdaTarget(sendNotificationsLambda), {
event: RuleTargetInput.fromObject({
testDataViews: config.testDataViews,
})
});

In tried to use input and event, but the custom data generated by the event was not passed to the lambda function.

For example
eventId: balba
responseData: {instances: [i-123, i-456]}

When the lambda runs, it’s not receiving the instances to show the message:

"The following instances were launched: [i-123, i-456]"

Is there a way to substantiate the custom resource to pass to the Lambda Function?

When we use an Event Bridge Rule and capture the event to go to the Lambda, we can have the Lambda process the event and manipulate it however we want.