aws_cdk.aws_stepfunctions_tasks.HttpInvoke: Get API Endpoint from state input

[bc-venkata-edara]

Describe the bug

Get API Endpoint from state input is not working throwing this error.

SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition

Expected Behavior

It should create stepfunction shown in below image with below definition.

"marketo-leads-creation-update": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "Authentication": {
          "ConnectionArn": "arn:aws:events:<region>:<account>:connection/<connection>
        },
        "RequestBody.$": "$.detail.object",
        "Method.$": "$.detail.object_method",
        "ApiEndpoint.$": "States.Format('https://<endpoint>/rest/v1/{}.json', $.detail.object_name)"
      }

Current Behavior

When we add code for step function httpinvoke task cdk python. it's throwing below error.

UPDATE_FAILED        | AWS::StepFunctions::StateMachine | EXAMPLERESOURCESsfMarketosync1464C05F
Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is n
ot valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c
714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)


 ❌  EIP-Example failed: Error: The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181438

 ❌ Deployment failed: Error: The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181438

The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)

Reproduction Steps

import os
import sys  # noqa: F401
from aws_cdk import (
    Stack,
    Tags,
    aws_lambda as lambda_,
    aws_stepfunctions as sfn,
    aws_stepfunctions_tasks as tasks,
    aws_events as events,
    aws_secretsmanager as sm,
    SecretValue
)
import aws_cdk as cdk
from constructs import Construct

connection = events.Connection(self, "Connection",
            authorization=events.Authorization.basic("username", SecretValue.unsafe_plain_text("password")))

        task = tasks.HttpInvoke(self, "Invoke HTTP API",
            api_root="States.Format('<endpoint>",  # Static API root, adjust as necessary
            api_endpoint=sfn.TaskInput.from_text("rest/v1/{}.json',$.detail.object_name"), # Use the constructed URL
            body=sfn.TaskInput.from_json_path_at('$.detail.object'),
            connection=connection,
            headers=sfn.TaskInput.from_object({"Content-Type": "application/json"}),
            method=sfn.TaskInput.from_json_path_at('$.detail.object_method'),
            url_encoding_format=tasks.URLEncodingFormat.BRACKETS
        )
 
        sfMarketo_sync = sfn.StateMachine(
            self,
            "sfMarketo-sync",
            state_machine_name="sfMarketo-sync-test",
            definition_body=sfn.DefinitionBody.from_chainable(task),
            timeout=cdk.Duration.minutes(5),
            state_machine_type=sfn.StateMachineType.STANDARD,
            #role=role
        )

cdk synth
cdk deploy

Possible Solution

When we run synth or deploy on backend it creates cloud formation template in cdk.out folder. The template should add ".$" suffix to "ApiEndpoint" parameter but it's not adding. For reference see the parameter "Method" which has suffix ".$". if it add that suffix it should work.

"EXAMPLERESOURCESsfMarketosync1464C05F": {
   "Type": "AWS::StepFunctions::StateMachine",
   "Properties": {
    "DefinitionString": {
     "Fn::Join": [
      "",
      [
       "{\"StartAt\":\"Invoke HTTP API\",\"States\":{\"Invoke HTTP API\":{\"End\":true,\"Type\":\"Task\",\"Resource\":\"arn:",
       {
        "Ref": "AWS::Partition"
       },
       ":states:::http:invoke\",\"Parameters\":{\"**ApiEndpoint**\":\"States.Format('<endpoint>/rest/v1/{}.json',$.detail.object_name\",\"Authentication\":{\"ConnectionArn\":\"",
       {
        "Fn::GetAtt": [
         "EXAMPLERESOURCESConnectionF6D008B6",
         "Arn"
        ]
       },
       "\"},\"**Method.$**\":\"$.detail.object_method\",\"Headers\":{\"Content-Type\":\"application/x-www-form-urlencoded\"},\"RequestBody.$\":\"$.detail.object\",\"Transform\":{\"RequestBodyEncoding\":\"URL_ENCODED\",\"RequestEncodingOptions\":{\"ArrayFormat\":\"BRACKETS\"}}}}},\"TimeoutSeconds\":300}"
      ]
     ]
    },

Additional Information/Context

No response

CDK CLI Version

2.146.0 (build b368c78)

Framework Version

No response

Node.js Version

v22.3.0

OS

Mac OS

Language

Python

Language Version

Python 3.12.3

Other information

No response

[ashishdhingra]

Reproducible.

Below is the equivalent code in TypeScript:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as events from 'aws-cdk-lib/aws-events';
import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks';
import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
export class TypescriptStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const connection = new events.Connection(this, 'Connection', {
      authorization: events.Authorization.basic('username', cdk.SecretValue.unsafePlainText('password'))
    });

    const task = new tasks.HttpInvoke(this, "Invoke HTTP API", {
      apiRoot:"States.Format('https://api.stripe.com')",  // Static API root, adjust as necessary,
      apiEndpoint: sfn.TaskInput.fromText("rest/v1/{}.json',$.detail.object_name"), // Use the constructed URL
      body: sfn.TaskInput.fromJsonPathAt('$.detail.object'),
      connection: connection,
      headers: sfn.TaskInput.fromObject({"Content-Type": "application/json"}),
      method: sfn.TaskInput.fromJsonPathAt('$.detail.object_method'),
      urlEncodingFormat: tasks.URLEncodingFormat.BRACKETS
    });

    const sfMarketo_sync = new sfn.StateMachine(this, 'sfMarketo-sync', {
      stateMachineName: 'sfMarketo-sync-test',
      definitionBody: sfn.DefinitionBody.fromChainable(task),
      timeout: cdk.Duration.minutes(5),
      stateMachineType: sfn.StateMachineType.STANDARD
    });
  }
}

Running cdk synth produces the following output:

Resources:
  Connection07624BCD:
    Type: AWS::Events::Connection
    Properties:
      AuthParameters:
        BasicAuthParameters:
          Password: password
          Username: username
      AuthorizationType: BASIC
    Metadata:
      aws:cdk:path: TypescriptStack/Connection/Connection
  sfMarketosyncRole95111CA2:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: states.amazonaws.com
        Version: "2012-10-17"
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Role/Resource
  sfMarketosyncRoleDefaultPolicy68A30164:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: events:RetrieveConnectionCredentials
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - Connection07624BCD
                - Arn
          - Action:
              - secretsmanager:DescribeSecret
              - secretsmanager:GetSecretValue
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - Connection07624BCD
                - SecretArn
          - Action: states:InvokeHTTPEndpoint
            Condition:
              StringLike:
                states:HTTPEndpoint: States.Format('https://api.stripe.com')*
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: sfMarketosyncRoleDefaultPolicy68A30164
      Roles:
        - Ref: sfMarketosyncRole95111CA2
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Role/DefaultPolicy/Resource
  sfMarketosync8373672D:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      DefinitionString:
        Fn::Join:
          - ""
          - - '{"StartAt":"Invoke HTTP API","States":{"Invoke HTTP API":{"End":true,"Type":"Task","Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::http:invoke","Parameters":{"ApiEndpoint":"States.Format('https://api.stripe.com')/rest/v1/{}.json',$.detail.object_name","Authentication":{"ConnectionArn":"
            - Fn::GetAtt:
                - Connection07624BCD
                - Arn
            - '"},"Method.$":"$.detail.object_method","Headers":{"Content-Type":"application/x-www-form-urlencoded"},"RequestBody.$":"$.detail.object","Transform":{"RequestBodyEncoding":"URL_ENCODED","RequestEncodingOptions":{"ArrayFormat":"BRACKETS"}}}}},"TimeoutSeconds":300}'
      RoleArn:
        Fn::GetAtt:
          - sfMarketosyncRole95111CA2
          - Arn
      StateMachineName: sfMarketo-sync-test
      StateMachineType: STANDARD
    DependsOn:
      - sfMarketosyncRoleDefaultPolicy68A30164
      - sfMarketosyncRole95111CA2
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Resource
...

The "ApiEndpoint":"States.Format('https://api.stripe.com')/rest/v1/{}.json',$.detail.object_name" doesn't appear to be correct since per HTTP Task fields, ApiEndpoint using intrinsic functions should be of format "ApiEndpoint.$":"States.Format('https://api.stripe.com/v1/customers/{}', $.customer_id)".

This could be related to the other issue #29925 as well.

[bsod85]

I was able to work around the issue until the fix is ready with the following code in Python CDK:

state_machine_cfn_construct = state_machine.node.find_child('Resource')

definition_string = state_machine_cfn_construct.definition_string

state_machine_cfn_construct.definition_string = Fn.join("\"ApiEndpoint.$\":", Fn.split("\"ApiEndpoint\":", definition_string))

I also altered the api_root and api_endpoint values to have a valid States.Format expression after the concatenation.