Invalid character in header content [\"user-agent\"] when using ScanCommand from client-dynamodb

[verebes1]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

One of our services has been brought down because of a change in the AWS-SDK. We have been using the "@aws-sdk/client-dynamodb" to instantiate a client:

const client = new DynamoDBClient({maxAttempts: 10});
this.docClient = DynamoDBDocumentClient.from(client);

In the code we were creating a ScanCommand that was retrieving all ids of items from a given DB. This code has been functioning fine without any issue up until 13-March-2025 when there was an autoredeployment of the lambda that updates connection certificates (so no code changes) This has broke our lambdas completely and we started getting the following error.

{
    "errorType": "TypeError",
    "errorMessage": "Invalid character in header content [\"user-agent\"]",
    "code": "ERR_INVALID_CHAR",
    "$metadata": {
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "TypeError [ERR_INVALID_CHAR]: Invalid character in header content [\"user-agent\"]",
        "    at ClientRequest.setHeader (node:_http_outgoing:658:3)",
        "    at new ClientRequest (node:_http_client:293:14)",
        "    at request (node:https:381:10)",
        "    at /var/task/node_modules/@smithy/node-http-handler/dist-cjs/index.js:366:19",
        "    at new Promise (<anonymous>)",
        "    at _NodeHttpHandler.handle (/var/task/node_modules/@smithy/node-http-handler/dist-cjs/index.js:300:12)",
        "    at async /var/task/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24",
        "    at async /var/task/node_modules/@smithy/core/dist-cjs/index.js:167:18",
        "    at async /var/task/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38",
        "    at async /var/task/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22"
    ]
}

After having a call with AWS Support and testing multiple solutions. The only solution that worked was to add the AWS SDK as a node-module to the lambda package and pin it down to an older version specifically:

        "@aws-sdk/client-cloudwatch": "3.658.0",
        "@aws-sdk/client-dynamodb": "3.658.0",
        "@aws-sdk/client-sqs": "3.658.0",
        "@aws-sdk/lib-dynamodb": "3.658.0",
        "@aws-sdk/types": "3.654.0",

Any version post that one creates the same issue
The error message seems to come from one of the aws-sdk dependencies. We have managed to pinpoint the exact line of code that fails with that error and it is the one which sends the scancommand using the docClient.

await this.docClient.send(command);

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-dynamodb@3.767.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

20.x

Reproduction Steps

Just send a ScanCommand to the DB using the latest package version and a Node 20.x runtime.
I am happy to provide more context if needed but this would have to be on a private call with AWS Support

Observed Behavior

{
    "errorType": "TypeError",
    "errorMessage": "Invalid character in header content [\"user-agent\"]",
    "code": "ERR_INVALID_CHAR",
    "$metadata": {
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "TypeError [ERR_INVALID_CHAR]: Invalid character in header content [\"user-agent\"]",
        "    at ClientRequest.setHeader (node:_http_outgoing:658:3)",
        "    at new ClientRequest (node:_http_client:293:14)",
        "    at request (node:https:381:10)",
        "    at /var/task/node_modules/@smithy/node-http-handler/dist-cjs/index.js:366:19",
        "    at new Promise (<anonymous>)",
        "    at _NodeHttpHandler.handle (/var/task/node_modules/@smithy/node-http-handler/dist-cjs/index.js:300:12)",
        "    at async /var/task/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24",
        "    at async /var/task/node_modules/@smithy/core/dist-cjs/index.js:167:18",
        "    at async /var/task/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38",
        "    at async /var/task/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22"
    ]
}

Expected Behavior

Retrieval of the item's ids from a Dynamo DB table

Possible Solution

Pin down the packages to these:

        "@aws-sdk/client-cloudwatch": "3.658.0",
        "@aws-sdk/client-dynamodb": "3.658.0",
        "@aws-sdk/client-sqs": "3.658.0",
        "@aws-sdk/lib-dynamodb": "3.658.0",
        "@aws-sdk/types": "3.654.0",

Additional Information/Context

No response

[zshzbh]

Hey @verebes1 ,

I can't reproduce this issue. And I'm able to a 200 response.
There might be some escaping issue when you set header, but I need more information to confirm. Could you please share the code on how you set the user-agent header?

Could you please add middlewareStack and log the header?

    ddbDocumentClient.middlewareStack.add(
      (next) => async (args) => {
        // Log request headers before the request is sent
        console.log('Request details:', {
          headers: args.request?.headers,
          method: args.request?.method,
          url: args.request?.url
        });

        const result = await next(args);

        // Log response details if needed
        console.log('Response metadata:', result.output?.$metadata);

        return result;
      },
      {
        name: "logRequestDetails",
        step: "build"
      }
    );

I'm also sharing my code with middlewareStack logging to log the request header.

import { DynamoDBClient, GetItemCommand } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand } from "@aws-sdk/lib-dynamodb";
export const handler = async (event) => {
  // TODO implement
  console.log('Full event object:', JSON.stringify(event, null, 2));
   
  try {
    const ddb = new DynamoDBClient({ region: "us-east-1" });

    const ddbDocumentClient = DynamoDBDocumentClient.from(ddb);
    ddbDocumentClient.middlewareStack.add(
      (next) => async (args) => {
        // Modify request headers
        // args.request.headers = {
        //   ...args.request.headers,
        //   'x-custom-timestamp': new Date().toISOString(),
        //   'x-custom-request-id': `req-${Date.now()}`,
        //   'x-application-name': 'my-lambda-function',
        // };

        // Log headers before request
        console.log('Request Headers:', args.request.headers);
        
        // Log request headers before the request is sent
        console.log('Request details:', {
          headers: args.request?.headers,
          method: args.request?.method,
          url: args.request?.url
        });
        
        const result = await next(args);
        
        // Log response details if needed
        console.log('Response metadata:', result.output?.$metadata);
        
        return result;
      },
      {
        name: "logRequestDetails",
        step: "build"
      }
    );

    const command = new GetCommand({
      TableName: "test",
      Key: { testKey: "aaa" },
      // ReturnConsumedCapacity: "TOTAL",
    });

    const result = await ddbDocumentClient.send(command);

    console.log(result);
    console.log("Operation completed");
  } catch (error) {
    console.error("Error:", error);
  }
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  };
  return response;
};

[zshzbh]

Could you please let us know what region you are using?

[verebes1]

We are using the eu-west-1 region.
When it comes to sharing more of the code. I will try to get an example and put it up here. But we already tried that with AWS Support.
The issue is this that we're not even setting the headers and those are causing an error without being touched at all. The way we initialise the client is literally as follows

const client = new DynamoDBClient({maxAttempts: 10});
this.docClient = DynamoDBDocumentClient.from(client);

Then in the methods we simply pass in the ScanCommand with the relevant table parameters
and execute the command using the following line:

const data = await this.docClient.send(command);

Hope this helps the investigation.

[zshzbh]

Thanks for the fast response.

I tried in eu-west-1 region and still couldn't reproduce this issue.

Could you please share your lambda config with us? What architecture are you using? I was using x86_64 and Node.js 22.x / Node.js 20.X and Node.js 18.X , and found no issues. But the aws js sdk version for those three lambda runtime environments are the same - aws-sdk-js/3.726.1 .

[zshzbh]

Hey @verebes1 ,

Since we can't reproduce this error, we still want to see what the headers look like on your end, could you please follow the comment I posted previously and share the raw response with us, so that we can dive deeper to investigate. I understand that you mentioned that you didn't touch this header, but the raw response is what we need to investigate.

Thanks!

[verebes1]

Hey @zshzbh ,
I have received an email that one of the engineers in AWS could reproduce the issue although I cannot see the comment here on GitHub.
I am happy to try and reproduce the headers using the snippet you provided next week at some point, however as I mentioned we are not touching the headers at all. We are simply using the SDK as mentioned in the snippet earlier.
We are happy to share the entire code that is interacting with the SDK under the NDA agreement, however we cannot do that here on GitHub.

[zshzbh]

Hey @verebes1 ,

Since you mentioned that the code is interacting with the SDK under NDA, I'll ask the AWS cloud support engineer to create an internal ticket so that we can chat more internally.

I have received an email that one of the engineers in AWS could reproduce the issue although I cannot see the comment here on GitHub.

The Cloud support engineer observed this issue during a call with you(your team). Sorry for the confusion.

[zshzbh]

ref: D229522017