request not captured in @sentry/aws-serverless + express

[relm923]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/aws-serverless

SDK Version

8.37.1

Framework Version

AWS Lambda + Express 4.21.1

Link to Sentry event

https://patina-health.sentry.io/issues/5996169896?project=6044571

Reproduction Example/SDK Setup

import * as Sentry from '@sentry/aws-serverless';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: 'development',
  integrations: [Sentry.expressIntegration()],
  tracesSampleRate: 1.0,
  profilesSampleRate: 1.0,
});

import * as Sentry from '@sentry/aws-serverless';
import express from 'express';
import serverless from 'serverless-http';

const app = express();

app.get('/debug-sentry', (req, res) => {
  throw new Error('My first Sentry error!');
});

Sentry.setupExpressErrorHandler(app);

export const handler = Sentry.wrapHandler(serverless(app));

Steps to Reproduce

Run the above code ideally deployed to AWS. Invoke with payload similar to:

{
  "headers": { "content-type": "application/json" },
  "httpMethod": "GET",
  "path": "/debug-sentry"
}

Error is reported to Sentry but the request information is not included

Expected Result

Sentry event should include the request information.

This worked correctly with the same setup on Sentry 7

Actual Result

Sentry event does include aws.cloudwatch.logs and aws.lambda but no request information

[chargome]

Hey @relm923, thanks for reaching out. integrations: [Sentry.expressIntegration()], is not needed anymore in v8, you can safely remove that. Theoretically the requestDataIntegration should be enabled by default and include your request data. So can you please remove the integrations field in your init call and if it still not works add debug: true and check the output?

The latest docs on express can be found here.

[relm923]

Hi @chargome removing integrations does not appear to enable Express

import * as Sentry from '@sentry/aws-serverless';

Sentry.init({
  debug: true,
  dsn: process.env.SENTRY_DSN,
  environment: 'development',
  tracesSampleRate: 1.0,
  profilesSampleRate: 1.0,
});

Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariablesAsync
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: ProcessAndThreadBreadcrumbs
Sentry Logger [log]: Integration installed: Modules
Sentry Logger [log]: Integration installed: Aws
Sentry Logger [log]: Integration installed: AwsLambda
Sentry Logger [log]: Running in CommonJS mode.

import * as Sentry from '@sentry/aws-serverless';

Sentry.init({
  debug: true,
  dsn: process.env.SENTRY_DSN,
  environment: 'development',
  integrations: [Sentry.expressIntegration()],
  tracesSampleRate: 1.0,
  profilesSampleRate: 1.0,
});

Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariablesAsync
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: ProcessAndThreadBreadcrumbs
Sentry Logger [log]: Integration installed: Modules
Sentry Logger [log]: Integration installed: Aws
Sentry Logger [log]: Integration installed: AwsLambda
Sentry Logger [log]: Integration installed: Express
Sentry Logger [log]: Running in CommonJS mode.

[chargome]

You won't need this integration since it gets auto-instrumented out of the box. Does the setup work like this?

[relm923]

It does not work. Without the integrations param this appears in the logs

[Sentry] express is not instrumented. This is likely because you required/imported express before calling `Sentry.init()`.

Looking at the code it looks like aws-serverless not auto-instrument performance/tracing integrations

sentry-javascript/packages/aws-serverless/src/sdk.ts

Line 69 in 0aa7f1d

return [...getDefaultIntegrationsWithoutPerformance(), awsIntegration(), awsLambdaIntegration()];

https://github.com/getsentry/sentry-javascript/blob/develop/packages/node/src/sdk/index.ts#L57

[chargome]

@relm923 some questions:

• How are you creating your lambda? e.g. are you uploading your code alongside your node modules? Are you transpiling your code to cjs or is the code in the issue exactly the code that is executed in your lambda?
• What kind of request information would you expect to see in your case?

[relm923]

How are you creating your lambda? e.g. are you uploading your code alongside your node modules? Are you transpiling your code to cjs or is the code in the issue exactly the code that is executed in your lambda?

Transpiring with ESBuild to CJS and explicitly including express outside of bundling to ensure OpenTelemetry can inject

I'm able to reproduce the same behavior locally without any bundling as well

What kind of request information would you expect to see in your case?

Everything from RequestData

[chargome]

got it, maybe the issue is related to #13746 then, wdyt @mydea ?

[mydea]

Yeah, I guess it is true that we do not set the request for AWS handlers. We will look into this and see if we can add this data to the events!

[mydea]

One question: You wrote this:

Run the above code ideally deployed to AWS. Invoke with payload similar to:

{
  "headers": { "content-type": "application/json" },
  "httpMethod": "GET",
  "path": "/debug-sentry"
}

Is this some kind of standard, and if so, where is this defined? We get an event and context passed to the handler, looking at the types I do not see an obvious/standard way to access this data 🤔 (I am not an AWS expert though, so def. possible I am missing something there). I found some posts indicating we may find something at event.requestContext.http but not sure if this is standardized 🤔 cc @andreiborza & @Lms24

I will put this in our backlog generally, as we are pretty swamped with stuff right now. PRs are also welcome if you want to take a stab yourself to fix this - you'd have to put code into the wrapHandler function similar to what we already do with enhanceScopeWithEnvironmentData, but with something like scope.setSDKProcessingMetadata({ request: { headers, method } }) or similar.

[relm923]

Generally for AWS Lambda event and context can basically be anything. For our setup we have APIGateway in front of our Lambdas so the inputs follow the API Gateway spec. There's no guarantee that all Lambdas will use this pattern though.

I would expect the raw request/input to the lambda (event & context) and the express request/input (headers, method, etc) both be capture by Sentry. In v7 the express was automatically captured when using the Express integration

I would also suggest updating the v8 docs as aws-serverless does NOT auto-instrument any of the performance/tracing integrations

[mydea]

Is this still an issue in v9? We changed some things around the request capturing there!

[rowbot-weisguy]

@mydea I'm experiencing a problem that resembles this issue: express app deployed via lambda which is missing expected request traces / profiles.

For local development my entrypoint is using @sentry/node.

Here's the entrypoint where Sentry is initialized:

// server.ts
// sort-imports-ignore
import * as Sentry from "@sentry/node";
import { nodeProfilingIntegration } from "@sentry/profiling-node";

import { env } from "./env";

Sentry.init({
  dsn: env.PILOT_API_SENTRY_DSN,
  integrations: [nodeProfilingIntegration()],
  environment: env.PILOT_API_SENTRY_ENVIRONMENT,
  tracesSampleRate: Number(env.PILOT_API_SENTRY_TRACING_SAMPLE_RATE),
  profilesSampleRate: Number(env.PILOT_API_SENTRY_PROFILING_SAMPLE_RATE),
});

import { createApp } from "./app";
import routes from "./routes";

const app = createApp(routes, { enableLogging: true });
const port = env.API_PORT || 4300;
app.listen(port, () => console.log(`Server is listening on port ${port}.`));

export default app;

It's giving all the traces / spans I expect!

For the deployed lambda I'm using @sentry/aws-serverless.

Here's the entrypoint where Sentry is initialized:

// lambda.ts
// sort-imports-ignore
import * as Sentry from "@sentry/aws-serverless";
import { nodeProfilingIntegration } from "@sentry/profiling-node";

import { env } from "./src/env";

Sentry.init({
  dsn: env.PILOT_API_SENTRY_DSN,
  integrations: [
    nodeProfilingIntegration(),
    // Sentry.expressIntegration(),
    // Sentry.redisIntegration(),
    // Sentry.postgresIntegration(),
  ],
  environment: env.PILOT_API_SENTRY_ENVIRONMENT,
  tracesSampleRate: Number(env.PILOT_API_SENTRY_TRACING_SAMPLE_RATE),
  profilesSampleRate: Number(env.PILOT_API_SENTRY_PROFILING_SAMPLE_RATE),
});

import { createApp } from "./src/app";
import serverlessExpress from "@vendia/serverless-express";
import router from "./src/routes";

const app = createApp(router);

export const handler = Sentry.wrapHandler(
  serverlessExpress({
    app,
    logSettings: {
      level: "info",
    },
  }),
);

export default handler;

The test error doesn't reveal any useful traces, and bunches everything under function.aws.lambda with the lambda name, rather than the request path.

In the right sidebar of the traces screen I see: "This transaction does not have any child spans. You can add more child spans via custom instrumentation."

If I uncomment those integrations I still get the same result.

Lastly, I'm noticing in our Cloudwatch logs that the Lambda is logging warnings:

[Sentry] express is not instrumented. This is likely because you required/imported express before calling `Sentry.init()`.

I believe I've followed the docs. Not sure what I'm missing re: lambda + express configuration.

[andreiborza]

The aws integration basically takes over the naming of the spans. I wonder if in your case it wouldn't be better to just disable the aws integration instead 🤔.

[relm923]

@rowbot-weisguy are you bundling/transpiling your lambda code? If so it will break Sentry OpenTelemetry automatic instrumentation. Need to ensure express and any other instrumented dependency are not bundled

@andreiborza unfortunately that does appear to be the case. I haven't been able to get Lambda & express spans working together. This did work with Sentry 7

[rowbot-weisguy]

@relm923 good point. We use CDK so I've been bundling it using esbuild via aws-cdk-lib/aws-lambda-nodejs.

Today, given your advice, I set up a separate layer and marked express as an external module. It works! Warning is gone and I managed to get traces that I wanted to see from the lambda as long as I explicitly include the integrations I care about (e.g. expressIntegration, redisIntegration, and postgresIntegration).

I'm still noticing the generated issue titles (span names?) are a little unhelpful.

In development, using @sentry/node I see:

Lovely! The path is so helpful to know immediately. Whereas in deployed lambdas, using @sentry/aws-serverless I see:

So I tried implementing your suggestion @andreiborza:

// instrument.lambda.ts
import * as Sentry from "@sentry/aws-serverless";
import { nodeProfilingIntegration } from "@sentry/profiling-node";

import { env } from "./env";

Sentry.init({
  dsn: env.PILOT_API_SENTRY_DSN,
  defaultIntegrations: false,
  integrations: [
    ...Sentry.getDefaultIntegrations({}).filter(
      (integration) => !["Aws", "AwsLambda"].includes(integration.name),
    ),
    nodeProfilingIntegration(),
    Sentry.expressIntegration(),
    Sentry.redisIntegration(),
    Sentry.postgresIntegration(),
  ],
  environment: env.PILOT_API_SENTRY_ENVIRONMENT,
  tracesSampleRate: Number(env.PILOT_API_SENTRY_TRACING_SAMPLE_RATE),
  profilesSampleRate: Number(env.PILOT_API_SENTRY_PROFILING_SAMPLE_RATE),
});

export const wrapHandler = Sentry.wrapHandler;

This change made no visible difference to spans / names, and I'm not sure if there's a different way I should disable the Aws integration.

Would love to make sure our Sentry issues are informative at a glance, so please let me know if there's something else I can do here. This is my first time instrumenting Sentry w/ Express & Lambda, so I have no experience with previous versions to share.

[s1gr1d]

@rowbot-weisguy I created a new issue for the issue title naming as it's not related to the rest of the thread anymore: #15788

[mydea]

This issue contains a bunch of semi-related problems by now, some of which have been spinned out into separate issues and some not. It is a bit hard to keep track of what has been changed or not 😅

• For naming-related stuff a new issue exists tot rack this (Issue titles from errors captured with @sentry/aws-serverless differ from @sentry/node #15788)
• request should be set on events even in lambda wherever we have proper instrumentation set up in v9. There are also further changes around this in the upcoming v9.16.0 release, that may fix some more problems around this.
• General instrumentation issues in AWS Lambda (??)

I will close this issue for now, and would ask you to open new more specific issues if you still encounter such problems on the latest version - as this issue is technically only about capturing the request data on a handler, which should work fine if instrumentation is done according to https://docs.sentry.io/platforms/javascript/guides/aws-lambda/, depending on if your setup uses CJS or ESM.

To be clear, there may def. still be something left to do, it is just a bit hard to be sure because the issue became a bit of a catch-all issue for aws related instrumentation issues - please bear with us and open a new issue with as much details as possible if problems persist, thank you!