iinteg-tests-alpha: can't have assertions make a deploy fail

[mlambir]

Describe the bug

I want to have Integration tests run both as standalone tests using integ-runner and as part of the regular deployment. Right now, I'm able to do this, but can't get the deployments to fail when tests fail.

Expected Behavior

Failed integration tests that are part of a regular deployment should make the deployment fail.

Current Behavior

Deployment shows the failed test output, but succeeds anyways.

Reproduction Steps

Given the following stack:

#import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import createAssertions from "./assertions";

export interface StackProps extends cdk.StackProps {
  runAssertions?: boolean
  envName?: string
}

export class IdemoStack extends cdk.Stack {
  readonly sampleFunction: lambda.Function;
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const {runAssertions = true, envName=""} = props??{}
    console.log("props: ", props)

    this.sampleFunction = new lambda.Function(this, `sampleFunction${envName? "-" + envName:""}`, {
      runtime: lambda.Runtime.NODEJS_LATEST,
      handler: "index.handler",
      code: lambda.Code.fromInline(`
        exports.handler = async (event) => {
          return true;
        };
      `),
    });
    this.sampleFunction.currentVersion

    if (runAssertions) {
      createAssertions(this)
    }
  }
}

the following integration test file

import { App } from "aws-cdk-lib";
import { IdemoStack } from "../lib/idemo-stack";
import { IntegTest } from "@aws-cdk/integ-tests-alpha";
import createAssertions from "../lib/assertions";
import { DeployAssert } from '@aws-cdk/integ-tests-alpha/lib/assertions/private/deploy-assert';

const app = new App();

const stack = new IdemoStack(app, "StackUnderTest", {runAssertions:false, envName:"integration"});

const integTest = new IntegTest(app, "IntegrationTests", {
  testCases: [stack],
  diffAssets: true,
  stackUpdateWorkflow: false,
  cdkCommandOptions: {
    destroy: {
      args: {
        force: true,
      },
    },
  },
});

const deployAssert = integTest.assertions;
if (!DeployAssert.isDeployAssert(deployAssert)) {
  throw new Error('Expected DeployAssert');
}
const assertionScope = deployAssert.scope;

createAssertions(stack, assertionScope);

app.synth()

and the assertions defined like this:

import * as integ_tests from "@aws-cdk/integ-tests-alpha";
import { IdemoStack } from "./idemo-stack";
import { CustomResource, Stack } from "aws-cdk-lib";

export default function createAssertions(
  stack: IdemoStack,
  testStack?: Stack,
) {
  const testStack_ = testStack ?? stack;
  const val = new integ_tests.LambdaInvokeFunction(
    testStack_,
    "testSampleFunction",
    {
      functionName: stack.sampleFunction.functionName,
    },
  );
  val.expect(integ_tests.ExpectedResult.objectLike({ Payload: 'false' }));
}

If I run the tests with integ-runner, they fail as expected. But if I do a cdk deploy, the test fails but the deployment succeeds

Possible Solution

I find a workaround to make the deployment fail as I expect it to. if I add the following to the end of the createAssertions function

  new integ_tests.EqualsAssertion(testStack_, "assertAssertionPassed", {
    failDeployment: true, 
    expected: integ_tests.ExpectedResult.objectLike({status: "success"}),
    actual: integ_tests.ActualResult.fromCustomResource(val.node.findChild("Default") as CustomResource, "assertion")
  }).node.tryRemoveChild("AssertionResults")

Then the deployment fails when the tests fail

Additional Information/Context

Ideally, I think there should be a way to add the failDeployment: true to the regular expect instead of having to check for the assertion value in a separate assertion

CDK CLI Version

2.143.1

Framework Version

No response

Node.js Version

20.12.2

OS

MacOs

Language

TypeScript

Language Version

5.4.5

Other information

No response

[pahud]

Thank you for you valuable feedback. We'll discuss this with the team.

[TheRealAmazonKendra]

This is a really interesting proposal. We don't currently have the ability to do this, but I think I like the idea.