aws-rds: export value from network-stack cannot be deleted as it is in use by database-stack

[mdesousa]

Describe the bug

i have created a network stack with 3 subnets: one public, one private with egress, and one isolated.
in a separate database stack i'm creating a database cluster (Aurora Serverless V2 Postgres) and initially used the public subnet by setting vpcSubnets: { subnetType: SubnetType.PUBLIC }.

Now I would like to move the database to the isolated subnet by setting vpcSubnets: { subnetType: SubnetType.PRIVATE_ISOLATED }, however this results in an error: ExportsOutputRefnetworkstackvpcingressSubnet3SubnetCD2EA007C4DD55EA cannot be deleted as it is in use by database-stack.

I have tried doing a deployment that adds the new isolated subnets without removing the public ones, and that succeeded. But when I try to remove the public subnets I face the same error.

Expected Behavior

the cdk should be able to reassign the subnets without errors

Current Behavior

deployment fails with an error: ExportsOutputRefnetworkstackvpcingressSubnet3SubnetCD2EA007C4DD55EA cannot be deleted as it is in use by database-stack.

Reproduction Steps

create a vpc with public and isolated subnets.
create a database cluster in a vpc with vpcSubnets: { subnetType: SubnetType.PUBLIC }.
then deploy again with vpcSubnets: { subnetType: SubnetType.PRIVATE_ISOLATED }

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.85.0

Framework Version

No response

Node.js Version

18.15.0

OS

macOS

Language

Typescript

Language Version

TypeScript (5.1.3)

Other information

No response

[pahud]

Looks like your database stack is importing an export value from network stack and your network stack is trying to update that export value, which is prohibited in cloudformation.

A common fix is to create an intrinsic function in your consuming stack(i.e. database-stack in your case) and point the import value to a static value, which means the exported value will not be consumed, and you should be able to update your network stack export value.

Check out the discussion in this repost:
https://repost.aws/knowledge-center/cloudformation-stack-export-name-error
or this blog post:
https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references

[mdesousa]

thanks @pahud , that was very helpful... now i understand the issue better.
so i'm trying to flag the subnets as exported from the network stack. i'm trying the approach below, but it doesn't seem to be working... cdk is still giving an error saying that they cannot be exported. is subnetId the right property to use to flag the subnet as exported by the stack?

vpc.publicSubnets.forEach(x => this.exportValue(x.subnetId))

[pahud]

@mdesousa As I can't see your source code I am not sure the details. But basically you probably should just export Vpc from your network stack.

I will investigate this and provide a suggested sample for you shortly.

[pahud]

Alright. Let's consider this example below:

export class NetworkStack extends cdk.Stack {
  readonly vpc: ec2.IVpc;
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    this.vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 3, natGateways: 1})
  }
}

export interface DatabaseStackProps extends cdk.StackProps {
  readonly vpc: ec2.IVpc;
}

export class DatabaseStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: DatabaseStackProps) {
    super(scope, id, props)

    // create the cluster
    new rds.DatabaseCluster(this, 'Database', {
      engine: rds.DatabaseClusterEngine.auroraMysql({ version: rds.AuroraMysqlEngineVersion.VER_3_03_0 }),
      writer: rds.ClusterInstance.provisioned('writer'),
      vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
      vpc: props.vpc,
    });
  }
}

And app.ts

const app = new cdk.App();
const env = { region: process.env.CDK_DEFAULT_REGION, account: process.env.CDK_DEFAULT_ACCOUNT };

const networkStack = new NetworkStack(app, 'Network', { env });
new DatabaseStack(app, 'DatabaseStack', {
  env,
  vpc: networkStack.vpc,
});

When we first deploy the two stacks, the database stack will have some imports from the network stack:

% npx cdk synth DatabaseStack | grep ImportValue
        - Fn::ImportValue: Network:ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940
        - Fn::ImportValue: Network:ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3
        - Fn::ImportValue: Network:ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD
        Fn::ImportValue: Network:ExportsOutputRefVpc8378EB38272D6E3A

And if we change the vpcSubnets of DatabaseCuster using ec2.SubnetType.PUBLIC

    new rds.DatabaseCluster(this, 'Database', {
      engine: rds.DatabaseClusterEngine.auroraMysql({ version: rds.AuroraMysqlEngineVersion.VER_3_03_0 }),
      writer: rds.ClusterInstance.provisioned('writer'),
      vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
      vpc: props.vpc,
    });

Let's synth again

% npx cdk synth DatabaseStack | grep ImportValue
        - Fn::ImportValue: Network:ExportsOutputRefVpcPublicSubnet1Subnet5C2D37C4FFA2B456
        - Fn::ImportValue: Network:ExportsOutputRefVpcPublicSubnet2Subnet691E08A351552740
        - Fn::ImportValue: Network:ExportsOutputRefVpcPublicSubnet3SubnetBE12F0B65CC33245
        Fn::ImportValue: Network:ExportsOutputRefVpc8378EB38272D6E3A

As you notice, the first 3 imports have changed from private to public and the network stack Export will change from

% npx cdk synth Network | grep Export        
  ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940:
    Export:
      Name: Network:ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940
  ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3:
    Export:
      Name: Network:ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3
  ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD:
    Export:
      Name: Network:ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD
  ExportsOutputRefVpc8378EB38272D6E3A:
    Export:
      Name: Network:ExportsOutputRefVpc8378EB38272D6E3A

to

 % npx cdk synth Network | grep Export
  ExportsOutputRefVpcPublicSubnet1Subnet5C2D37C4FFA2B456:
    Export:
      Name: Network:ExportsOutputRefVpcPublicSubnet1Subnet5C2D37C4FFA2B456
  ExportsOutputRefVpcPublicSubnet2Subnet691E08A351552740:
    Export:
      Name: Network:ExportsOutputRefVpcPublicSubnet2Subnet691E08A351552740
  ExportsOutputRefVpcPublicSubnet3SubnetBE12F0B65CC33245:
    Export:
      Name: Network:ExportsOutputRefVpcPublicSubnet3SubnetBE12F0B65CC33245
  ExportsOutputRefVpc8378EB38272D6E3A:
    Export:
      Name: Network:ExportsOutputRefVpc8378EB38272D6E3A

And if you run cdk diff

% npx cdk diff Network              
Stack Network
Outputs
[-] Output ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940: {"Value":{"Ref":"VpcPrivateSubnet1Subnet536B997A"},"Export":{"Name":"Network:ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940"}}
[-] Output ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3: {"Value":{"Ref":"VpcPrivateSubnet2Subnet3788AAA1"},"Export":{"Name":"Network:ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3"}}
[-] Output ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD: {"Value":{"Ref":"VpcPrivateSubnet3SubnetF258B56E"},"Export":{"Name":"Network:ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD"}}

It will remove 3 exports which are in used. This not allowed in cloudformation because the private subnet exports are in used by the consumer stack and can't be modified or deleted.

Now, this is our hack. Let's use this.exportValue() to create dummy exports for those private subnets.

export class NetworkStack extends cdk.Stack {
  readonly vpc: ec2.IVpc;
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    this.vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 3, natGateways: 1})
    this.vpc.selectSubnets({ subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS}).subnetIds.map(
      x => this.exportValue(x)
    )   
  }
}

Now, run cdk diff again

% npx cdk diff Network
Stack Network
There were no differences

YES! No difference. Let's check out all the exports:

% npx cdk synth Network | grep Export
  ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940:
    Export:
      Name: Network:ExportsOutputRefVpcPrivateSubnet1Subnet536B997AFD4CC940
  ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3:
    Export:
      Name: Network:ExportsOutputRefVpcPrivateSubnet2Subnet3788AAA1380949A3
  ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD:
    Export:
      Name: Network:ExportsOutputRefVpcPrivateSubnet3SubnetF258B56EC7CF32DD
  ExportsOutputRefVpcPublicSubnet1Subnet5C2D37C4FFA2B456:
    Export:
      Name: Network:ExportsOutputRefVpcPublicSubnet1Subnet5C2D37C4FFA2B456
  ExportsOutputRefVpcPublicSubnet2Subnet691E08A351552740:
    Export:
      Name: Network:ExportsOutputRefVpcPublicSubnet2Subnet691E08A351552740
  ExportsOutputRefVpcPublicSubnet3SubnetBE12F0B65CC33245:
    Export:
      Name: Network:ExportsOutputRefVpcPublicSubnet3SubnetBE12F0B65CC33245
  ExportsOutputRefVpc8378EB38272D6E3A:
    Export:
      Name: Network:ExportsOutputRefVpc8378EB38272D6E3A

We will export 7 values(3 public, 3 private(dummy) and 1 vpc)

OK let's cdk deploy to update the network stack:

Now, let's cdk deploy to update the database stack. You will see this error:

7:58:40 PM | UPDATE_FAILED        | AWS::RDS::DBSubnetGroup                     | DatabaseSubnets56F17B9A
Resource handler returned message: "Some of the subnets to be deleted are currently in use: subnet-0af5aaa99d09f81a1 (Service: Rds, Status Code: 400, Request
ID: 86abcfff-ad88-4eb4-8998-7601be5d1198)" (RequestToken: 33c50547-5e78-4590-1361-77c3fac3a2f8, HandlerErrorCode: InvalidRequest)

This is because we can't in-place replace the SubnetGroup to remove subnets which are in-used by the DB.

We need to create a separate subnetGroup like this and specify it in the DatabaseCluster property.

export class DatabaseStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: DatabaseStackProps) {
    super(scope, id, props)

    // create a new subnetGroup
    const subnetGroup = new rds.SubnetGroup(this, 'SubnetGroup', {
      vpc: props.vpc,
      vpcSubnets:  { subnetType: ec2.SubnetType.PUBLIC },
      description: 'public subnet group',
    })
    // create the cluster
    new rds.DatabaseCluster(this, 'Database', {
      engine: rds.DatabaseClusterEngine.auroraMysql({ version: rds.AuroraMysqlEngineVersion.VER_3_03_0 }),
      writer: rds.ClusterInstance.provisioned('writer'),
      // vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
      subnetGroup,
      vpc: props.vpc,
    });
  }
}

Now it deploys and our database cluster should be running in public subnets now.

Last but not least, we should remove the this.exportValue() statements and cdk deploy the network stack as dummy exports are no longer required.

This is a little bit complicated but I hope my sample helps you understand how to deal with this problem in CDK.

[mdesousa]

thanks @pahud ! this was a very thorough and clear explanation.
looks like i was close in my last snippet that was exporting the subnets.
i was still running into that last issue with the subnet deletions, and after adding the subnet group the deployment succeeded 👍
however... my database got dropped and recreated, which was unexpected. possibly because of the addition of the subnet group?

[mdesousa]

quick update: it appears that changing the vpcSubnets congifuration for the subnet group result in the "Some of the subnets to be deleted are currently in use" error. if i create a new subnet group with a different id the database gets dropped and recreated.