Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

aws-mwaa: CfnEnvironment logging configuration required parameters set to optional #30645

Open
simond opened this issue Jun 24, 2024 · 2 comments
Labels
@aws-cdk/aws-mwaa bug This issue is a bug. effort/medium Medium work item – several days of effort needs-cfn This issue is waiting on changes to CloudFormation before it can be addressed. p2

Comments

@simond
Copy link

simond commented Jun 24, 2024

Describe the bug

When trying to instantiate a new CfnEnvironment with logging configuration set to the following:

loggingConfiguration: {
    taskLogs: {enabled: false},
    webserverLogs: {enabled: false},
    schedulerLogs: {enabled: false},
    workerLogs: {enabled: false},
    dagProcessingLogs: {enabled: false}
}

I get an error when running CDK deploy:

4:23:16 pm | CREATE_FAILED        | AWS::MWAA::Environment      | MwaaEnvironment
Resource handler returned message: "Invalid request provided: 5 validation errors detected: Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfigur
ation.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfigurati
on.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.taskLogs.logLevel' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 916

The error is due to the logLevel attribute not being passed into the logging options. The following fixes the issue:

loggingConfiguration: {
    taskLogs: {enabled: false, logLevel: 'INFO'},
    webserverLogs: {enabled: false, logLevel: 'INFO'},
    schedulerLogs: {enabled: false, logLevel: 'INFO'},
    workerLogs: {enabled: false, logLevel: 'INFO'},
    dagProcessingLogs: {enabled: false, logLevel: 'INFO'},
}

The issue is that that logLevel is not a required parameter according to the TypeScript interface:

/**
 * Defines the Apache Airflow logs to send for the log type (e.g. `DagProcessingLogs` ) to CloudWatch Logs. Valid values: `CRITICAL` , `ERROR` , `WARNING` , `INFO` .
 *
 * @see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-mwaa-environment-moduleloggingconfiguration.html#cfn-mwaa-environment-moduleloggingconfiguration-loglevel
 */
readonly logLevel?: string;

Expected Behavior

CDK should either:

  • Allow me to exclude a logLevel (since I've disabled logging anyway) and deploy the instance of MWAA
    or
  • Force me to provide a logLevel

Current Behavior

Error message asking me to provide the logLevel

Reproduction Steps

        const mwaaEnvironment = new mwaa.CfnEnvironment(this, "MwaaEnvironment", {
            name: 'MyMwaaEnvironment',
            airflowVersion: '2.0.2',
            environmentClass: 'mw1.small',
            maxWorkers: 1,
            minWorkers: 1,
            dagS3Path: 'dags',
            sourceBucketArn: {some bucket arn},
            executionRoleArn: {some role arn},
            requirementsS3Path: 'requirements.txt',
            pluginsS3Path: "plugins.zip",
            networkConfiguration: {
                securityGroupIds: [securityGroup.securityGroupId],
                subnetIds: vpc.selectSubnets({subnetType: ec2.SubnetType.PUBLIC}).subnetIds.slice(0, 2)
            },
            webserverAccessMode: 'PUBLIC_ONLY',
            loggingConfiguration: {
                taskLogs: {enabled: false, logLevel: 'INFO'},
                webserverLogs: {enabled: false},
                schedulerLogs: {enabled: false},
                workerLogs: {enabled: false},
                dagProcessingLogs: {enabled: false},
            }
        });

Possible Solution

Provide a logLevel

Additional Information/Context

No response

CDK CLI Version

2.145.0 (build fdf53ba)

Framework Version

No response

Node.js Version

v22.0.0

OS

osx

Language

TypeScript

Language Version

v10.9.2

Other information

No response

@simond simond added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 24, 2024
@ashishdhingra ashishdhingra self-assigned this Jun 24, 2024
@ashishdhingra ashishdhingra added needs-reproduction This issue needs reproduction. and removed needs-triage This issue or PR still needs to be triaged. labels Jun 24, 2024
@ashishdhingra
Copy link
Contributor

ashishdhingra commented Jun 24, 2024

Findings:

  • Per AWS::MWAA::Environment LoggingConfiguration, the properties DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs and WorkerLogs are of type ModuleLoggingConfiguration.
  • ModuleLoggingConfiguration, LogLevel is optional.
    LogLevel
    Defines the Apache Airflow logs to send for the log type (e.g. DagProcessingLogs) to CloudWatch Logs. Valid values: CRITICAL, ERROR, WARNING, INFO.
    
    Required: No
    
    Type: String
    
    Allowed values: CRITICAL | ERROR | WARNING | INFO | DEBUG
    
    Update requires: [No interruption](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
    
    The CloudFormation resource spec here also states LogLevel as optional.

However, deploying the following CDK stack:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as mwaa from 'aws-cdk-lib/aws-mwaa';

export class Issue30645Stack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const mwaaEnvironment = new mwaa.CfnEnvironment(this, "MwaaEnvironment", {
      name: 'MyMwaaEnvironment',
      loggingConfiguration: {
          taskLogs: {enabled: false, logLevel: 'INFO'},
          webserverLogs: {enabled: false},
          schedulerLogs: {enabled: false},
          workerLogs: {enabled: false},
          dagProcessingLogs: {enabled: false},
          }
        });
  }
}

gives the below validation error from CloudFormation:

11:23:22 AM | CREATE_FAILED        | AWS::MWAA::Environment | MwaaEnvironment
Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value nu
ll at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to
satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggin
gConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null
; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Serv
ice: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)


 ❌  Issue30645Stack failed: Error: The stack named Issue30645Stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:454:181438

 ❌ Deployment failed: Error: The stack named Issue30645Stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:454:181438

The stack named Issue30645Stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)

Same goes for SourceBucketArn which is specified as not required at AWS::MWAA::Environment.

Most likely an issue with CloudFormation resource specification.

@ashishdhingra ashishdhingra added p1 effort/medium Medium work item – several days of effort needs-cfn This issue is waiting on changes to CloudFormation before it can be addressed. and removed needs-reproduction This issue needs reproduction. labels Jun 24, 2024
@ashishdhingra ashishdhingra removed their assignment Jun 24, 2024
@ashishdhingra ashishdhingra added p2 and removed p1 labels Jun 24, 2024
@ashishdhingra
Copy link
Contributor

ashishdhingra commented Jun 24, 2024

Internal tracking ticket: P137161118

Another internal ticket (from above ticket): P137427298

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
@aws-cdk/aws-mwaa bug This issue is a bug. effort/medium Medium work item – several days of effort needs-cfn This issue is waiting on changes to CloudFormation before it can be addressed. p2
Projects
None yet
Development

No branches or pull requests

2 participants