aws-codebuild: Can't use standard images on GPU instance

[claytonparnell]

Describe the bug

Creating codebuild project via console, I can create using standard images e.g. aws/codebuild/standard:7.0.
However, using CDK, in order to create a GPU project AFAICT I need to use LinuxGpuBuildImage - so I can only create using DLC images - see https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-codebuild/lib/linux-gpu-build-image.ts#L22 . Not only that, but the DLC images available as constants are 4 years old
Why not allow to create via image id here as well?

Expected Behavior

Create GPU project using standard image, from image id, or from LinuxGpuBuildImage constant.
Or, have separate way to specify that the build uses GPU, regardless of BuildImage type (similar to console).

Current Behavior

In order to use GPU, must use LinuxGpuBuildImage. Selection of LinuxGpuBuildImage is limited compared to console, and doesn't offer standard images.

Reproduction Steps

LinuxGpuBuildImage.fromCodeBuildImageId - doesn't exist
LinuxGpuBuildImage.STANDARD_7_0 - doesn't exist
LinuxBuildImage.STANDARD_7_0 - no other way to specify to use GPU on this image

Possible Solution

1. Add separate Project field specifying GPU-enabled build
2. Add ability to select Standard LinuxGpuBuildImage by image id

Additional Information/Context

No response

CDK CLI Version

2.127.0

Framework Version

No response

Node.js Version

18

OS

AL2

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

I think you can use awsDeepLearningContainersImage by specifying repositoryName, tag and optionally the account.

LinuxGpuBuildImage.STANDARD_7_0 - doesn't exist

If this image is available, we should cut a little PR to get it in the enums. But I tried

aws codebuild list-curated-environment-images

and it does not return any images with LINUX_GPU_CONTAINER environment type so I am guessing we still need to specify the repoName, tag and account for that.

Can you share how do you configure the STANDARD_7_0 with GPU support from the console or CLI ? This would help us correctly add the correspond enum in the class.

[claytonparnell]

Thanks for taking a look! Here's a screenshot of the console configuration. In the console, selecting GPU is just a checkbox unrelated to the image type or anything, but when using CDK it seems the supported images are tied to whether or not GPU is being used. Additionally I didn't see ECR repository published for aws/standard/7.0. Here is the definition for STANDARD_7_0 in LinuxBuildImage, just can't find any way to use it for a GPU project 😅

[claytonparnell]

Running aws codebuild list-curated-environment-images for me just returns a list of images, with no reference to GPU. This aligns with how it works in console, allowing me to enable GPU for any image. But does not align with CDK/CLI which is where I am confused.

[claytonparnell]

@pahud does the problem make sense? Just trying to do through CDK what was previously done through console.

[pahud]

I guess the GPU supported images are actually from custom images which are not included in aws codebuild list-curated-environment-images and will need to specify repositoryName, tag and optionally the account. If you select Custom Image from the console you should be allowed to specify all those properties.

[claytonparnell]

@pahud The AL/Ubuntu curated images allow enabling GPU compute on seemingly the same curated image.

Right now to use Ubuntu GPU with CDK, I need to create the Project with LinuxGpuBuildImage using an old GPU DLC, and then override the image with Ubuntu Standard image when I start the build.

However this would fail with cannot use a CodeBuild curated image with imagePullCredentialsType SERVICE_ROLE, which suggests that it is indeed a curated image. Overriding both the image and the imagePullCredentialsType on start build allows it to work, but it would be nice to have some other way to achieve this.

[pahud]

Overriding both the image and the imagePullCredentialsType on start build allows it to work, but it would be nice to have some other way to achieve this.

@claytonparnell we would love to explore better support for that. Can you share your workaround CDK code snippets do I can deploy it in my account?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[claytonparnell]

Sure. So in CDK I would have

        const buildTestProject = new Project(this, 'build-test-public-image', {
            buildSpec: BuildSpec.fromAsset('buildspecs/testing-buildspec.yml'),
            source: Source.gitHub({
                owner: *
                repo: *
            }),
            environment: {
                privileged: true,
                // Placeholder - we will override with STANDARD_CODEBULD_7
                // Limitation in CDK forces us to use DLC image.
                // Track https://github.com/aws/aws-cdk/issues/29094
                buildImage: LinuxGpuBuildImage.DLC_TENSORFLOW_1_14_0,
                // Large only has 50GB disk space, so we use small (220GB)
                computeType: ComputeType.SMALL,
            },
            role: buildTestRole,
            timeout: Duration.hours(8),
        });

Then when I call the project from GitHub Action (which uses start-build), I pass in

buildspec: '{"imageOverride": "aws/codebuild/standard:7.0", "imagePullCredentialsTypeOverride": "CODEBUILD"}'

Using this I am able to replicate the functionality of choosing standard 7.0 image in console, then checking the box to use GPU-enhanced compute

[claytonparnell]

Any update on this?

[claytonparnell]

Just want to check in- if we're limited by CFN support and need to escalate there instead, please let me know