Skip to content

Commit

Permalink
chore(glue): timeout and worker type validation for Ray jobs (#32119)
Browse files Browse the repository at this point in the history
### Issue # (if applicable)

Closes #29612.

### Reason for this change

AWS Glue Ray job has some restriction.
- must use Z.2X worker type
```sh
CREATE_FAILED [...] Worker type cannot be null and only [Z.2X] worker types are supported for glueray jobs
```
- must not specify timeout
```sh
UPDATE_FAILED [...] Timeout not supported for Ray jobs
```

### Description of changes

Add validation for above restriction.

```ts
    if (executable.type.name === JobType.RAY.name) {
      if (props.workerType !== WorkerType.Z_2X) {
        throw new Error(`WorkerType must be Z_2X for Ray jobs, got: ${props.workerType}`);
      }
      if (props.timeout !== undefined) {
        throw new Error('Timeout cannot be set for Ray jobs');
      }
    }
```

### Description of how you validated changes

Add unit test.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
badmintoncryer authored Nov 20, 2024
1 parent c768554 commit be33610
Show file tree
Hide file tree
Showing 2 changed files with 43 additions and 0 deletions.
11 changes: 11 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/job.ts
Original file line number Diff line number Diff line change
Expand Up @@ -738,6 +738,17 @@ export class Job extends JobBase {
}
}

// Validate Ray job properties
// See https://github.com/aws/aws-cdk/issues/29612
if (executable.type.name === JobType.RAY.name) {
if (props.workerType !== WorkerType.Z_2X) {
throw new Error(`WorkerType must be Z_2X for Ray jobs, got: ${props.workerType}`);
}
if (props.timeout !== undefined) {
throw new Error('Timeout cannot be set for Ray jobs');
}
}

let maxCapacity = props.maxCapacity;
if (maxCapacity !== undefined && (props.workerType && props.workerCount !== undefined)) {
throw new Error('maxCapacity cannot be used when setting workerType and workerCount');
Expand Down
32 changes: 32 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/test/job.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -896,6 +896,38 @@ describe('Job', () => {
workerCount: 2,
})).toThrow('Runtime is required for Ray jobs');
});

test.each([
glue.WorkerType.G_025X,
glue.WorkerType.G_1X,
glue.WorkerType.G_2X,
glue.WorkerType.G_4X,
glue.WorkerType.G_8X,
])('throw error for unsupported worker type', (workerType) => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonRay({
glueVersion: glue.GlueVersion.V4_0,
pythonVersion: glue.PythonVersion.THREE_NINE,
runtime: glue.Runtime.RAY_TWO_FOUR,
script,
}),
workerType,
workerCount: 2,
})).toThrow(`WorkerType must be Z_2X for Ray jobs, got: ${workerType}`);
});
});

test('throw error for specifying timeout', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonRay({
glueVersion: glue.GlueVersion.V4_0,
pythonVersion: glue.PythonVersion.THREE_NINE,
runtime: glue.Runtime.RAY_TWO_FOUR,
script,
}),
workerType: glue.WorkerType.Z_2X,
timeout: cdk.Duration.minutes(5),
})).toThrow('Timeout cannot be set for Ray jobs');
});

test('etl job with all props should synthesize correctly', () => {
Expand Down

0 comments on commit be33610

Please # to comment.