Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat(middleware-flexible-checksums): use Node.js native CRC32 checksum API #6641

Merged
merged 3 commits into from
Nov 8, 2024

Conversation

trivikr
Copy link
Member

@trivikr trivikr commented Nov 7, 2024

Issue

Description

Uses Node.js native CRC32 checksum API, when available

Testing

import assert from "assert";
import { S3 } from "../aws-sdk-js-v3/clients/client-s3/dist-cjs/index.js";

const s3 = new S3({ region: "us-west-2" });
const Bucket = "test-flexible-checksums"; // Change to your bucket name
const Key = "test.txt";
const Body = "helloworld".repeat(10_000_000); // A string of size ~100MB

const putObjectStart = performance.now();
await s3.putObject({ Bucket, Key, Body, ChecksumAlgorithm: "CRC32" });
console.log("time taken for putObject request:", (performance.now() - putObjectStart).toFixed(2), "ms");

const getObjectStart = performance.now();
const response = await s3.getObject({ Bucket, Key, ChecksumMode: "ENABLED" });
console.log("time taken for getObject request:", (performance.now() - getObjectStart).toFixed(2), "ms");

const transformStart = performance.now();
const body = await response.Body.transformToString();
console.log("time taken for transform (checksum validation):", (performance.now() - transformStart).toFixed(2), "ms");

assert(body === Body);

Without Node.js native CRC32 (node v20.10.0)

time taken for putObject request: 3399.14 ms
time taken for getObject request: 1145.27 ms
time taken for transform (checksum validation): 2137.78 ms

With Node.js native CRC32 (node v20.17.0)

time taken for putObject request: 1427.59 ms
time taken for getObject request: 1141.85 ms
time taken for transform (checksum validation): 246.76 ms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@trivikr trivikr marked this pull request as ready for review November 7, 2024 23:54
@trivikr trivikr requested a review from a team as a code owner November 7, 2024 23:54
@trivikr

This comment was marked as outdated.

@trivikr trivikr merged commit 6a48760 into aws:main Nov 8, 2024
4 checks passed
@trivikr trivikr deleted the checksum-node-native branch November 8, 2024 16:21
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants