From 422140229a3af9906c04d9366ce1b1a1f7289635 Mon Sep 17 00:00:00 2001 From: Nsupyq <93358967+Nsupyq@users.noreply.github.com> Date: Thu, 4 Nov 2021 20:07:35 +0800 Subject: [PATCH 1/3] feat: add idempontent example for aws node express --- .../.gitignore | 2 + .../README.md | 159 ++++++++++++++++++ .../handler.js | 80 +++++++++ .../package.json | 9 + .../serverless.template.yml | 6 + .../serverless.yml | 45 +++++ 6 files changed, 301 insertions(+) create mode 100644 idempotent-aws-node-express-dynamodb-api/.gitignore create mode 100644 idempotent-aws-node-express-dynamodb-api/README.md create mode 100644 idempotent-aws-node-express-dynamodb-api/handler.js create mode 100644 idempotent-aws-node-express-dynamodb-api/package.json create mode 100644 idempotent-aws-node-express-dynamodb-api/serverless.template.yml create mode 100644 idempotent-aws-node-express-dynamodb-api/serverless.yml diff --git a/idempotent-aws-node-express-dynamodb-api/.gitignore b/idempotent-aws-node-express-dynamodb-api/.gitignore new file mode 100644 index 000000000..2c4448065 --- /dev/null +++ b/idempotent-aws-node-express-dynamodb-api/.gitignore @@ -0,0 +1,2 @@ +node_modules +.serverless diff --git a/idempotent-aws-node-express-dynamodb-api/README.md b/idempotent-aws-node-express-dynamodb-api/README.md new file mode 100644 index 000000000..9f3d4df75 --- /dev/null +++ b/idempotent-aws-node-express-dynamodb-api/README.md @@ -0,0 +1,159 @@ + + +# Serverless Framework Node Express API on AWS with Idempotence Guaranteed + +This template demonstrates how to add idempotence in a simple Node Express API service, backed by DynamoDB database, running on AWS Lambda using the traditional Serverless Framework. It is based on the example of [Serverless Framework Node Express API on AWS](../aws-node-express-dynamodb-api/README.md). + +## Anatomy of the template + +This template configures a single function, `api`, which is responsible for handling all incoming requests thanks to the `httpApi` event. To learn more about `httpApi` event configuration options, please refer to [httpApi event docs](https://www.serverless.com/framework/docs/providers/aws/events/http-api/). As the event is configured in a way to accept all incoming requests, `express` framework is responsible for routing and handling requests internally. Implementation takes advantage of `serverless-http` package, which allows you to wrap existing `express` applications. To learn more about `serverless-http`, please refer to corresponding [GitHub repository](https://github.com/dougmoscrop/serverless-http). Additionally, it also handles provisioning of a DynamoDB database that is used for storing data about users. It uses context from the request as the `clientRequestToken` parameter in the [transactional write api](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html) of DynamoDB to enable the guarantee of idempotence. The `express` application exposes two endpoints, `POST /users` and `GET /user/{userId}`, which allow to create and retrieve users. + +## Usage + +### Deployment + +Install dependencies with: + +``` +npm install +``` + +and then deploy with: + +``` +serverless deploy +``` + +After running deploy, you should see output similar to: + +```bash +Serverless: Packaging service... +Serverless: Excluding development dependencies... +Serverless: Creating Stack... +Serverless: Checking Stack create progress... +........ +Serverless: Stack create finished... +Serverless: Uploading CloudFormation file to S3... +Serverless: Uploading artifacts... +Serverless: Uploading service aws-node-express-dynamodb-api.zip file to S3 (718.53 KB)... +Serverless: Validating template... +Serverless: Updating Stack... +Serverless: Checking Stack update progress... +.................................... +Serverless: Stack update finished... +Service Information +service: aws-node-express-dynamodb-api +stage: dev +region: us-east-1 +stack: aws-node-express-dynamodb-api-dev +resources: 13 +api keys: + None +endpoints: + ANY - https://xxxxxxx.execute-api.us-east-1.amazonaws.com/ +functions: + api: aws-node-express-dynamodb-api-dev-api +layers: + None +``` + +_Note_: In current form, after deployment, your API is public and can be invoked by anyone. For production deployments, you might want to configure an authorizer. For details on how to do that, refer to [`httpApi` event docs](https://www.serverless.com/framework/docs/providers/aws/events/http-api/). Additionally, in current configuration, the DynamoDB table will be removed when running `serverless remove`. To retain the DynamoDB table even after removal of the stack, add `DeletionPolicy: Retain` to its resource definition. + +### Invocation + +After successful deployment, you can create a new user by calling the corresponding endpoint: + +```bash +curl --request POST 'https://xxxxxx.execute-api.us-east-1.amazonaws.com/users' --header 'Content-Type: application/json' --data-raw '{"name": "John", "userId": "someUserId"}' +``` + +Which should result in the following response: + +```bash +{"userId":"someUserId","name":"John"} +``` + +You can later retrieve the user by `userId` by calling the following endpoint: + +```bash +curl https://xxxxxxx.execute-api.us-east-1.amazonaws.com/users/someUserId +``` + +Which should result in the following response: + +```bash +{"userId":"someUserId","name":"John"} +``` + +If you try to retrieve user that does not exist, you should receive the following response: + +```bash +{"error":"Could not find user with provided \"userId\""} +``` + +### Local development + +It is also possible to emulate DynamoDB, API Gateway and Lambda locally using the `serverless-dynamodb-local` and `serverless-offline` plugins. In order to do that, run: + +```bash +serverless plugin install -n serverless-dynamodb-local +serverless plugin install -n serverless-offline +``` + +It will add both plugins to `devDependencies` in `package.json` file as well as will add it to `plugins` in `serverless.yml`. Make sure that `serverless-offline` is listed as last plugin in `plugins` section: + +``` +plugins: + - serverless-dynamodb-local + - serverless-offline +``` + +You should also add the following config to `custom` section in `serverless.yml`: + +``` +custom: + (...) + dynamodb: + start: + migrate: true + stages: + - dev +``` + +Additionally, we need to reconfigure `AWS.DynamoDB.DocumentClient` to connect to our local instance of DynamoDB. We can take advantage of `IS_OFFLINE` environment variable set by `serverless-offline` plugin and replace: + +```javascript +const dynamoDbClient = new AWS.DynamoDB.DocumentClient(); +``` + +with the following: + +```javascript +const dynamoDbClientParams = {}; +if (process.env.IS_OFFLINE) { + dynamoDbClientParams.region = 'localhost' + dynamoDbClientParams.endpoint = 'http://localhost:8000' +} +const dynamoDbClient = new AWS.DynamoDB.DocumentClient(dynamoDbClientParams); +``` + +After that, running the following command with start both local API Gateway emulator as well as local instance of emulated DynamoDB: + +```bash +serverless offline start +``` + +To learn more about the capabilities of `serverless-offline` and `serverless-dynamodb-local`, please refer to their corresponding GitHub repositories: +- https://github.com/dherault/serverless-offline +- https://github.com/99x/serverless-dynamodb-local diff --git a/idempotent-aws-node-express-dynamodb-api/handler.js b/idempotent-aws-node-express-dynamodb-api/handler.js new file mode 100644 index 000000000..ba204ca52 --- /dev/null +++ b/idempotent-aws-node-express-dynamodb-api/handler.js @@ -0,0 +1,80 @@ +const AWS = require("aws-sdk"); +const express = require("express"); +const serverless = require("serverless-http"); + +const app = express(); + +const USERS_TABLE = process.env.USERS_TABLE; +const dynamoDbClient = new AWS.DynamoDB.DocumentClient(); + +app.use(express.json()); + +app.get("/users/:userId", async function (req, res) { + const params = { + TableName: USERS_TABLE, + Key: { + userId: req.params.userId, + }, + }; + + try { + const { Item } = await dynamoDbClient.get(params).promise(); + if (Item) { + const { userId, name } = Item; + res.json({ userId, name }); + } else { + res + .status(404) + .json({ error: 'Could not find user with provided "userId"' }); + } + } catch (error) { + console.log(error); + res.status(500).json({ error: "Could not retreive user" }); + } +}); + +app.post("/users", async function (req, res) { + const { userId, name } = req.body; + if (typeof userId !== "string") { + res.status(400).json({ error: '"userId" must be a string' }); + } else if (typeof name !== "string") { + res.status(400).json({ error: '"name" must be a string' }); + } + + const params = { + ClientRequestToken: req.context.awsRequestId, + TransactItems: [ + { + Update: { + TableName: USERS_TABLE, + Key: { userId: userId }, + UpdateExpression: 'set #a = :v', + ExpressionAttributeNames: {'#a' : 'name'}, + ExpressionAttributeValues: { + ':v': name + } + } + } + ] + }; + try { + await dynamoDbClient.transactWrite(params).promise(); + res.json({ userId, name }); + } catch (error) { + console.log(error); + res.status(500).json({ error: "Could not create user" }); + } +}); + +app.use((req, res, next) => { + return res.status(404).json({ + error: "Not Found", + }); +}); + + +module.exports.handler = serverless(app,{ + request: function(req, _event, context) { + req.context = context; + } +}); diff --git a/idempotent-aws-node-express-dynamodb-api/package.json b/idempotent-aws-node-express-dynamodb-api/package.json new file mode 100644 index 000000000..2e1c7cd4f --- /dev/null +++ b/idempotent-aws-node-express-dynamodb-api/package.json @@ -0,0 +1,9 @@ +{ + "name": "aws-node-express-dynamodb-api", + "version": "1.0.0", + "description": "", + "dependencies": { + "express": "^4.17.1", + "serverless-http": "^2.7.0" + } +} diff --git a/idempotent-aws-node-express-dynamodb-api/serverless.template.yml b/idempotent-aws-node-express-dynamodb-api/serverless.template.yml new file mode 100644 index 000000000..520752083 --- /dev/null +++ b/idempotent-aws-node-express-dynamodb-api/serverless.template.yml @@ -0,0 +1,6 @@ +name: aws-node-express-dynamodb-api +org: serverlessinc +description: Deploys a Node Express API service backed by DynamoDB with Serverless Framework +keywords: aws, serverless, faas, lambda, node, express, dynamodb +repo: https://github.com/serverless/examples/aws-node-express-dynamodb-api +license: MIT diff --git a/idempotent-aws-node-express-dynamodb-api/serverless.yml b/idempotent-aws-node-express-dynamodb-api/serverless.yml new file mode 100644 index 000000000..c103746a0 --- /dev/null +++ b/idempotent-aws-node-express-dynamodb-api/serverless.yml @@ -0,0 +1,45 @@ +service: aws-node-express-dynamodb-api +frameworkVersion: '2' + +custom: + tableName: 'users-table-${sls:stage}' + +provider: + name: aws + runtime: nodejs12.x + lambdaHashingVersion: '20201221' + iam: + role: + statements: + - Effect: Allow + Action: + - dynamodb:Query + - dynamodb:Scan + - dynamodb:GetItem + - dynamodb:PutItem + - dynamodb:UpdateItem + - dynamodb:DeleteItem + Resource: + - Fn::GetAtt: [ UsersTable, Arn ] + environment: + USERS_TABLE: ${self:custom.tableName} + +functions: + api: + handler: handler.handler + events: + - httpApi: '*' + +resources: + Resources: + UsersTable: + Type: AWS::DynamoDB::Table + Properties: + AttributeDefinitions: + - AttributeName: userId + AttributeType: S + KeySchema: + - AttributeName: userId + KeyType: HASH + BillingMode: PAY_PER_REQUEST + TableName: ${self:custom.tableName} From 52c1e4234884ccff026807a54bcf974415e8c412 Mon Sep 17 00:00:00 2001 From: Nsupyq <93358967+Nsupyq@users.noreply.github.com> Date: Fri, 5 Nov 2021 11:08:51 +0800 Subject: [PATCH 2/3] Update package.json Add example name of idempotent-aws-node-express-dynamodb-api --- idempotent-aws-node-express-dynamodb-api/package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/idempotent-aws-node-express-dynamodb-api/package.json b/idempotent-aws-node-express-dynamodb-api/package.json index 2e1c7cd4f..daaf32c51 100644 --- a/idempotent-aws-node-express-dynamodb-api/package.json +++ b/idempotent-aws-node-express-dynamodb-api/package.json @@ -1,5 +1,5 @@ { - "name": "aws-node-express-dynamodb-api", + "name": "idempotent-aws-node-express-dynamodb-api", "version": "1.0.0", "description": "", "dependencies": { From 6272d80be72a364a6d5914fa816dad3c25a77199 Mon Sep 17 00:00:00 2001 From: Nsupyq <93358967+Nsupyq@users.noreply.github.com> Date: Fri, 5 Nov 2021 18:04:10 +0800 Subject: [PATCH 3/3] Introduce idempotence in README --- .../README.md | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/idempotent-aws-node-express-dynamodb-api/README.md b/idempotent-aws-node-express-dynamodb-api/README.md index 9f3d4df75..026e75b7a 100644 --- a/idempotent-aws-node-express-dynamodb-api/README.md +++ b/idempotent-aws-node-express-dynamodb-api/README.md @@ -11,13 +11,35 @@ authorName: 'Serverless, inc.' authorAvatar: 'https://avatars1.githubusercontent.com/u/13742415?s=200&v=4' --> -# Serverless Framework Node Express API on AWS with Idempotence Guaranteed +# Serverless Framework Node Express API on AWS with Idempotence Guarantee This template demonstrates how to add idempotence in a simple Node Express API service, backed by DynamoDB database, running on AWS Lambda using the traditional Serverless Framework. It is based on the example of [Serverless Framework Node Express API on AWS](../aws-node-express-dynamodb-api/README.md). +## What is idempotence + +Idempotence means multiple invocations of a function have the same side-effect as one invocation. + +## Why we need idempotence + +AWS Lambda uses retry to perform fault tolerance. +When your function fails because of out of memory or some other reasons, it will be directly retried until it finishes successfully. +For serverless functions with side-effect, retry may cause data inconsistency. +For example, retrying a function purchasing a product may cause multiple deduction of money. +Therefore, AWS Lambda requires programmers to write [idempotent function](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-idempotent/). + ## Anatomy of the template -This template configures a single function, `api`, which is responsible for handling all incoming requests thanks to the `httpApi` event. To learn more about `httpApi` event configuration options, please refer to [httpApi event docs](https://www.serverless.com/framework/docs/providers/aws/events/http-api/). As the event is configured in a way to accept all incoming requests, `express` framework is responsible for routing and handling requests internally. Implementation takes advantage of `serverless-http` package, which allows you to wrap existing `express` applications. To learn more about `serverless-http`, please refer to corresponding [GitHub repository](https://github.com/dougmoscrop/serverless-http). Additionally, it also handles provisioning of a DynamoDB database that is used for storing data about users. It uses context from the request as the `clientRequestToken` parameter in the [transactional write api](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html) of DynamoDB to enable the guarantee of idempotence. The `express` application exposes two endpoints, `POST /users` and `GET /user/{userId}`, which allow to create and retrieve users. +This template configures a single function, `api`, which is responsible for handling all incoming requests thanks to the `httpApi` event. To learn more about `httpApi` event configuration options, please refer to [httpApi event docs](https://www.serverless.com/framework/docs/providers/aws/events/http-api/). As the event is configured in a way to accept all incoming requests, `express` framework is responsible for routing and handling requests internally. Implementation takes advantage of `serverless-http` package, which allows you to wrap existing `express` applications. To learn more about `serverless-http`, please refer to corresponding [GitHub repository](https://github.com/dougmoscrop/serverless-http). Additionally, it also handles provisioning of a DynamoDB database that is used for storing data about users. The `express` application exposes two endpoints, `POST /users` and `GET /user/{userId}`, which allow to create and retrieve users. + +## How to guarantee idempotence + +The side effect of the function is writing the username. +AWS DynamoDB provides an [idempotent transactional write API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html), which writes name for only once and does nothing on retry. +It needs an argument `clientRequestToken` to check whether the current transactional write is a retry. +The `clientRequestToken` should be a universally unique identifier. +Then we use [`awsRequestId`](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html) to be the `clientRequestToken`. +This is a unique identifier provided by AWS Lambda. +When Lambda retries a function, it will use the same `awsRequestId` as that in the first invocation. ## Usage