Hello World Protocol

Workspace Environment

% uname -a
Linux 3c5d38601884 4.14.62-70.117.amzn2.x86_64 #1 SMP Fri Aug 10 20:14:53 UTC 2018 x86_64 GNU/Linux

% cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

% whoami
root

Prerequisites

% docker-machine --version
docker-machine version 0.14.0, build 89b8332

% aws --version
aws-cli/1.15.80 Python/2.7.13 Linux/4.14.62-70.117.amzn2.x86_64 botocore/1.10.79

Prepare AWS User

Create User

Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:*",
                "sts:DecodeAuthorizationMessage",
                "ec2:*"
            ],
            "Resource": "*"
        }
    ]
}

// TODO: more restricted permissions

Configure your shell with this user's credentials.

% aws configure
AWS Access Key ID [None]: ****************KQMQ # <- What you got
AWS Secret Access Key [None]: ****************wwzG # <- What you got
Default region name [None]: ap-south-1
Default output format [None]: json
% 
% aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************KQMQ shared-credentials-file
secret_key     ****************wwzG shared-credentials-file
    region               ap-south-1      config-file    ~/.aws/config
%

Seems you are all up.

Get hotsub

Install hotsub binary.

% export HOTSUB_VERSION=v0.5.0
% wget -q https://github.com/otiai10/hotsub/releases/download/${HOTSUB_VERSION}/hotsub.linux_amd64.tar.gz
% tar -xzvf hotsub.linux_amd64.tar.gz

# If you want
% mv ./hotsub /usr/local/bin

Check it

% ./hotsub --version
% ./hotsub --help

Write your first job!

% mkdir -p ~/tmp/hello-world
% cd ~/tmp/hello-world

% vi hello.sh

./hello.sh

#!/bin/sh

echo "Hello! My name is ${NAME}! I love ${FAV}!"

% vi hello.csv

./hello.csv

--env NAME, --env FAV
Hiromu OCHIAI, sushi

Submit it!

% ./hotsub run \
  --script ./hello.sh \
  --tasks ./hello.csv \
  --aws-region ap-south-1 \
  --verbose

Then you will get

2018/08/17 04:16:57 [COMMAND]	Your tasks file is parsed and decoded to 1 job(s) ✅
2018/08/17 04:16:57 [COMMAND]	See logs here -> /root/tmp/hello-world/log/20180817_041657
[hello.csv 0]	[CREATE]	Creating computing instance for this job...
[hello.csv 0]	[CONSTRUCT]	Constructing containers for this job...
[hello.csv 0]	[CONSTRUCT]	Constructing workflow container inside the computing instance...
[hello.csv 0]	[CONSTRUCT]	Constructing routine container inside the computing instance...
[hello.csv 0]	[EXECUTE]	&1> Hello! My name is Hiromu OCHIAI! I love sushi!
[hello.csv 0]	[DESTROY]	Terminating computing instance for this job...
2018/08/17 04:19:50 [COMMAND]	All of your 1 job(s) are completed 🎉

Let's play with hello.csv

Add your name and favorite things to the last line, like this

--env NAME, --env FAV
Hiromu OCHIAI, sushi
Ritsu TAINAKA, music and drums

then, hit the same command

% ./hotsub run \
  --script ./hello.sh \
  --tasks ./hello.csv \
  --aws-region ap-south-1 \
  --verbose

you are supposed to see these 2 lines for 2 data,

[hello.csv 0]	[EXECUTE]	&1> Hello! My name is Hiromu OCHIAI! I love sushi!
[hello.csv 1]	[EXECUTE]	&1> Hello! My name is Ritsu TAINAKA! I love music and drums!