Support flink rest api (#14)

* Replace the Flink cli actions with REST calls

* Flink rest api alter docker base (#13)

* Replace the Flink cli actions with REST calls

* Build on top of latest alpine image and remove setup for Flink CLI

* Remove unneeded env vars

* Use different env vars for Flink host/port

* Remove env vars from dockerfile and list in README

* Replace the Flink cli actions with REST calls

* Refactor all the things

* Alter ing.com to proper github.com/ing-bank location

* Support testing all packages and report coverage in travis

* Adjust README to use test-with-coverage script

* Update README with new update command, set deploy as default for compose

* Use the correct PATCH API to terminate the job

* Method comments and linting happiness

* Allow for specifying a savepoint-dir when deploying

* Do not deploy when updating a job which isn't running

* Move some example commands to separate markdown doc

* Add the savepoint-dir as a CLI option for deploy

.gitignore

@@ -1,4 +1,4 @@
 cover.out
-cli
 target/
 .idea
+/cli

.travis.yml

@@ -18,7 +18,7 @@ before_install:
 - docker build -t "nielsdenissen/flink-deployer:$TRAVIS_BRANCH" .
 
 script:
-- go test ./cmd/cli -coverprofile=cover.out
+- sh test-with-coverage.sh
 
 after_success:
 - bash <(curl -s https://codecov.io/bash)

Dockerfile

@@ -1,26 +1,10 @@
-FROM flink:1.5.1
+FROM alpine:3.8
 
-ENV HIGH_AVAILABILITY=none \
-    JOB_MANAGER_RPC_ADDRESS=localhost \
-    JOB_MANAGER_RPC_PORT=6123 \
-    JOB_MANAGER_WEB_ADDRESS=0.0.0.0 \
-    JOB_MANAGER_WEB_PORT=8081
-
-# Required to run our cli binary
-RUN apt-get update && \
-    apt-get install -y gettext vim musl
-
-COPY ./conf/flink-conf-template.yaml /flink-deployer/conf/flink-conf-template.yaml
-COPY docker-entrypoint.sh /flink-deployer/docker-entrypoint.sh
 COPY cli /flink-deployer/cli
 
-RUN mkdir -p /data/flink && \
-    chgrp -R root /opt/flink && \
-    chmod -R 775 /flink-deployer ${FLINK_HOME} /opt/flink
-
 WORKDIR /flink-deployer
 
 VOLUME [ "/data/flink" ]
 
-ENTRYPOINT [ "/flink-deployer/docker-entrypoint.sh" ]
+ENTRYPOINT [ "/flink-deployer/cli" ]
 CMD [ "help" ]

Gopkg.toml

@@ -32,3 +32,7 @@
 [[constraint]]
   name = "github.com/spf13/afero"
   version = "1.0.0"
+
+[[constraint]]
+  name = "github.com/cenkalti/backoff"
+  version = "2.0.0"

README.md

@@ -20,9 +20,10 @@ For a full overview of the commands and flags, run `flink-job-deployer help`
 
 To be able to test the deployer locally, follow these steps:
 
-1. ***optional***: `cd flink-sample-job; sbt clean assembly; cd ..` (Builds a jar with small stateful test job)
-2. `docker-compose up -d jobmanager taskmanager` (start a Flink job- and taskmanager)
-3. `docker-compose run deployer help` (run the Flink deployer with argument `help`)
+1. Build the CLI tool: `env GOOS=linux GOARCH=amd64 go build ./cmd/cli && docker-compose build deployer`
+2. ***optional***: `cd flink-sample-job; sbt clean assembly; cd ..` (Builds a jar with small stateful test job)
+3. `docker-compose up -d jobmanager taskmanager` (start a Flink job- and taskmanager)
+4. `docker-compose run deployer help` (run the Flink deployer with argument `help`)
 
 Repeat step 3 with any commands you'd like to try. 
 
@@ -43,19 +44,10 @@ This will print a simple word count to the output console, you can view it by ch
 docker-compose logs -f taskmanager
 ```
 
-Finally if you want to update the job, you can run:
-
-```bash
-docker-compose run deployer update 
-    -job-name-base "Windowed WordCount" 
-    -file-name "/tmp/flink-sample-job/flink-stateful-wordcount-assembly-0.jar" 
-    -run-args "-p 2 -d -c WordCountStateful" 
-    -jar-args "--intervalMs 1000" 
-    -savepoint-dir "/data/flink/savepoints"
-```
-
 If all went well you should see the word counter continue with where it was.
 
+A list of some example commands to run can be found [here](./docs/example-commands.md).
+
 # Development
 
 ## Managing dependencies
@@ -88,19 +80,27 @@ docker-compose build deployer
 ## Test
 
 ```bash
-go test
+go test ./cmd/cli ./cmd/cli/flink ./cmd/cli/operations
 ```
 
 Or with coverage:
 
 ```bash
-go test -coverprofile=cover.out && go tool cover
+sh test-with-coverage.sh
 ```
 
 # Docker
 
 A docker image for this repo is available from the docker hub: `nielsdenissen/flink-deployer`
 
+The image expects the following env vars:
+
+```bash
+FLINK_HOST=localhost
+FLINK_PORT=6123
+```
+
+
 # Kubernetes
 
 When running in Kubernetes (or Openshift), you'll have to deploy the container to the cluster. A reason for this is Flink will try to reroute you to the internal Kubernetes address of the cluster, which doesn't resolve from outside. Besides that it'll give you the necessary access to the stored savepoints when you're using persistent volumes to store those.
@@ -137,14 +137,10 @@ Here's an example of how such a kubernetes yaml could look like:
                 env:
                 -   name: FLINK_JOB_NAME_BASE
                     value: "${FLINK_JOB_NAME_BASE}"
-                -   name: JOB_MANAGER_RPC_ADDRESS
+                -   name: FLINK_PORT
                     value: "jobmanager"
-                -   name: JOB_MANAGER_RPC_PORT
+                -   name: FLINK_HOST
                     value: "8081"
-                -   name: HIGH_AVAILABILITY
-                    value: "zookeeper"
-                -   name: ZOOKEEPER_QUORUM
-                    value: "zookeeper:2181"
                 -   name: MAIN_CLASS_NAME
                     value: "${MAIN_CLASS_NAME}"
                 -   name: FLINK_JOB_ID