Bare metal scripts and documentation

[hyperbolic2346]

I'm interested in trying paz, but I have an existing coreos cluster on bare metal. I assume I just need to wget some unit files to pull down and run paz, but I don't see anything in the documentation about this.

It seems as simple as clone the repo and run

scripts/start-runlevel.sh 1 && scripts/start-runlevel.sh 2

but I would expect some documentation if it were that simple. Is the documentation just missing for bare metal?

[jacyzon]

Bare metal machine is most likely not feed in variable like $private_ipv4 and $public_ipv4, if you like to run PAZ on bare metal, the cloud-init need a little tweak.

• Use https://github.com/kelseyhightower/setup-network-environment to set up network environment file.

• Note that the unitfiles for the sidekick service, use /etc/environment as environment file, the file does not exist on bare metal machine, you may simply create the file at /etc/environment with following info:

  COREOS_PRIVATE_IPV4=YOUR_PRIVATE_IPV4
  COREOS_PUBLIC_IPV4=YOUR_PUBLIC_IPV4

and hope following cloud-config will do the trick:

#cloud-config

ssh_authorized_keys:
  - ssh-rsa YOUR_PUB_KEY
coreos:
  update:
    reboot-strategy: off
  units:
    - name: setup-network-environment.service
      command: start
      content: |
        [Unit]
        Description=Setup Network Environment
        Documentation=https://github.com/kelseyhightower/setup-network-environment
        Requires=network-online.target
        After=network-online.target
        [Service]
        ExecStartPre=-/usr/bin/mkdir -p /opt/bin
        ExecStartPre=/usr/bin/wget -N -P /opt/bin https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment
        ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment
        ExecStart=/opt/bin/setup-network-environment
        RemainAfterExit=yes
        Type=oneshot
    - name: etcd.service
      command: start
      content: |
        [Unit]
        Description=etcd
        Requires=setup-network-environment.service
        After=setup-network-environment.service
        [Service]
        User=etcd
        PermissionsStartOnly=true
        EnvironmentFile=/etc/network-environment
        ExecStart=/usr/bin/etcd \
        --name ${DEFAULT_IPV4} \
        --addr ${DEFAULT_IPV4}:4001 \
        --data-dir /var/lib/etcd \
        --peer-addr ${DEFAULT_IPV4}:7001 \
        --discovery https://discovery.etcd.io/NEW_TOKEN #generate a new token for each unique cluster from https://discovery.etcd.io/new
        Restart=always
        RestartSec=10s
    - name: fleet.service
      command: start
    - name: docker.service
      drop-ins:
        - name: 50-docker-dns.conf
          content: |
            [Service]
            Environment='DOCKER_OPTS=--restart=false -D --dns=172.17.42.1 --dns=8.8.8.8'
    - name: create-swap.service
      command: start
      runtime: true
      content: |
        [Unit]
        Description=Create swap file
        Before=swap.service
        [Service]
        Type=oneshot
        Environment="SWAPFILE=/2GiB.swap"
        ExecStart=/usr/bin/touch ${SWAPFILE}
        ExecStart=/usr/bin/chattr +C ${SWAPFILE}
        ExecStart=/usr/bin/fallocate -l 2048m ${SWAPFILE}
        ExecStart=/usr/bin/chmod 600 ${SWAPFILE}
        ExecStart=/usr/sbin/mkswap ${SWAPFILE}
        [Install]
        WantedBy=multi-user.target
    - name: swap.service
      command: start
      content: |
        [Unit]
        Description=Turn on swap
        [Service]
        Type=oneshot
        Environment="SWAPFILE=/2GiB.swap"
        RemainAfterExit=true
        ExecStartPre=/usr/sbin/losetup -f ${SWAPFILE}
        ExecStart=/usr/bin/sh -c "/sbin/swapon $(/usr/sbin/losetup -j ${SWAPFILE} | /usr/bin/cut -d : -f 1)"
        ExecStop=/usr/bin/sh -c "/sbin/swapoff $(/usr/sbin/losetup -j ${SWAPFILE} | /usr/bin/cut -d : -f 1)"
        ExecStopPost=/usr/bin/sh -c "/usr/sbin/losetup -d $(/usr/sbin/losetup -j ${SWAPFILE} | /usr/bin/cut -d : -f 1)"
        [Install]
        WantedBy=multi-user.target
    - name: cadvisor.service
      runtime: true
      command: start
      content: |
        [Unit]
        Description=Analyzes resource usage and performance characteristics of running containers.
        After=docker.service
        Requires=docker.service
        [Service]
        Restart=always
        ExecStartPre=/usr/bin/docker pull google/cadvisor:latest
        ExecStartPre=-/bin/bash -c " \
            docker inspect cadvisor >/dev/null 2>&1 \
            && docker rm -f cadvisor || true"
        ExecStart=/usr/bin/docker run --volume=/var/run:/var/run:rw --volume=/sys/fs/cgroup/:/sys/fs/cgroup:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --name=cadvisor google/cadvisor:latest
        ExecStop=/usr/bin/docker rm -f cadvisor
    - name: paz-dnsmasq.service
      runtime: true
      command: start
      content: |
        [Unit]
        Description=*.paz traffic will go to the private_ipv4 addr
        After=docker.service
        Requires=docker.service
        After=etcd.service
        Requires=etcd.service
        After=fleet.service
        Requires=fleet.service
        [Service]
        Restart=always
        EnvironmentFile=/etc/network-environment
        ExecStartPre=/usr/bin/docker pull tomgco/dnsmasq-catch:latest
        ExecStartPre=-/bin/bash -c " \
            docker inspect paz-dnsmasq >/dev/null 2>&1 \
            && docker rm -f paz-dnsmasq || true"
        ExecStart=/usr/bin/docker run -p ${DEFAULT_IPV4}:53:53/udp --privileged --name=paz-dnsmasq tomgco/dnsmasq-catch paz ${DEFAULT_IPV4}
        ExecStop=/usr/bin/docker rm -f paz-dnsmasq
    - name: paz-haproxy.service
      runtime: true
      command: start
      content: |
        [Unit]
        Description=paz HAProxy instance that enables service discovery.
        After=docker.service
        Requires=docker.service
        After=etcd.service
        Requires=etcd.service
        After=fleet.service
        Requires=fleet.service
        [Service]
        User=core
        Restart=always
        RestartSec=5s
        EnvironmentFile=/etc/network-environment
        ExecStartPre=/usr/bin/docker pull tomgco/dnsmasq-catch:latest
        ExecStartPre=/usr/bin/docker pull quay.io/yldio/paz-haproxy:latest
        ExecStartPre=-/bin/bash -c " \
          docker inspect paz-haproxy >/dev/null 2>&1 \
          && docker rm -f paz-haproxy || true"
        ExecStart=/usr/bin/docker run -p 80:80 -p 1936:1936 -e ETCD=${DEFAULT_IPV4}:4001 --name paz-haproxy quay.io/yldio/paz-haproxy
        ExecStop=/usr/bin/docker rm -f paz-haproxy
        TimeoutStartSec=20m
    - name: paz-pubkey-watcher.service
      runtime: true
      command: start
      content: |
        [Unit]
        Description=Watch etcd for scheduler public key changes and update authorized_hosts.
        After=etcd.service
        Requires=etcd.service
        After=fleet.service
        Requires=fleet.service
        [Service]
        User=core
        Restart=always
        ExecStartPre=/home/core/bin/paz-pubkey-watcher.sh once
        ExecStart=/home/core/bin/paz-pubkey-watcher.sh
write_files:
  - path: /etc/paz-environment
    permissions: 0644
    content: |
      PAZ_ORCHESTRATOR_DNS_DISABLED=true
      PAZ_DOMAIN=
  - path: /home/core/bin/paz-pubkey-watcher.sh
    owner: core
    permissions: 0754
    content: |
      #!/bin/bash
      set -e
      if [[ "$1" == "once" ]]; then
        FN=`mktemp /tmp/paz-pubkey.XXXX`
        until etcdctl get /paz/config/scheduler/_pubkey 2>/dev/null > $FN.tmp; do sleep 2; done && base64 -d < $FN.tmp > $FN && /usr/bin/update-ssh-keys -u core -a paz-scheduler $FN
        rm $FN $FN.tmp
      else
        while :; do
          FN=`mktemp /tmp/paz-pubkey.XXXX`
          etcdctl watch /paz/config/scheduler/_pubkey | base64 -d > $FN && /usr/bin/update-ssh-keys -u core -a paz-scheduler $FN;
          rm $FN
        done;
      fi
  - path: /etc/sysctl.d/swap.conf
    permissions: 0644
    owner: root
    content: |
     vm.swappiness=10
     vm.vfs_cache_pressure=50

[tomgco]

I am going to be playing around with a PXE coreos / paz installation tonight, if I bump into anything then I will post my results.

[hyperbolic2346]

Be sure to check out the latest documentation and let us know how it goes.

[lukebond]

@hyperbolic2346 where are we on this issue atm? would we say the "bare metal scripts and documentation" issue is ready to close?

[hyperbolic2346]

@lukebond I think initial documentation is good to go. From here I think we can rely on new issues brought up to fix things if something is wrong.

[lukebond]

Thanks @hyperbolic2346!