diff --git a/content/en/docs/setup/independent/high-availability.md b/content/en/docs/setup/independent/high-availability.md index ceff01d0ed9d3..a4edff600ae48 100644 --- a/content/en/docs/setup/independent/high-availability.md +++ b/content/en/docs/setup/independent/high-availability.md @@ -1,538 +1,522 @@ --- reviewers: -- mikedanese -- luxas -- errordeveloper -- jbeda -title: Creating HA clusters with kubeadm +- sig-cluster-lifecycle +title: Creating Highly Available Clusters with kubeadm content_template: templates/task --- {{% capture overview %}} -This guide shows you how to install and set up a highly available Kubernetes cluster using kubeadm. +This page explains two different approaches to setting up a highly available Kubernetes +cluster using kubeadm: -This document shows you how to perform setup tasks that kubeadm doesn't perform: provision hardware; configure multiple systems; and load balancing. +- With stacked masters. This approach requires less infrastructure. etcd members +and control plane nodes are co-located. +- With an external etcd cluster. This approach requires more infrastructure. The +control plane nodes and etcd members are separated. -{{< note >}} -**Note:** This guide is only one potential solution, and there are many ways to configure a highly available cluster. If a better solution works for you, please use it. If you find a better solution that can be adopted by the community, feel free to contribute it back. -{{< /note >}} +Your clusters must run Kubernetes version 1.11 or later. You should also be aware that +setting up HA clusters with kubeadm is still experimental. You might encounter issues +with upgrading your clusters, for example. We encourage you to try either approach, +and provide feedback. + +{{< caution >}} +**Caution**: This page does not address running your cluster on a cloud provider. +In a cloud environment, neither approach documented here works with Service objects +of type LoadBalancer, or with dynamic PersistentVolumes. +{{< /caution >}} {{% /capture %}} {{% capture prerequisites %}} -- Three machines that meet [kubeadm's minimum requirements](/docs/setup/independent/install-kubeadm/#before-you-begin) for the masters -- Three machines that meet [kubeadm's minimum requirements](/docs/setup/independent/install-kubeadm/#before-you-begin) for the workers -- **Optional:** At least three machines that meet [kubeadm's minimum requirements](/docs/setup/independent/install-kubeadm/#before-you-begin) - if you intend to host etcd on dedicated nodes (see information below) -- 1GB or more of RAM per machine (any less will leave little room for your apps) -- Full network connectivity between all machines in the cluster (public or - private network is fine) +For both methods you need this infrastructure: -{{% /capture %}} - -{{% capture steps %}} - -## Installing prerequisites on masters +- Three machines that meet [kubeadm's minimum + requirements](/docs/setup/independent/install-kubeadm/#before-you-begin) for + the masters +- Three machines that meet [kubeadm's minimum + requirements](/docs/setup/independent/install-kubeadm/#before-you-begin) for + the workers +- Full network connectivity between all machines in the cluster (public or + private network is fine) +- SSH access from one device to all nodes in the system +- sudo privileges on all machines -For each master that has been provisioned, follow the [installation guide](/docs/setup/independent/install-kubeadm/) on how to install kubeadm and its dependencies. At the end of this step, you should have all the dependencies installed on each master. +For the external etcd cluster only, you also need: -## Setting up an HA etcd cluster +- Three additional machines for etcd members -For highly available setups, you will need to decide how to host your etcd cluster. A cluster is composed of at least 3 members. We recommend one of the following models: +{{< note >}} +**Note**: The following examples run Calico as the Pod networking provider. If +you run another networking provider, make sure to replace any default values as +needed. +{{< /note >}} -- Hosting etcd cluster on separate compute nodes (Virtual Machines) -- Hosting etcd cluster on the master nodes. +{{% /capture %}} -While the first option provides more performance and better hardware isolation, it is also more expensive and requires an additional support burden. +{{% capture steps %}} -For **Option 1**: create 3 virtual machines that follow [CoreOS's hardware recommendations](https://coreos.com/etcd/docs/latest/op-guide/hardware.html). For the sake of simplicity, we will refer to them as `etcd0`, `etcd1` and `etcd2`. +## First steps for both methods -For **Option 2**: you can skip to the next step. Any reference to `etcd0`, `etcd1` and `etcd2` throughout this guide should be replaced with `master0`, `master1` and `master2` accordingly, since your master nodes host etcd. +{{< note >}} +**Note**: All commands in this guide on any control plane or etcd node should be +run as root. +{{< /note >}} -### Create etcd CA certs +- Find your pod CIDR. For details, see [the CNI network + documentation](/docs/setup/independent/create-cluster-kubeadm/#pod-network). + The example uses Calico, so the pod CIDR is `192.168.0.0/16`. -1. Install `cfssl` and `cfssljson` on all etcd nodes: +### Configure SSH - ```bash - curl -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 - curl -o /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 - chmod +x /usr/local/bin/cfssl* - ``` +1. Enable ssh-agent on your main device that has access to all other nodes in + the system: -1. SSH into `etcd0` and run the following: - - ```bash - mkdir -p /etc/kubernetes/pki/etcd - cd /etc/kubernetes/pki/etcd - - cat >ca-config.json <ca-csr.json <}}**Optional:** You can modify `ca-csr.json` to add a section for `names`. - See [the CFSSL wiki](https://github.com/cloudflare/cfssl/wiki/Creating-a-new-CSR) for an example. - {{< /note >}} +1. Add your SSH identity to the session: -1. Next, generate the CA certs: - ```bash - cfssl gencert -initca ca-csr.json | cfssljson -bare ca - ``` - -### Generate etcd client certs - -Generate the client certificates. While on `etcd0`, run the following: - -```bash -cat >client.json <" - ``` + - When you SSH to any node, make sure to add the `-A` flag: - Make sure to replace `` with your email, a placeholder, or an empty string. Keep hitting enter until files exist in `~/.ssh`. - -1. Output the contents of the public key file for `etcd1` and `etcd2`: - ```bash - cat ~/.ssh/id_rsa.pub - ``` - -1. Finally, copy the output for each and paste them into `etcd0`'s `~/.ssh/authorized_keys` file. This will permit `etcd1` and `etcd2` to SSH in to the machine. - -### Generate etcd server and peer certs - -1. In order to generate certs, each etcd machine needs the root CA generated by `etcd0`. On `etcd1` and `etcd2`, run the following: - ```bash - mkdir -p /etc/kubernetes/pki/etcd - cd /etc/kubernetes/pki/etcd - scp root@:/etc/kubernetes/pki/etcd/ca.pem . - scp root@:/etc/kubernetes/pki/etcd/ca-key.pem . - scp root@:/etc/kubernetes/pki/etcd/client.pem . - scp root@:/etc/kubernetes/pki/etcd/client-key.pem . - scp root@:/etc/kubernetes/pki/etcd/ca-config.json . - ``` - - Where `` corresponds to the public or private IPv4 of `etcd0`. + ``` + ssh -A 10.0.0.7 + ``` -1. Once this is done, run the following on all etcd machines: - ```bash - cfssl print-defaults csr > config.json - sed -i '0,/CN/{s/example\.net/'"$PEER_NAME"'/}' config.json - sed -i 's/www\.example\.net/'"$PRIVATE_IP"'/' config.json - sed -i 's/example\.net/'"$PEER_NAME"'/' config.json - ``` - ```bash - cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server config.json | cfssljson -bare server - cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer config.json | cfssljson -bare peer - ``` + - When using sudo on any node, make sure to preserve the environment so SSH + forwarding works: - The above will replace the default configuration with your machine's hostname as the peer name, and its IP addresses. Make sure - these are correct before generating the certs. If you found an error, reconfigure `config.json` and re-run the `cfssl` commands. + ``` + sudo -E -s + ``` -This results in the following files: `peer.pem`, `peer-key.pem`, `server.pem`, `server-key.pem`. +### Create load balancer for kube-apiserver -### {{< tabs name="etcd_mode" >}} -{{% tab name="Choose one..." %}} -Please select one of the tabs to see installation instructions for the respective way to set up a virtual IP. -{{% /tab %}} -{{% tab name="systemd" %}} -1. First, install etcd binaries: - ```bash - ETCD_VERSION="v3.1.12" curl -sSL https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz | tar -xzv --strip-components=1 -C /usr/local/bin/ - ``` +{{< note >}} +**Note**: There are many configurations for load balancers. The following +example is only one option. Your cluster requirements may need a +different configuration. +{{< /note >}} - It is worth noting that etcd v3.1.12 is the preferred version for Kubernetes v1.10. For other versions of Kubernetes please consult [the changelog](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md). +1. Create a kube-apiserver load balancer with a name that resolves to DNS. + + - In a cloud environment you should place your control plane nodes behind a TCP + forwarding load balancer. This load balancer distributes traffic to all + healthy control plane nodes in its target list. The health check for + an apiserver is a TCP check on the port the kube-apiserver listens on + (default value `:6443`). + + - It is not recommended to use an IP address directly in a cloud environment. + + - The load balancer must be able to communicate with all control plane nodes + on the apiserver port. It must also allow incoming traffic on its + listening port. + +1. Add the first control plane nodes to the load balancer and test the + connection: + + ```sh + nc -v LOAD_BALANCER_IP PORT + ``` + + - A connection refused error is expected because the apiserver is not yet + running. A timeout, however, means the load balancer cannot communicate + with the control plane node. If a timeout occurs, reconfigure the load + balancer to communicate with the control plane node. + +1. Add the remaining control plane nodes to the load balancer target group. + +## Stacked control plane nodes + +### Bootstrap the first stacked control plane node + +1. Create a `kubeadm-config.yaml` template file: + + apiVersion: kubeadm.k8s.io/v1alpha2 + kind: MasterConfiguration + kubernetesVersion: v1.11.0 + apiServerCertSANs: + - "LOAD_BALANCER_DNS" + api: + controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" + etcd: + local: + extraArgs: + listen-client-urls: "https://127.0.0.1:2379,https://CP0_IP:2379" + advertise-client-urls: "https://CP0_IP:2379" + listen-peer-urls: "https://CP0_IP:2380" + initial-advertise-peer-urls: "https://CP0_IP:2380" + initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380" + serverCertSANs: + - CP0_HOSTNAME + - CP0_IP + peerCertSANs: + - CP0_HOSTNAME + - CP0_IP + networking: + # This CIDR is a Calico default. Substitute or remove for your CNI provider. + podSubnet: "192.168.0.0/16" + + +1. Replace the following variables in the template with the appropriate + values for your cluster: + + * `LOAD_BALANCER_DNS` + * `LOAD_BALANCER_PORT` + * `CP0_HOSTNAME` + * `CP0_IP` + +1. Run `kubeadm init --config kubeadm-config.yaml` + +### Copy required files to other control plane nodes + +The following certificates and other required files were created when you ran `kubeadm init`. +Copy these files to your other control plane nodes: + +- `/etc/kubernetes/pki/ca.crt` +- `/etc/kubernetes/pki/ca.key` +- `/etc/kubernetes/pki/sa.key` +- `/etc/kubernetes/pki/sa.pub` +- `/etc/kubernetes/pki/front-proxy-ca.crt` +- `/etc/kubernetes/pki/front-proxy-ca.key` +- `/etc/kubernetes/pki/etcd/ca.crt` +- `/etc/kubernetes/pki/etcd/ca.key` + +Copy the admin kubeconfig to the other control plane nodes: + +- `/etc/kubernetes/admin.conf` + +In the following example, replace +`CONTROL_PLANE_IPS` with the IP addresses of the other control plane nodes. + +```sh +USER=ubuntu # customizable +CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8" +for host in ${CONTROL_PLANE_IPS}; do + scp /etc/kubernetes/pki/ca.crt "${USER}"@$host: + scp /etc/kubernetes/pki/ca.key "${USER}"@$host: + scp /etc/kubernetes/pki/sa.key "${USER}"@$host: + scp /etc/kubernetes/pki/sa.pub "${USER}"@$host: + scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host: + scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host: + scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt + scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key + scp /etc/kubernetes/admin.conf "${USER}"@$host: +done +``` - Also, please realize that most distributions of Linux already have a version of etcd installed, so you will be replacing the system default. +{{< note >}} +**Note**: Remember that your config may differ from this example. +{{< /note >}} -1. Next, generate the environment file that systemd will use: - ```bash - touch /etc/etcd.env - echo "PEER_NAME=${PEER_NAME}" >> /etc/etcd.env - echo "PRIVATE_IP=${PRIVATE_IP}" >> /etc/etcd.env - ``` +### Add the second stacked control plane node + +1. Create a second, different `kubeadm-config.yaml` template file: + + apiVersion: kubeadm.k8s.io/v1alpha2 + kind: MasterConfiguration + kubernetesVersion: v1.11.0 + apiServerCertSANs: + - "LOAD_BALANCER_DNS" + api: + controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" + etcd: + local: + extraArgs: + listen-client-urls: "https://127.0.0.1:2379,https://CP1_IP:2379" + advertise-client-urls: "https://CP1_IP:2379" + listen-peer-urls: "https://CP1_IP:2380" + initial-advertise-peer-urls: "https://CP1_IP:2380" + initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380,CP1_HOSTNAME=https://CP1_IP:2380" + initial-cluster-state: existing + serverCertSANs: + - CP1_HOSTNAME + - CP1_IP + peerCertSANs: + - CP1_HOSTNAME + - CP1_IP + networking: + # This CIDR is a calico default. Substitute or remove for your CNI provider. + podSubnet: "192.168.0.0/16" + +1. Replace the following variables in the template with the appropriate values for your cluster: + + - `LOAD_BALANCER_DNS` + - `LOAD_BALANCER_PORT` + - `CP0_HOSTNAME` + - `CP0_IP` + - `CP1_HOSTNAME` + - `CP1_IP` + +1. Move the copied files to the correct locations: + + ```sh + USER=ubuntu # customizable + mkdir -p /etc/kubernetes/pki/etcd + mv /home/${USER}/ca.crt /etc/kubernetes/pki/ + mv /home/${USER}/ca.key /etc/kubernetes/pki/ + mv /home/${USER}/sa.pub /etc/kubernetes/pki/ + mv /home/${USER}/sa.key /etc/kubernetes/pki/ + mv /home/${USER}/front-proxy-ca.crt /etc/kubernetes/pki/ + mv /home/${USER}/front-proxy-ca.key /etc/kubernetes/pki/ + mv /home/${USER}/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt + mv /home/${USER}/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key + mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf + ``` -1. Now copy the systemd unit file: - ```none - cat >/etc/systemd/system/etcd.service < --data-dir /var/lib/etcd --listen-client-urls http://localhost:2379 --advertise-client-urls http://localhost:2379 --listen-peer-urls http://localhost:2380 --initial-advertise-peer-urls http://localhost:2380 --cert-file=/etc/kubernetes/pki/etcd/server.pem --key-file=/etc/kubernetes/pki/etcd/server-key.pem --client-cert-auth --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem --peer-client-cert-auth --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem --initial-cluster =https://:2380,=https://:2380,=https://:2380 --initial-cluster-token my-etcd-token --initial-cluster-state new - - [Install] - WantedBy=multi-user.target - EOF - ``` +1. Run the kubeadm phase commands to bootstrap the kubelet: - Make sure you replace ``, `` and `` with the appropriate IPv4 addresses. Replace `` with the name of this etcd member. Modify the values of `--listen-client-urls`, `--advertise-client-urls`, `--listen-peer-urls` and `--initial-advertise-peer-urls` if needed. Replace ``, `` and `` with real hostnames of each machine. These machines must be able to reach every other using DNS or make sure that records are added to `/etc/hosts`. + ```sh + kubeadm alpha phase certs all --config kubeadm-config.yaml + kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml + kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml + kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml + systemctl start kubelet + ``` -1. Finally, launch etcd: - ```bash - systemctl daemon-reload - systemctl start etcd - ``` +1. Run the commands to add the node to the etcd cluster: -1. Check that it launched successfully: - ```bash - systemctl status etcd - ``` + ```sh + CP0_IP=10.0.0.7 + CP0_HOSTNAME=cp0 + CP1_IP=10.0.0.8 + CP1_HOSTNAME=cp1 -{{% /tab %}} -{{% tab name="Static Pods" %}} -**Note**: This is only supported on nodes that have the all dependencies for the kubelet installed. If you are hosting etcd on the master nodes, this has already been set up. If you are hosting etcd on dedicated nodes, you should either use systemd or run the [installation guide](/docs/setup/independent/install-kubeadm/) on each dedicated etcd machine. - -Run the following to generate the manifest file: - - - cat >/etc/kubernetes/manifests/etcd.yaml < - namespace: kube-system - spec: - containers: - - command: - - etcd --name - - --data-dir /var/lib/etcd - - --listen-client-urls http://localhost:2379 - - --advertise-client-urls http://localhost:2379 - - --listen-peer-urls http://localhost:2380 - - --initial-advertise-peer-urls http://localhost:2380 - - --cert-file=/certs/server.pem - - --key-file=/certs/server-key.pem - - --client-cert-auth - - --trusted-ca-file=/certs/ca.pem - - --peer-cert-file=/certs/peer.pem - - --peer-key-file=/certs/peer-key.pem - - --peer-client-cert-auth - - --peer-trusted-ca-file=/certs/ca.pem - - --initial-cluster etcd0=https://:2380,etcd1=https://:2380,etcd2=https://:2380 - - --initial-cluster-token my-etcd-token - - --initial-cluster-state new - image: k8s.gcr.io/etcd-amd64:3.1.10 - livenessProbe: - httpGet: - path: /health - port: 2379 - scheme: HTTP - initialDelaySeconds: 15 - timeoutSeconds: 15 - name: etcd - env: - - name: PUBLIC_IP - valueFrom: - fieldRef: - fieldPath: status.hostIP - - name: PRIVATE_IP - valueFrom: - fieldRef: - fieldPath: status.podIP - - name: PEER_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - volumeMounts: - - mountPath: /var/lib/etcd - name: etcd - - mountPath: /certs - name: certs - hostNetwork: true - volumes: - - hostPath: - path: /var/lib/etcd - type: DirectoryOrCreate - name: etcd - - hostPath: - path: /etc/kubernetes/pki/etcd - name: certs - EOF - -Make sure you replace: -* `` with the name of the node you're running on (e.g. `etcd0`, `etcd1` or `etcd2`) -* ``, `` and `` with the public IPv4s of the other machines that host etcd. -{{% /tab %}} -{{< /tabs >}} - - -## {{< tabs name="lb_mode" >}} -{{% tab name="Choose one..." %}} -Please select one of the tabs to see installation instructions for the respective way to set up a virtual IP. -{{% /tab %}} -{{% tab name="Cloud" %}} -Some examples of cloud provider solutions are: - -* [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/) -* [GCE Load Balancing](https://cloud.google.com/compute/docs/load-balancing/) -* [Azure](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview) - -You will need to ensure that the load balancer routes to **just `master0` on port 6443**. This is because kubeadm will perform health checks using the load balancer IP. Since `master0` is set up individually first, the other masters will not have running apiservers, which will result in kubeadm hanging indefinitely. - -If possible, use a smart load balancing algorithm like "least connections", and use health checks so unhealthy nodes can be removed from circulation. Most providers will provide these features. -{{% /tab %}} -{{% tab name="On-Site" %}} -In an on-site environment there may not be a physical load balancer available. Instead, a virtual IP pointing to a healthy master node can be used. There are a number of solutions for this including keepalived, Pacemaker and probably many others, some with and some without load balancing. - -As an example we outline a simple setup based on keepalived. Depending on environment and requirements people may prefer different solutions. The configuration shown here provides an _active/passive_ failover without load balancing. If required, load balancing can by added quite easily by setting up HAProxy, NGINX or similar on the master nodes (not covered in this guide). - -1. Install keepalived, e.g. using your distribution's package manager. The configuration shown here works with version `1.3.5` but is expected to work with may other versions. Make sure to have it enabled (chkconfig, systemd, ...) so that it starts automatically when the respective node comes up. - -2. Create the following configuration file `/etc/keepalived/keepalived.conf` on all master nodes: - ```none - ! Configuration File for keepalived - global_defs { - router_id LVS_DEVEL - } - - vrrp_script check_apiserver { - script "/etc/keepalived/check_apiserver.sh" - interval 3 - weight -2 - fall 10 - rise 2 - } - - vrrp_instance VI_1 { - state - interface - virtual_router_id 51 - priority - authentication { - auth_type PASS - auth_pass 4be37dc3b4c90194d1600c483e10ad1d - } - virtual_ipaddress { - - } - track_script { - check_apiserver - } - } - ``` + KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380 + kubeadm alpha phase etcd local --config kubeadm-config.yaml + ``` - In the section `vrrp_instance VI_1`, change few lines depending on your setup: + - This command causes the etcd cluster to become unavailable for a + brief period, after the node is added to the running cluster, and before the + new node is joined to the etcd cluster. - * `state` is either `MASTER` (on the first master nodes) or `BACKUP` (the other master nodes). - * `interface` is the name of an existing public interface to bind the virtual IP to (usually the primary interface). - * `priority` should be higher for the first master node, e.g. 101, and lower for the others, e.g. 100. - * `auth_pass` use any random string here. - * `virtual_ipaddresses` should contain the virtual IP for the master nodes. +1. Deploy the control plane components and mark the node as a master: -3. Install the following health check script to `/etc/keepalived/check_apiserver.sh` on all master nodes: - ```bash - #!/bin/sh + ```sh + kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml + kubeadm alpha phase controlplane all --config kubeadm-config.yaml + kubeadm alpha phase mark-master --config kubeadm-config.yaml + ``` - errorExit() { - echo "*** $*" 1>&2 - exit 1 - } +### Add the third stacked control plane node + +1. Create a third, different `kubeadm-config.yaml` template file: + + apiVersion: kubeadm.k8s.io/v1alpha2 + kind: MasterConfiguration + kubernetesVersion: v1.11.0 + apiServerCertSANs: + - "LOAD_BALANCER_DNS" + api: + controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" + etcd: + local: + extraArgs: + listen-client-urls: "https://127.0.0.1:2379,https://CP2_IP:2379" + advertise-client-urls: "https://CP2_IP:2379" + listen-peer-urls: "https://CP2_IP:2380" + initial-advertise-peer-urls: "https://CP2_IP:2380" + initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380,CP1_HOSTNAME=https://CP1_IP:2380,CP2_HOSTNAME=https://CP2_IP:2380" + initial-cluster-state: existing + serverCertSANs: + - CP2_HOSTNAME + - CP2_IP + peerCertSANs: + - CP2_HOSTNAME + - CP2_IP + networking: + # This CIDR is a calico default. Substitute or remove for your CNI provider. + podSubnet: "192.168.0.0/16" + +1. Replace the following variables in the template with the appropriate values for your cluster: + + - `LOAD_BALANCER_DNS` + - `LOAD_BALANCER_PORT` + - `CP0_HOSTNAME` + - `CP0_IP` + - `CP1_HOSTNAME` + - `CP1_IP` + - `CP2_HOSTNAME` + - `CP2_IP` + +1. Move the copied files to the correct locations: + + ```sh + USER=ubuntu # customizable + mkdir -p /etc/kubernetes/pki/etcd + mv /home/${USER}/ca.crt /etc/kubernetes/pki/ + mv /home/${USER}/ca.key /etc/kubernetes/pki/ + mv /home/${USER}/sa.pub /etc/kubernetes/pki/ + mv /home/${USER}/sa.key /etc/kubernetes/pki/ + mv /home/${USER}/front-proxy-ca.crt /etc/kubernetes/pki/ + mv /home/${USER}/front-proxy-ca.key /etc/kubernetes/pki/ + mv /home/${USER}/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt + mv /home/${USER}/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key + mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf + ``` - curl --silent --max-time 2 --insecure https://localhost:6443/ -o /dev/null || errorExit "Error GET https://localhost:6443/" - if ip addr | grep -q ; then - curl --silent --max-time 2 --insecure https://:6443/ -o /dev/null || errorExit "Error GET https://:6443/" - fi - ``` +1. Run the kubeadm phase commands to bootstrap the kubelet: - Replace the `` by your chosen virtual IP. + ```sh + kubeadm alpha phase certs all --config kubeadm-config.yaml + kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml + kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml + kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml + systemctl start kubelet + ``` -4. Restart keepalived. While no Kubernetes services are up yet it will log health check fails on all master nodes. This will stop as soon as the first master node has been bootstrapped. -{{% /tab %}} -{{< /tabs >}} +1. Run the commands to add the node to the etcd cluster: + ```sh + CP0_IP=10.0.0.7 + CP0_HOSTNAME=cp0 + CP2_IP=10.0.0.9 + CP2_HOSTNAME=cp2 -## Acquire etcd certs + KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP2_HOSTNAME} https://${CP2_IP}:2380 + kubeadm alpha phase etcd local --config kubeadm-config.yaml + ``` -Only follow this step if your etcd is hosted on dedicated nodes (**Option 1**). If you are hosting etcd on the masters (**Option 2**), you can skip this step since you've already generated the etcd certificates on the masters. +1. Deploy the control plane components and mark the node as a master: -1. Generate SSH keys for each of the master nodes by following the steps in the [create ssh access](#create-ssh-access) section. After doing this, each master will have an SSH key in `~/.ssh/id_rsa.pub` and an entry in `etcd0`'s `~/.ssh/authorized_keys` file. + ```sh + kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml + kubeadm alpha phase controlplane all --config kubeadm-config.yaml + kubeadm alpha phase mark-master --config kubeadm-config.yaml + ``` -1. Run the following: - ```bash - mkdir -p /etc/kubernetes/pki/etcd - scp root@:/etc/kubernetes/pki/etcd/ca.pem /etc/kubernetes/pki/etcd - scp root@:/etc/kubernetes/pki/etcd/client.pem /etc/kubernetes/pki/etcd - scp root@:/etc/kubernetes/pki/etcd/client-key.pem /etc/kubernetes/pki/etcd - ``` +## External etcd -## Run kubeadm init on master0 {#kubeadm-init-master0} - -1. In order for kubeadm to run, you first need to write a configuration file: - - - cat >config.yaml < - controlPlaneEndpoint: - etcd: - endpoints: - - https://:2379 - - https://:2379 - - https://:2379 - caFile: /etc/kubernetes/pki/etcd/ca.pem - certFile: /etc/kubernetes/pki/etcd/client.pem - keyFile: /etc/kubernetes/pki/etcd/client-key.pem - networking: - podSubnet: - apiServerCertSANs: - - - - - apiServerExtraArgs: - apiserver-count: "3" - EOF - -Ensure that the following placeholders are replaced: - -- `` with the private IPv4 of the master server. -- ``, `` and `` with the IP addresses of your three etcd nodes -- `` with your Pod CIDR. Please read the [CNI network section](/docs/setup/independent/create-cluster-kubeadm/#pod-network) of the docs for more information. Some CNI providers do not require a value to be set. -- `` with the virtual IP. Please read [setting up a master load balancer](/docs/setup/independent/high-availability/#set-up-master-load-balancer) section of the docs for more information. - -{{< note >}}**Note:** If you are using Kubernetes 1.9+, you can replace the `apiserver-count: 3` extra argument with `endpoint-reconciler-type: lease`. For more information, see [the documentation](/docs/admin/high-availability/#endpoint-reconciler).{{< /note >}} - -1. When this is done, run kubeadm: - ```bash - kubeadm init --config=config.yaml - ``` +### Set up the cluster -## Run kubeadm init on master1 and master2 +- Follow [these instructions](/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm/) + to set up the etcd cluster. -Before running kubeadm on the other masters, you need to first copy the K8s CA cert from `master0`. To do this, you have two options: +### Copy required files to other control plane nodes -### Option 1: Copy with scp +The following certificates were created when you created the cluster. Copy them +to your other control plane nodes: -1. Follow the steps in the [create ssh access](#create-ssh-access) section, but instead of adding to `etcd0`'s `authorized_keys` file, add them to `master0`. -1. Once you've done this, run: +- `/etc/kubernetes/pki/etcd/ca.crt` +- `/etc/kubernetes/pki/apiserver-etcd-client.crt` +- `/etc/kubernetes/pki/apiserver-etcd-client.key` - ```bash - scp root@:/etc/kubernetes/pki/* /etc/kubernetes/pki - rm /etc/kubernetes/pki/apiserver* - ``` +In the following example, replace `USER` and `CONTROL_PLANE_HOSTS` values with values +for your environment. -### Option 2: Copy paste +```sh +USER=ubuntu +CONTROL_PLANE_HOSTS="10.0.0.7 10.0.0.8 10.0.0.9" +for host in $CONTROL_PLANE_HOSTS; do + scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host: + scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${USER}"@$host: + scp /etc/kubernetes/pki/apiserver-etcd-client.key "${USER}"@$host: +done +``` -Copy the contents of `/etc/kubernetes/pki/ca.crt`, `/etc/kubernetes/pki/ca.key`, `/etc/kubernetes/pki/sa.key` and `/etc/kubernetes/pki/sa.pub` and create these files manually on `master1` and `master2`. +### Set up the first control plane node + +1. Create a `kubeadm-config.yaml` template file: + + apiVersion: kubeadm.k8s.io/v1alpha2 + kind: MasterConfiguration + kubernetesVersion: v1.11.0 + apiServerCertSANs: + - "LOAD_BALANCER_DNS" + api: + controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" + etcd: + external: + endpoints: + - https://ETCD_0_IP:2379 + - https://ETCD_1_IP:2379 + - https://ETCD_2_IP:2379 + caFile: /etc/kubernetes/pki/etcd/ca.crt + certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt + keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key + networking: + # This CIDR is a calico default. Substitute or remove for your CNI provider. + podSubnet: "192.168.0.0/16" + +1. Replace the following variables in the template with the appropriate values for your cluster: + + - `LOAD_BALANCER_DNS` + - `LOAD_BALANCER_PORT` + - `ETCD_0_IP` + - `ETCD_1_IP` + - `ETCD_2_IP` + +1. Run `kubeadm init --config kubeadm-config.yaml` + +### Copy required files to the correct locations + +The following certificates and other required files were created when you ran `kubeadm init`. +Copy these files to your other control plane nodes: + +- `/etc/kubernetes/pki/ca.crt` +- `/etc/kubernetes/pki/ca.key` +- `/etc/kubernetes/pki/sa.key` +- `/etc/kubernetes/pki/sa.pub` +- `/etc/kubernetes/pki/front-proxy-ca.crt` +- `/etc/kubernetes/pki/front-proxy-ca.key` + +In the following example, replace the list of +`CONTROL_PLANE_IPS` values with the IP addresses of the other control plane nodes. + +```sh +USER=ubuntu # customizable +CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8" +for host in ${CONTROL_PLANE_IPS}; do + scp /etc/kubernetes/pki/ca.crt "${USER}"@CONTROL_PLANE_IP: + scp /etc/kubernetes/pki/ca.key "${USER}"@CONTROL_PLANE_IP: + scp /etc/kubernetes/pki/sa.key "${USER}"@CONTROL_PLANE_IP: + scp /etc/kubernetes/pki/sa.pub "${USER}"@CONTROL_PLANE_IP: + scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@CONTROL_PLANE_IP: + scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@CONTROL_PLANE_IP: +done +``` -When this is done, you can follow the [previous step](#kubeadm-init-master0) to install the control plane with kubeadm. +{{< note >}} +**Note**: Remember that your config may differ from this example. +{{< /note >}} -## Add master1 and master2 to load balancer +### Set up the other control plane nodes -Once kubeadm has provisioned the other masters, you can add them to the load balancer pool. +1. Verify the location of the copied files. + Your `/etc/kubernetes` directory should look like this: -## Install CNI network + - `/etc/kubernetes/pki/apiserver-etcd-client.crt` + - `/etc/kubernetes/pki/apiserver-etcd-client.key` + - `/etc/kubernetes/pki/ca.crt` + - `/etc/kubernetes/pki/ca.key` + - `/etc/kubernetes/pki/front-proxy-ca.crt` + - `/etc/kubernetes/pki/front-proxy-ca.key` + - `/etc/kubernetes/pki/sa.key` + - `/etc/kubernetes/pki/sa.pub` + - `/etc/kubernetes/pki/etcd/ca.crt` -Follow the instructions [here](/docs/setup/independent/create-cluster-kubeadm/#pod-network) to install the pod network. Make sure this corresponds to whichever pod CIDR you provided in the master configuration file. +1. Run `kubeadm init --config kubeadm-config.yaml` on each control plane node, where + `kubeadm-config.yaml` is the file you already created. -## Install workers +## Common tasks after bootstrapping control plane -Next provision and set up the worker nodes. To do this, you will need to provision at least 3 Virtual Machines. +### Install a pod network -1. To configure the worker nodes, [follow the same steps](/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes) as non-HA workloads. +[Follow these instructions](/docs/setup/independent/create-cluster-kubeadm/#pod-network) to install +the pod network. Make sure this corresponds to whichever pod CIDR you provided +in the master configuration file. -## Configure workers +### Install workers -1. Reconfigure kube-proxy to access kube-apiserver via the load balancer: - ```bash - kubectl get configmap -n kube-system kube-proxy -o yaml > kube-proxy-cm.yaml - sed -i 's#server:.*#server: https://:6443#g' kube-proxy-cm.yaml - kubectl apply -f kube-proxy-cm.yaml --force - # restart all kube-proxy pods to ensure that they load the new configmap - kubectl delete pod -n kube-system -l k8s-app=kube-proxy - ``` +Each worker node can now be joined to the cluster with the command returned from any of the +`kubeadm init` commands. -1. Reconfigure the kubelet to access kube-apiserver via the load balancer: - ```bash - sudo sed -i 's#server:.*#server: https://:6443#g' /etc/kubernetes/kubelet.conf - sudo systemctl restart kubelet - ``` +{{% /capture %}}