Skip to content

Latest commit

 

History

History
183 lines (140 loc) · 14 KB

chapter04.md

File metadata and controls

183 lines (140 loc) · 14 KB

<< Back

4. Component Level Architecture

scope

Table of Contents

4.1 Introduction

This chapter will describe in detail the Kubernetes Reference Architecture in terms of the functional capabilities and how they relate to the Reference Model requirements, i.e. how the infrastructure profiles are delivered and described.

Figure 4-1 below shows the architectural components that are described in the subsequent sections of this chapter.

Kubernetes Reference Architecture

Figure 4-1: Kubernetes Reference Architecture

4.2 Host OS

In order for a Host OS to be compliant with this Reference Architecture it must meet the following requirements:

  • A deb/rpm compatible distribution of Linux (this must be used for the master nodes, and can be used for worker nodes).
  • A version of the Linux kernel that is compatible with kubeadm - this has been chosen as the baseline because kubeadm is focussed on installing and managing the lifecycle of Kubernetes and nothing else, hence it is easily integrated into higher-level and more complete tooling for the full lifecycle management of the infrastructure, cluster add-ons, etc.
  • Windows Server 2019 (this can be used for worker nodes, but be aware of the limitations).
  • In order to support req.gen.cnt.03 (immutable infrastructure), the Host OS must be disposable, meaning the configuration of the Host OS (and associated infrastructure such as VM or bare metal server) must be consistent - e.g. the system software and configuration of that software must be identical apart from those areas of configuration that must be different such as IP addresses and hostnames.
  • This approach to configuration management supports req.lcm.gen.01 (automated deployments)

Table 4-1 lists the Linux kernel versions that comply with this Reference Architecture specification.

OS Family Version(s) Notes
Linux 3.10+
Windows 1809 (10.0.17763) For worker nodes only

Table 4-1: Compliant OS Kernels

4.3 Kubernetes

This chapter should discuss:

  • The version of version range of Kubernetes and the mandatory components needed for Kubernetes (e.g.: etcd, cadvisor)
  • Which optional features are used and which optional API-s are available
  • Which alfa or beta features are used

In alignment with the Kubernetes version support policy, a Reference Implementation must use one of three latest minor versions (n-2) - e.g. if the latest version is 1.17 then the RI must use either 1.17, 1.16 or 1.15. The Kubernetes distribution, product, or installer used in the RI must be listed in the Kubernetes Distributions and Platforms document and marked (X) as conformant for the Kubernetes version that is being used.

This Reference Architecture also specifies:

  • Master nodes must run the following Kubernetes control plane services:
    • kube-apiserver
    • kube-scheduler
    • kube-controller-manager
  • Master nodes can also run the etcd service and host the etcd database, however etcd can also be hosted on separate nodes
  • In order to support req.gen.rsl.01, req.gen.rsl.02 and req.gen.avl.01 a Reference Implementation must:
    • Consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes)
    • At least one master node per availability zone or fault domain to ensure the high availability and resilience of Kubernetes control plane services
    • At least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes
  • Master node services, including etcd, and worker node services (e.g. consumer workloads) must be kept separate - i.e. there must be at least one master node, and at least one worker node
  • Workloads must not rely on the availability of the master nodes for the successful execution of their functionality (i.e. loss of the master nodes may affect non-functional behaviours such as healing and scaling, but components that are already running will continue to do so without issue)
  • The following kubelet features must be enabled
    • CPU Manager
    • Device Plugin
    • Topology Manager

All kubelet features can be enabled/disabled by using the feature-gates: section in the kubelet config file. e.g.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
feature-gates:
  CPUManager: true|false (BETA - default=true)
  DevicePlugins: true|false (BETA - default=true)
  TopologyManager: true|false (ALPHA - default=false)

4.4 Container runtimes

In order to support req.inf.com.03, the chosen runtime must be compliant with the Kubernetes Container Runtime Interface (CRI) and the Open Container Initiative (OCI) runtime spec. Examples of container runtimes that are compliant with these specification are (note this is not a complete list and in no particular order):

  • container-d (with CRI plugin enabled, which it is by default)
  • Docker CE (via the dockershim, which is currently built in to the kubelet)
  • CRI-O
  • Frakti

These specifications cover the full lifecycle of a container creating > created > running > stopped which includes the use of storage that is required during this lifecycle - this is management of the Host OS filesystem by the container runtime. This lifecycle management by the container runtime (when compliant with the above specifications) supports the requirement req.inf.stg.06 for ephemeral storage for Pods.

Todo: details and RA2 specifications relating to runtimes in order to meet RM features and requirements from RM chapters 4 and 5.

4.5 CNI plugins

Editors note: The following chapter lists a set of CNI plugins compliant with the Reference Architecture. In future releases the list of CNI plugins should be refined in a way that there is only component selected for each functionality.

The used CNI multiplexer/metapulgin may be DANM as it provides the possibility to use several other CNI plugins (req.inf.ntw.16) and provides an API based solution to administer the networks (req.inf.ntw.10) from a central point (req.inf.ntw.11).

The following table contains a comparision of relevant features and requirements in Multus and DANM.

Requirement Support in Multus Support in DANM
req.inf.ntw.01 Supported Supported
req.inf.ntw.02 Supported via an other CNI plugin Supported via an other CNI plugin
req.inf.ntw.03 Supported via an other CNI plugin Supported
req.inf.ntw.04 Supported via an other CNI plugin Supported via an other CNI plugin
req.inf.ntw.06 Supported Supported
req.inf.ntw.07 Supported Supported
req.inf.ntw.08 Supported Supported
req.inf.ntw.09 Supported via LCM tools Supported via LCM tools
req.inf.ntw.10 Not supported Suported
req.inf.ntw.11 Not supported Partially supported
req.inf.ntw.14 Supported via an other CNI plugin Supported via an other CNI plugin
req.inf.ntw.15 Not relevant Not relevant
req.inf.ntw.16 Supported Supported
Cluster wide IP address management Not suported Supported
Service based dicovery of all provisioned interfaces Not supported Supported

Calico may be used as the CNI what complies with the basic networking assumptions of Kubernetes based on the requirement req.inf.ntw.02 due to it's capability to handle NetworkPolicies, what is missing from Flannel. For the network of signalling connections the built in IPVLAN CNI of DANM or the MACVLAN CNI may be used as these provide NAT-less connectivity (req.inf.ntw.03). For the user plane network(s) fullfilling requirement req.inf.ntw.04 the User Space CNI may be used. The User Space CNI may use VPP or OVS-DPDK as a backend.

Editors note: The usage SR-IOV in container environments, therefore the inclusion of an SR-IOV CNI plugin and the SR-IOV Device Plugin to the architecture are under debate.

4.5 Storage components

As described in chapter 3, storage in Kubernetes consists of three types of storage:

  1. Ephemeral storage that is used to execute the containers
    • Ephemeral storage follows the lifecycle of a container
    • See the Container runtimes section above for more information how this meets the requirement req.inf.stg.06 for ephemeral storage for Pods
  2. Kubernetes Volumes, which are used to present additional storage to containers
    • A Volume follow the lifecycle of a Pod
    • This is a native Kubernetes capability and therefore req.inf.stg.01 is supported by default
    • This capability also delivers support for req.inf.stg.06 although depending on the Volume Plugin used there may be additional steps required in order to remove data from disk (not all plugins manage the full lifecycle of the storage mounted using Volumes)
  3. Kubernetes Persistent Volumes, which are a subset of the above whose lifecycle persists beyond the lifetime of a Pod to allow for data persistence
    • Persistent Volumes have a lifecycle that is independent of Containers and/or Pods
    • This supports the requirement req.inf.stg.07 for persistent storage for Pods

Volume plugins are used in Kubernetes to allow for the use of a range of backend storage systems. There are two types of Volume plugin:

  1. In-tree
    • These plugins are built, linked, compiled and shipped with the core Kubernetes binaries
    • Therefore if a new backend storage system needs adding this is a change to the core Kubernetes code
  2. Out-of-tree
    • These plugins allow new storage plugins to be created without any changes to the core Kubernetes code
    • The Container Storage Interface (CSI) is such an out-of-tree plugin and many in-tree drivers are being migrated to use the CSI plugin instead (e.g. the Cinder CSI plugin)
    • In order to support the requirement req.inf.stg.03 (CSI support), the following feature gates must be enabled:
      • CSIDriverRegistry
      • CSINodeInfo
    • In addition to these feature gates, a CSI driver must be used (as opposed to an in-tree volume plugin) - a full list of CSI drivers can be found here
    • In order to support ephemeral storage use through a CSI-compatible volume plugin, the CSIInlineVolume feature gate must be enabled
    • In order to support Persistent Volumes through a CSI-compatible volume plugin, the CSIPersistentVolume feature gate must be enabled

Should the following paragraph be moved to the Security chapter?

In order to support req.sec.gen.09 and more generally to support automation and the separation of concerns between providers of a service and consumers of the service, Kubernetes Storage Classes should be used. Storage Classes allow a consumer of the Kubernetes platform to request Persistent Storage using a Persistent Volume Claim and for a Persistent Volume to be dynamically created based on the "class" that has been requested. This avoids having to grant create/update/delete permissions in RBAC to PersistentVolume resources, which are cluster-scoped rather than namespace-scoped (meaning an identity can manage all PVs or none).

A note on object storage:

  • This Reference Architecture does not include any specifications for object storage, as this is neither a native Kubernetes object, nor something that is required by CSI drivers. Object storage is an application-level requirement that would ordinarily be provided by a highly scalable service offering rather than being something an individual Kubernetes cluster could offer.

Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest).

4.7 Service meshes

No service meshes are part of the architecture.

4.8 Kubernetes Application package manager

The reference architecture must support the usage of a Kubernetes Application package manager using the Kubernetes API-s, like Helm v3.

4.9 Supplementary components (okay, this is a bad heading, but I do not have any better)

This chapter should list all the supplementary components needed to provide the services defined in Chapter 3.2 (e.g: Prometheus)