# Overview [![star this repo](https://badgen.net/github/stars/feiskyer/kubernetes-handbook)](https://github.com/feiskyer/kubernetes-handbook) [![fork this repo](https://badgen.net/github/forks/feiskyer/kubernetes-handbook)](https://github.com/feiskyer/kubernetes-handbook/fork) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/feiskyer/kubernetes-handbook/issues) Kubernetes is an open-source container cluster management system developed by Google. It is the open-source version of Google's large-scale container management technology Borg and one of the most important projects of CNCF. Its main features include: * Container-based application deployment, maintenance, and rolling upgrades * Load balancing and service discovery * Cluster scheduling across machines and regions * Automatic scaling * Stateless services and stateful services * Extensive volume support * Plugin mechanism to ensure scalability Kubernetes has developed rapidly and has become a leader in the field of container orchestration. There is also a wealth of Chinese resources for Kubernetes, but there are relatively few systematic ones that keep up with community updates. The "Kubernetes Handbook" open-source e-book aims to organize reference guides and practice summaries for development and use of Kubernetes, forming a systematic reference guide for easy access. Everyone is welcome to follow and contribute. ## Online Reading \*GitBook: [kubernetes.feisky.xyz](https://kubernetes.feisky.xyz/) \* Github: [github.com/feiskyer/kubernetes-handbook](https://github.com/feiskyer/kubernetes-handbook/blob/master/SUMMARY.md) ## Project Source Code The project source code is stored on Github, . ## WeChat Scan the QR code to follow the WeChat public account, reply with keywords to view relevant chapters in WeChat. ![](/files/89T94pHgpb0wFs7C6opm) ## Contributors Welcome to participate in contributing content improvement methods refer t o \[CONTRIBUTING] ( https : // github .com / fei skyer/kubern etes-han dboo k/blob/mas ter/C ON TRIBU TIN G.m d). Thank you all contributors , contributor list see [contributors](https://github.com/feiskyer/kubernetes-handbook/graphs/contributors) . [![](https://opencollective.com/kubernetes-handbook/contributors.svg?width=890\&button=false)](https://github.com/feiskyer/kubernetes-handbook/graphs/contributors) ## LICENSE ![LICENSE](https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png) [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.zh)。 # Kubernetes Introduction Kubernetes is an open-source container orchestration system released by Google. It's essentially the open-source version of Google's Borg, which represents years of Google's expertise in large-scale container management. Its key capabilities include: * Applications deployment, maintenance, and rolling updates using containers * Load balancing and service discovery * Cluster scheduling across machines and regions * Autoscaling * Management of stateless and stateful services * Extensive support for volumes * Plug-in mechanism to ensure extensibility Kubernetes has evolved quickly and has become the leader in the field of container orchestration. ## Kubernetes as a Platform Kubernetes furnishes a plethora of functionalities that streamline the workflow of applications, thus accelerating development pace. Generally, a successful orchestration system needs robust automation, which is why Kubernetes was designed as a platform to build ecosystems of components and tools, making it easier to deploy, scale, and manage applications. Users can manage resources in their own way using Labels and customize resource descriptions with Annotations—for example, providing state checks to management tools. Moreover, Kubernetes Controllers are also built on the same APIs that are used by developers and users. One can write their own Controllers and Schedulers or expand the system's functionality with various plug-in mechanisms. This design makes it convenient to build a variety of application systems atop Kubernetes. ## What Kubernetes is Not Kubernetes is not your traditional, all-encompassing PaaS (Platform as a Service) system. It preserves the freedom of choice for its users. * It does not restrict the types of applications it supports; it doesn't meddle with application frameworks or predetermine supported languages (like Java, Python, Ruby, etc.), with the only requirement being the application should be [12-factor](http://12factor.net/) compliant. Kubernetes aims to support an extraordinarily diverse array of workloads, including stateless, stateful, and data-processing workloads. If an app can run in a container, it should thrive on Kubernetes. * It doesn't offer built-in middleware (like messaging), data-processing frameworks (like Spark), databases (like mysql), or cluster storage systems (like Ceph) - these apps run on top of Kubernetes. * It doesn’t provide a click-to-deploy service marketplace. * It doesn't deploy code or build your app, but you can build the necessary Continuous Integration (CI) workflows on Kubernetes. * It allows users to opt for their logging, monitoring, and alerting systems. * It doesn't provide an application configuration language or system (like [jsonnet](https://github.com/google/jsonnet)). * It doesn’t provide machine configuration, maintenance, management, or self-healing systems. Moreover, many PaaS systems operate atop Kubernetes, like [Openshift](https://github.com/openshift/origin), [Deis](http://deis.io/), and [Eldarion](http://eldarion.cloud/). You can build your own PaaS, or simply utilize Kubernetes to manage your container applications. Indeed, Kubernetes is more than just an "orchestration system"; it obviates the need for orchestration. Kubernetes maintains the desired state of applications through a declarative API and a series of independent, composable controllers, and users do not need to concern themselves with how intermediate states are achieved. This makes the system easier to use and more powerful, reliable, resilient, and scalable. ## Core Components The core components of Kubernetes include: * etcd holds the entire cluster state; * apiserver acts as the front-end to the cluster, providing mechanisms for authentication, authorization, access control, API registration, and discovery; * controller manager is responsible for maintaining the cluster's state—repairing perturbations, automatically scaling, rolling updates, etc.; * scheduler dispatches pods to suitable machines according to predefined scheduling policies; * kubelet is in charge of maintaining the container's lifecycle, and it also manages Volumes (CVI) and networking (CNI); * Container runtime is responsible for image management and the actual running of Pods and containers (CRI); * kube-proxy provides service discovery and load balancing within the cluster ![](/files/HBtRt9UPLBaryZgzbP5f) ## Kubernetes Versions Kubernetes stable versions are supported for 9 months after their release. The support cycle for each version is as follows: | Kubernetes version | Release month | End-of-life month | | :----------------: | :------------: | :---------------: | | v1.6.x | March 2017 | December 2017 | | v1.7.x | June 2017 | March 2018 | | v1.8.x | September 2017 | June 2018 | | v1.9.x | December 2017 | September 2018 | | v1.10.x | March 2018 | December 2018 | | v1.11.x | June 2018 | March 2019 | ## Reference Documents * [What is Kubernetes?](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) * [HOW CUSTOMERS ARE REALLY USING KUBERNETES](https://apprenda.com/blog/customers-really-using-kubernetes/) # Kubernetes Concepts ## Container A container is a lightweight, portable, OS-level virtualization technology that utilizes namespaces to isolate various software execution environments. Containers come with their runtime environments encapsulated within images, making them easily deployable anywhere. Due to the small size and rapid startup of containers, each container image can contain a single application—this one-to-one application-to-image relationship brings many benefits. Containers do not require binding to any external infrastructure since each application is self-contained, with no outside dependencies needed. This effectively solves the consistency problems from development all the way to production. Containers are also more transparent than virtual machines, aiding in their monitoring and management. Notably, the container life cycle is managed by the infrastructure, as opposed to being concealed within the container by a process manager. Lastly, managing container deployments is essentially managing application deployments. Other advantages of containers include: * Agile application creation and deployment: Container images are easier and more efficient to work with compared to virtual machine images. * Continuous development, integration, and deployment: Provide reliable and frequent container image build and deployment with quick and easy rollbacks (because images are immutable). * Dev and Ops separation of concerns: Create container images at build/release time, thus decoupling applications from infrastructure. * Environment consistency across development, testing, and production: Runs the same on a laptop as it does in the cloud. * Observability: Not only surface OS-level information and metrics but also application health and other metrics. * Portability across clouds and OS distributions: Run on Ubuntu, RHEL, CoreOS, on-prem, Google Kubernetes Engine, and anywhere else. * Application-centric management: Elevate from deploying OSes on hardware to running applications on an OS. * Loosely coupled, distributed, elastic, liberate microservices architectures: Applications are broken into smaller, independent chunks and can be managed dynamically—not monolithic stacks running on one big single-purpose machine. * Resource isolation: Predictable application performance. * Resource utilization: High efficiency and density. ## Pod Kubernetes employs Pods to orchestrate containers, with each Pod capable of holding one or more closely related containers. A Pod is a collection of containers that are tightly linked, sharing IPC and Network namespaces, and it is the basic unit that Kubernetes schedules. Containers within a Pod share network and file systems, which allows them to carry out service tasks through a simple and efficient method of inter-process communication and file sharing. ![pod](/files/iIzHUc1FOA0O44704zQM) In Kubernetes, all objects are defined using manifest files (yaml or json). For example, a simple nginx service can be defined in an nginx.yaml file containing a container with the nginx image: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 ``` ## Node A Node is the actual host where Pods run and can be either a physical or a virtual machine. To manage Pods, each Node has to run at least a container runtime such as docker or rkt, `kubelet`, and `kube-proxy` services. ![node](/files/uzvFpOqR2ksX0Dp17hFd) ## Namespace A Namespace is an abstract collection of a set of resources and objects, such as for separating internal system objects into different project or user groups. Common entities like pods, services, replication controllers, and deployments belong to a namespace (default is `default`), whereas nodes and persistentVolumes do not belong to any namespace. ## Service A Service is an abstraction that provides load balancing and service discovery for applications, using labels. A list of Pod IPs and ports matching the labels form the endpoints, which `kube-proxy` manages to load-balance across these endpoints. Each Service is automatically assigned a cluster IP (a virtual address accessible only within the cluster) and DNS name. Other containers can access the service using this address or DNS name, without needing to be aware of the backend containers running it. ![](/files/kR4slz1lJX7y3cwr3ajp) ```yaml apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - port: 8078 # the port that this service should serve on name: http # the container on each pod to connect to, can be a name # (e.g. 'www') or a number (e.g. 80) targetPort: 80 protocol: TCP selector: app: nginx ``` ## Label Labels are tags that identify Kubernetes objects, attached to objects in a key/value format (where key length cannot exceed 63 bytes, and value can be empty or a string not exceeding 253 bytes). Labels do not denote uniqueness and are often used by many objects (like Pods) to signify a particular application. Once labels are defined, other objects may use a Label Selector to select a set of objects with the same label (for example, ReplicaSets and Services use labels to select a set of Pods). Label Selector supports the following types: * Equality, like `app=nginx` and `env!=production` * Set-based, like `env in (production, qa)` * Multiple labels (which are ANDed together), like `app=nginx,env=test` ## Annotations Annotations are notes attached to objects in a key/value manner. Unlike labels, which are used to identify and select objects, annotations store additional information that assists with application deployment, security strategies, and scheduling policies, among others. For instance, deployments use annotations to keep track of the state of rolling updates. # Kubernetes 101 ## Kubernetes 101 The simplest way to experience Kubernetes is to run an nginx container and then use kubectl to manage it. Kubernetes offers a command similar to `docker run` called `kubectl run`, which conveniently creates a container (actually, it creates a Pod managed by a deployment): ```bash $ kubectl run --image=nginx:alpine nginx-app --port=80 deployment "nginx-app" created $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-app-4028413181-cnt1i 1/1 Running 0 52s ``` Once the container is up and running, you can operate it using `kubectl` commands, such as: * `kubectl get` - similar to `docker ps`, to query the list of resources * `kubectl describe` - similar to `docker inspect`, to get detailed information about a resource * `kubectl logs` - similar to `docker logs`, to get container logs * `kubectl exec` - similar to `docker exec`, to execute a command inside a container ```bash $ kubectl get pods ... $ kubectl exec nginx-app-4028413181-cnt1i -- ps aux ... $ kubectl describe pod nginx-app-4028413181-cnt1i ... $ curl http://172.17.0.3 ... $ kubectl logs nginx-app-4028413181-cnt1i ... ``` ### Defining a Pod with yaml Above, we launched our first Pod using `kubectl run`, but `kubectl run` does not support all functionalities. In Kubernetes, it's more common to define resources using yaml files and to create resources with `kubectl create -f file.yaml`. For example, a simple nginx Pod could be defined as: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 ``` Previously mentioned, `kubectl run` doesn't directly create a Pod; it first creates a Deployment resource (replicas=1), after which the associated ReplicaSet automatically creates a Pod. This is equivalent to the following configuration: ```yaml ... ``` ### Using Volume Pod lifecycles are typically short, and a new Pod is created to replace it as soon as any issues occur. What about data generated by containers? It vanishes along with the Pod's demise. Volumes exist for the sole purpose of persisting container data. For instance, you could specify a hostPath for a redis container to store data: ```yaml ... ``` Kubernetes volumes support various plugins, allowing selection based on actual needs: * emptyDir * hostPath * gcePersistentDisk * awsElasticBlockStore * nfs * iscsi * flocker * glusterfs * rbd * cephfs * gitRepo * secret * persistentVolumeClaim * downwardAPI * azureFileVolume * vsphereVolume ### Using Service Although we've created a Pod, in Kubernetes, it's ill-advised to interact directly with a Pod's IP address since it changes with Pod restarts. So how do we access the services offered by these Pods? By using Service. Service provides a unified entry for a set of Pods (selected via labels), offering load balancing and automatic service discovery. For example, you can create a service for the previous `nginx-app`: ```yaml $ kubectl expose deployment nginx-app --port=80 --target-port=80 --type=NodePort service "nginx-app" exposed $ kubectl describe service nginx-app ... ``` Now, inside the cluster, nginx-app can be accessed via `http://10.0.0.66` and `http://node-ip:30772`. From outside the cluster, only `http://node-ip:30772` is accessible. *** ## Diving into Kubernetes: An Introductory Guide Eager to get hands-on with Kubernetes? The easiest start is running an nginx container, then bossing it around with kubectl. Picture Kubernetes' `kubectl run` like a beefed-up `docker run`—it spins up a container, but behind the scenes, it's conjuring a Pod managed by a deployment: ```bash $ kubectl run --image=nginx:alpine nginx-app --port=80 deployment "nginx-app" created $ kubectl get pods ... ``` After our container hits the 'Running' stage, it's fair game for kubectl commands. Notable moves include: * `kubectl get` - peek at resources, kind of like peering at your running processes with `docker ps` * `kubectl describe` - snag that coveted detailed resource intel, much like `docker inspect` * `kubectl logs` - reel in those container logs, a la `docker logs` * `kubectl exec` - drop commands into your container, reminiscent of `docker exec` ```bash $ kubectl get pods ... $ kubectl exec nginx-app-4028413181-cnt1i -- ps aux ... $ kubectl describe pod nginx-app-4028413181-cnt1i ... $ curl http://172.17.0.3 ... $ kubectl logs nginx-app-4028413181-cnt1i ... ``` ### Crafting a Pod with yaml We kicked things off with `kubectl run`, which is easy but not all-powerful. For the full spectrum of control, yaml files are Kubernetes' blueprint of choice, paired with a `kubectl create -f file.yaml` incantation: ```yaml ... ``` Earlier we learned `kubectl run` isn't a straight shot to a Pod; it's more like a detour through Deployment Town. Your command shapes a Deployment (replicas=1), which then commands a ReplicaSet to whip up your Pod, echoing this yaml setup: ```yaml ... ``` ### Embracing Volumes Pods live fast and die young, any hiccup and *poof*, it's clone time. What of your container's precious data? Without volumes, it's dust. Enter volumes, your data's immortality charm. Say you're running redis—just anchor it with a hostPath, and your data's grounded: ```yaml ... ``` Got a specific storage fancy? Kubernetes has volume plugins aplenty to satisfy your whims: * emptyDir * hostPath * gcePersistentDisk * awsElasticBlockStore * nfs * iscsi * flocker * glusterfs * rbd * cephfs * gitRepo * secret * persistentVolumeClaim * downwardAPI * azureFileVolume * vsphereVolume ### Harnessing Service Crafting a Pod is a win, but in Kubernetes, nodding to a Pod's IP for chit-chats is a no-no—they're the chameleons of the IP world. Craving some consistency in your network? Service is your ticket. It rallies a pod posse (huddled under label umbrellas) providing a stalwart gateway, load balancing heroes, and service discovery sorcery. Need to expose `nginx-app` to adoring fans? Deploy a service: ```yaml $ kubectl expose deployment nginx-app --port=80 --target-port=80 --type=NodePort service "nginx-app" exposed $ kubectl describe service nginx-app ... ``` Inside the cluster, `http://10.0.0.66` or `http://node-ip:30772` are your golden tickets to nginx-app. From the great beyond? `http://node-ip:30772` is your sole portal. # Kubernetes 201 ## Kubernetes Mastery: Scaling and Upgrading ### Scaling Applications Scaling your application up or down in Kubernetes is as easy as adjusting the number of replicas in your Deployment: ![scale](/files/xOgSXeu9upLRnVdTNrKN) Containers that automatically scale up will join the service pool, and those that are scaled down will similarly be removed from the service pool with no manual intervention required. ```bash $ kubectl scale --replicas=3 deployment/nginx-app $ kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-app 3 3 3 3 10m ``` ### Rolling Updates Rolling updates allow for seamless application upgrades by replacing containers incrementally: ``` kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2 ``` ![update1](/files/lT1ByDdyoSzCR6xwiPi9) ![update2](/files/r7PCrpRlGLCwbvoaK7gS) ![update3](/files/ubYDq9fIDEVCnMCs7nqb) ![update4](/files/Jqm6Pww7ueqKtAw0ZXIi) If an update fails or there is a configuration mistake, you can revert changes on-the-go during a rolling update: ``` kubectl rolling-update frontend-v1 frontend-v2 --rollback ``` It’s important to note that `kubectl rolling-update` is specific to ReplicationController. For Deployments with an update strategy set to RollingUpdate (this is the default when specified in the spec), the application will be automatically updated in a rolling fashion: ```yaml spec: replicas: 3 selector: matchLabels: run: nginx-app strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate ``` For updating applications, the `kubectl set` command can be used directly: ```bash kubectl set image deployment/nginx-app nginx-app=nginx:1.9.1 ``` You can monitor the rolling update process using the `rollout` command: ```bash $ kubectl rollout status deployment/nginx-app Waiting for rollout to finish: 2 out of 3 new replicas have been updated... Waiting for rollout to finish: 2 of 3 updated replicas are available... Waiting for rollout to finish: 2 of 3 updated replicas are available... Waiting for rollout to finish: 2 of 3 updated replicas are available... Waiting for rollout to finish: 2 of 3 updated replicas are available... Waiting for rollout to finish: 2 of 3 updated replicas are available... deployment "nginx-app" successfully rolled out ``` Deployments also support rollback: ```bash $ kubectl rollout history deployment/nginx-app deployments "nginx-app" REVISION CHANGE-CAUSE 1 2 $ kubectl rollout undo deployment/nginx-app deployment "nginx-app" rolled back ``` ### Resource Constraints Through the use of cgroups, Kubernetes offers container resource management, which allows setting limits on CPU and memory usage for each container. For instance, you can restrict the nginx container from the aforementioned deployment to use a maximum of 50% CPU and 128MB of memory: ```bash $ kubectl set resources deployment nginx-app -c=nginx --limits=cpu=500m,memory=128Mi deployment "nginx" resource requirements updated ``` This is equivalent to setting resource limits in each Pod: ```yaml apiVersion: v1 kind: Pod metadata: labels: app: nginx name: nginx spec: containers: - image: nginx name: nginx resources: limits: cpu: "500m" memory: "128Mi" ``` ### Health Checks As a container orchestration tool designed for applications, Kubernetes needs to ensure that containers are truly running properly after being deployed. It provides two probes (Probe, supporting exec, tcpSocket, and httpGet) to detect the status of the containers: * LivenessProbe: Determines if an application is in a healthy state. If not, the container will be terminated and recreated. * ReadinessProbe: Determines if an application has started properly and is ready to service traffic. If it’s not ready, it will not receive traffic from Kubernetes Services. For deployments that are already up and running, manifest updates with health checks can be added using `kubectl edit deployment/nginx-app`: ```yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: app: nginx name: nginx-default spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx imagePullPolicy: Always name: http resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File resources: limits: cpu: "500m" memory: "128Mi" livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 15 timeoutSeconds: 1 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 timeoutSeconds: 1 ``` *** Now, let's refine the given content into a format that's more fitting for a popular science magazine. Here's how the translation and adaptation might look in that context: ## Exploring the Power of Kubernetes: Flex Your Cloud Muscles ### Supercharge Your Software with Easy Expansion Imagine a garden where plants grow or shrink at your command. That's what scaling in Kubernetes feels like! When your app is the plant, all you need to do is turn the dial for the number of replicas in your Deployment and watch it magically adjust: ![scale](/files/xOgSXeu9upLRnVdTNrKN) This bit of wizardry means your app's capacity grows or contracts as needed—no potion required. Just one command sets your desired state: ```bash $ kubectl scale --replicas=3 deployment/nginx-app $ kubectl get deploy ``` Voilà! Your once tiny app is now flexing like a tech titan. ### Non-Stop Improvement with Rolling Updates Gone are the days of down times and "Sorry, we're updating" signs. With rolling updates, your app gets a refresh, one piece at a time, without breaking a sweat (or your user's experience): ![update1](/files/lT1ByDdyoSzCR6xwiPi9) ... ![update4](/files/Jqm6Pww7ueqKtAw0ZXIi) And if something's amiss, just roll it back. It's like a time machine for your deployments! ``` kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2 ``` Easily glide into the future or rewind to the good ol' days—your choice, captain! ### Stay Healthy, Stay Happy: Kubernetes' Fitness Tracker for Apps Kubernetes doesn't just launch your containers—it makes sure they're in tip-top shape! With probes that are kind of like a fitness tracker for your apps, it ensures everything is up and running clean: * LivenessProbe: Keeps an eye on your app's health. Is it lagging? Time for a fresh start. * ReadinessProbe: Checks if your app is all set to welcome users. No traffic until it’s at 100% performance. These health checkups are just edits away to keep your apps in peak condition: ```yaml apiVersion: extensions/v1beta1 kind: Deployment ... livenessProbe: httpGet: path: / port: 80 ... readinessProbe: httpGet: path: / port: 80 ... ``` Stay fit and online—Kubernetes has your back. By adopting a playful tone and vivid analogies, we've transformed an academic explanation into an engaging story that captures the essence of Kubernetes’ features in a way that's enticing to the broader audience. # Kubernetes Cluster ## Inside a Kubernetes Cluster ![](/files/KUaQxKL07eE7ibzOtfu8) At the heart of a Kubernetes cluster, lies a harmonious blend of distributed storage, etcd, control nodes, and service nodes known as Nodes. * Control nodes are the orchestrators of the cluster, responsible for container scheduling, maintaining resource states, automatic scaling, and rolling updates. * Service nodes are the workhorses that run containers, managing images and containers, as well as handling service discovery and load balancing within the cluster. * An etcd cluster holds the entire state of the Kubernetes cluster. For a more detailed introduction, please refer to [Kubernetes Architecture](/en/concepts/architecture). ### Cluster Federation Cluster Federation (Federation) extends Kubernetes across multiple availability zones and is realized in conjunction with cloud service providers such as GCE and AWS. ![](/files/Fl3lf220fROFuaUjmWgN) For a more detailed introduction, please refer to [Federation](/en/concepts/components/federation). ### Setting Up a Kubernetes Cluster You can deploy a Kubernetes cluster by following the [Kubernetes Deployment Guide](/en/setup/index). For beginners or for simple validation tests, the following are easier methods. #### minikube The easiest way to create a Kubernetes cluster (single-node version) is with [minikube](https://github.com/kubernetes/minikube): ```bash $ minikube start Starting local Kubernetes cluster... Kubectl is now configured to use the cluster. $ kubectl cluster-info Kubernetes master is running at https://192.168.64.12:8443 kubernetes-dashboard is running at https://192.168.64.12:8443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` #### play-with-k8s [Play with Kubernetes](http://play-with-k8s.com) provides a free Kubernetes learning environment, giving you access to kubeadm to create clusters directly at . Be mindful that each created cluster can only be used for up to 4 hours. A handy feature of Play with Kubernetes is the automatic display of all NodePort type service ports on the page. Simply clicking on the port allows access to the corresponding service. For detailed usage, refer to [Play-With-Kubernetes](https://github.com/feiskyer/kubernetes-handbook/tree/549e0e3c9ba0175e64b2d4719b5a46e9016d532b/appendix/play-with-k8s.md). *** ## Discover the World of Kubernetes Clusters ![](/files/KUaQxKL07eE7ibzOtfu8) Delve into the ecosystem of a Kubernetes cluster, an intricate infrastructure comprised of distributed storage with etcd, central command centers known as control nodes, and the workstations called service nodes or nodes. * **Control nodes** like conductors in an orchestra, they manage the overall operations of the cluster, ensuring containers are aptly scheduled, resources statuses are up to date, scaling is done automatically, and updates roll out seamlessly. * **Service nodes** are the powerhouse of the cluster, directly hosting and handling containers, busying themselves with the management of images and containers, alongside the pivotal roles of service discovery and load balancing within the 'Kubernetes universe'. * The **etcd cluster’s** task is momentous, as it meticulously archives the entire state of the Kubernetes cluster. For a deep dive into this topic, check out the [Kubernetes Architecture](/en/concepts/architecture). ### Expanding Horizons with Cluster Federation Cluster Federation (Federation) is designed to scale Kubernetes across several availability zones, tightly integrating with cloud service giants like GCE and AWS. ![](/files/Fl3lf220fROFuaUjmWgN) Tackle more on this subject in the [Federation Overview](/en/concepts/components/federation). ### Crafting Your Kubernetes Cluster Embark on setting up your own Kubernetes cluster via the comprehensive [Kubernetes Deployment Guide](/en/setup/index). If you're just getting your feet wet or simply tinkering for testing purposes, here are some straightforward alternatives: #### minikube For a breezy introduction to a single-node Kubernetes cluster, [minikube](https://github.com/kubernetes/minikube) is your best bet: ```bash $ minikube start Starting local Kubernetes cluster... Kubectl is now configured to use the cluster. $ kubectl cluster-info Kubernetes master is running at https://192.168.64.12:8443 kubernetes-dashboard is running at https://192.168.64.12:8443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` #### play-with-k8s For a hands-on, no-cost experience with Kubernetes, [Play with Kubernetes](http://play-with-k8s.com) is the go-to platform. It allows you to dive in using kubeadm to craft clusters right there on , though remember each cluster exists for a fleeting 4 hours maximum. What's neat about Play with Kubernetes is its intuitive presentation of all NodePort type service ports, just a click away from accessing the services they lead to. Uncover tips and tricks in the realm of Play with Kubernetes by visiting [Play-With-Kubernetes](https://github.com/feiskyer/kubernetes-handbook/tree/549e0e3c9ba0175e64b2d4719b5a46e9016d532b/appendix/play-with-k8s.md). # Concepts Explore the architecture and key components of Kubernetes, which include: * [The Architectural Principles](/en/concepts/architecture) * [Design Philosophy](/en/concepts/concepts) * [Core Components](/en/concepts/components) * [etcd](/en/concepts/components/etcd) * [kube-apiserver](/en/concepts/components/apiserver) * [kube-scheduler](/en/concepts/components/scheduler) * [kube-controller-manager](/en/concepts/components/controller-manager) * [kubelet](/en/concepts/components/kubelet) * [kube-proxy](/en/concepts/components/kube-proxy) * [kube-dns](/en/concepts/index) * [Federation](/en/concepts/components/federation) * [kubeadm](/en/concepts/components/kubeadm) * [hyperkube](/en/concepts/components/hyperkube) * [kubectl](/en/concepts/components/kubectl) # Architecture Kubernetes, originally sourced from Google's in-house tool Borg, offers a container-centric, application-oriented cluster deployment, and management system. Its ultimate goal is to alleviate the burdensome orchestration of physical/virtual computing, networking, and storage infrastructure, allowing app operators and developers to focus fully on self-operation with container-centric primitives. Furthermore, Kubernetes offers a stable and compatible platform for crafting custom workflows and advanced automation tasks. Impressively, it possesses robust cluster management capabilities, including multilevel security defenses and admission mechanisms, support for multi-tenant apps, transparent service registration and discovery mechanisms, built-in load balancers, fault detection and self-repair mechanisms, service rolling upgrades and online expansion, scalable automatic resource scheduling mechanisms and multi-granularity resource quota management capabilities. Besides, Kubernetes features a comprehensive suite of management tools, covering the entire course of development, deployment testing, and operational monitoring. ## An Introduction to Borg Borg serves as Google's internal large-scale cluster management system, handling the scheduling and management of many of Google's core services. Borg aims to let users put aside resource management concerns, allowing them to concentrate on their primary businesses while maximizing resource use across multiple data centers. The Borg system mainly consists of BorgMaster, Borglet, borgcfg, and Scheduler, as shown in the following diagram ![borg](/files/5LLaDUxogMh9veUFBfdQ) * BorgMaster is the brain of the whole cluster, maintaining the overall cluster status and persisting data into Paxos storage; * The Scheduler is responsible for task scheduling, assigning specific tasks to specific machines based on application characteristics; * Borglet is tasked with the actual running of tasks (in containers); * borgcfg is Borg’s command-line tool for interacting with the Borg system, usually by submitting tasks through a configuration file. ## Kubernetes Architecture Bringing Borg's design philosophy into play, including concepts such as Pod, Service, Labels, and single Pod single IP, the overall architecture of Kubernetes is strikingly similar to Borg's, as can be seen in the picture below ![architecture](/files/6hCI9OxBEizdwD1l4RxN) Kubernetes mainly comprises the following core components: * etcd stores the whole cluster's state; * kube-apiserver offers a unique entry for resource operations, and supplies authentication, authorization, access control, API registration, and discovery mechanisms; * kube-controller-manager is tasked with maintaining the cluster's state, handling tasks like fault detection, automatic expansion, rolling updates, etc.; * kube-scheduler is responsible for resource scheduling, allocating Pods to respective machines following predetermined scheduling strategies; * kubelet is in charge of container lifecycle maintenance, along with Volume (CVI) and network (CNI) management; * Container runtime handles image management and the actual running of Pods and containers (CRI), with Docker being the default container runtime; * kube-proxy provides service discovery and load balancing within the cluster for Service entities; ![](/files/luGHexMEIdZbqZGDn6Hd) Other than core components, there are some recommended add-ons: * kube-dns provides the entire cluster with DNS services * Ingress Controller provides external entrances for services * Heapster offers resource monitoring * Dashboard provides GUI * Federation provides clusters that span availability zones * Fluentd-elasticsearch provides cluster log collection, storage, and query ### Layered Architecture Kubernetes' design principles and functions mirror a Linux-style layered architecture, as the diagram below depicts ![](/files/xvMK9BqudAXyHTDAOJal) * Core layer: Kubernetes offers core functions, providing an API for building higher-level applications externally and offering a plugin-style application execution environment internally. * Application layer: Includes deployment (stateless applications, stateful applications, batch processing tasks, cluster applications, etc.) and routing (service discovery, DNS resolution, etc.) * Management layer: Measures system metrics (such as infrastructure, containers, and network metrics), enhances automation (like automatic scaling, dynamic provisioning, etc.), and manages policies (like RBAC, Quota, PSP, NetworkPolicy, etc.) * Interface layer: Includes the kubectl command-line tool, client SDK, and cluster federation. * Ecosystem: Above the interface layer lies a vast ecosystem of container cluster management scheduling, divisible into two realms: * External to Kubernetes: Logging, monitoring, configuration management, CI, CD, workflow, FaaS, OTS applications, ChatOps, etc. * Internal to Kubernetes: CRI, CNI, CVI, image repositories, Cloud Provider, cluster's configuration and management, etc. ### Core Components ![](/files/PyzGIkIkVv0lBIG6Tghb) ### Core API ![](/files/a11BHJd0GIpmZxPm6wMR) ### Ecosystem ![](/files/p39MVP5RQhuGylvxJmsT) For more information on the layered architecture, be sure to look into the [Kubernetes architectural roadmap](https://github.com/kubernetes/community/tree/master/sig-architecture) that the Kubernetes community is currently promoting. ## References * [Kubernetes design and architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md) * * * * [Kubernetes Architecture SIG](https://github.com/kubernetes/community/tree/master/sig-architecture) # Design Principles ### Design Concepts and Distributed Systems Breaking down and understanding the design principles behind Kubernetes allows us to delve deeper into the Kubernetes system. It supports better usage management of cloud-native applications deployed in a distributed way. On the other hand, it can lead us to learn from their experience in the design of distributed systems. ### API Design Principles For cloud computing systems, the system API actually assumes the role of system design. For each new feature supported by the Kubernetes cluster system, or each new technology introduced, a corresponding API object will be introduced to manage the feature's operation. Understanding these APIs is like grabbing hold of the key aspects of the K8s system. The design of Kubernetes system API follows these principles: 1. **All APIs should be declarative**. Declarative operations, compared to the imperative ones, have a steady effect on repeated operations, which is essential in a distributed environment where data loss or duplication is common. Moreover, declarative operations are easier for users to use. They allow the system to hide implementation details from the user while retaining the system's potential for continuous optimization. Also, the declarative API implies that all API objects are nouns like Service, Volume. These descriptive words represent the target object that the user expects to achieve. 2. **API objects should be complementary and combinable**. This encourages API objects to adhere to object-oriented design principles, achieving "high cohesion, loose coupling", breaking down business-related concepts appropriately and increasing the reusability of the decomposed objects. 3. **High-level APIs are built on operational intentions**. Good API design parallels good application system design using object-oriented methods; high-level design should be business-driven, not technological implementation driven. Therefore, the design of Kubernetes' high-level APIs is based on the operation and management of system scheduling containers. 4. **Low-level APIs are designed according to the control needs of high-level APIs**. The aim of designing and implementing low-level APIs is their use by high-level APIs. The design of low-level APIs should be based on demand, and be able to resist the temptation of technical execution. 5. **Avoid simple encapsulation and do not rely on internal hidden mechanisms unknowable through the external API**. Simple encapsulation does not necessarily provide new functionality but increases the dependability of the encapsulated API. Internal hidden mechanisms are detrimental to system maintenance. For instance, StatefulSet and ReplicaSet, which are two sets of Pods, are defined by Kubernetes using different API objects, not by distinguishing different types of ReplicaSet with a special internal algorithm. 6. **API operation complexity should be proportional to the number of objects**. This point is mainly considered from the system performance - to ensure that as the system size expands, the performance doesn't slow down rapidly. A minimum constraint is that an API's operational complexity should not exceed O(N), where N is the number of objects; otherwise, the system will not be scalable. 7. **The status of an API object should not depend on the network connection status**. It is well known that in a distributed environment, network disconnections are frequently occurring. Therefore, to ensure that the API object's state can accommodate network instability, the object's state must not rely on network connection status. # Components ![components](/files/4ws2uapDhFbPQNxIkQU4) Kubernetes is mainly composed of the following core components: * Etcd that preserves the state of the whole cluster; * API Server offers a unique entrance for resource operations, providing mechanisms such as authentication, authorization, access control, API registration and discovery; * Controller Manager that maintains the state of the cluster, such as fault detection, automatic expansion, rolling updates, etc.; * Scheduler that manages the allocation of resources, scheduling Podes to the corresponding machines based on the predetermined scheduling strategies; * Kubelet that maintains the lifecycle of containers, as well as managing Volumes (CVI) and networks (CNI); * Container Runtime, being responsible for image management and the actual operation of Pods and containers (CRI); * Kube-proxy, providing service discovery and load balancing within the cluster for the service. ## Component Communication The principle of communication between multiple components in Kubernetes is described as follows: * API Server manages all operations with the etcd storage, while it’s the only one that operates the etcd cluster directly. * API Server provides a unified REST API for the inside (other components in the cluster) and the outside (users), and all other components communicate with each other through the API Server. * Controller Manager, Scheduler, Kube-proxy, Kubelet, etc., all watch the changes in the resources through the watch API of the API Server, and manipulate the resources accordingly. * All operations that require updating the status of resources happen through the REST API of the API Server. * API Server also directly calls Kubelet API (such as logs, exec, attach, etc.), does not verify the Kubelet certificate by default, but can be turned on with `--kubelet-certificate-authority` (while GKE protects their communication through SSH tunnel). The typical flow of creating a Pod goes as follows: ![](/files/oUiVbV3rFIpcK8qnYqdA) 1. A user creates a Pod via REST API 2. API Server writes into etcd 3. Scheduler checks unbound node Pod, begins scheduling and updates the node binding of the Pod 4. Kubelet detects a new scheduled Pod and runs it via Container Runtime 5. Kubelet gets the status of Pod via Container Runtime and updates it to API Server ## Port Numbers ![ports](/files/lUxaVx8382bbJK5l3pCq) ### Master node(s) | Protocol | Direction | Port Range | Purpose | | -------- | --------- | ---------- | -------------------------------- | | TCP | Inbound | 6443\* | Kubernetes API server | | TCP | Inbound | 8080 | Kubernetes API insecure server | | TCP | Inbound | 2379-2380 | etcd server client API | | TCP | Inbound | 10250 | Kubelet API | | TCP | Inbound | 10251 | kube-scheduler healthz | | TCP | Inbound | 10252 | kube-controller-manager healthz | | TCP | Inbound | 10253 | cloud-controller-manager healthz | | TCP | Inbound | 10255 | Read-only Kubelet API | | TCP | Inbound | 10256 | kube-proxy healthz | ### Worker node(s) | Protocol | Direction | Port Range | Purpose | | -------- | --------- | ----------- | --------------------- | | TCP | Inbound | 4194 | Kubelet cAdvisor | | TCP | Inbound | 10248 | Kubelet healthz | | TCP | Inbound | 10249 | kube-proxy metrics | | TCP | Inbound | 10250 | Kubelet API | | TCP | Inbound | 10255 | Read-only Kubelet API | | TCP | Inbound | 10256 | kube-proxy healthz | | TCP | Inbound | 30000-32767 | NodePort Services\*\* | ## Version Support Strategy ## Active Versions The Kubernetes community currently maintains the latest three minor versions (such as 1.21.x, 1.20.x, 1.19.x), and each minor version has a one-year patch support cycle (9 months before 1.18). Patches for active versions are released approximately once a month, and the detailed release schedule can be found [here](https://github.com/kubernetes/website/blob/main/content/en/releases/patch-releases.md#upcoming-monthly-releases). ## Version Compatibility * In a HA cluster, all kube-apiserver instances can only have a minor version difference at most (e.g., some are 1.21, some are 1.20) * Kubelet can have up to two minor versions difference with kube-apiserver (like when kube-apiserver is 1.21, kubelet can be 1.21, 1.20, 1.19) * Kube-controller-manager, kube-scheduler, and cloud-controller-manager can only have a minor version difference with kube-apiserver (like when kube-apiserver is 1.21, kube-controller-manager is 1.20) ## Upgrade Order When upgrading the Kubernetes cluster (e.g., from 1.20.1 to 1.21.1), the following upgrade order and dependencies should be ensured: * Before upgrading, make sure that the ValidatingWebhookConfiguration and MutatingWebhookConfiguration are upgraded to the latest API version (compatible with the new and old versions of kube-apiserver) * All instances of kube-apiserver actually need to be upgraded before other components (like kube-controller-manager) * Kube-controller-manager, kube-scheduler, and cloud-controller-manager can only be upgraded after kube-apiserver is upgraded * Kubelet can only be upgraded after kube-apiserver is upgraded, and before upgrading, you need to `kubectl drain ` (i.e., kubelet does not support minor version upgrades locally) * Kube-proxy needs to ensure that it is the same version as the kubelet on the same node. ## Reference Documents * [Master-Node communication](https://kubernetes.io/docs/concepts/architecture/master-node-communication/) * [Core Kubernetes: Jazz Improv over Orchestration](https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca) * [Installing kubeadm](https://kubernetes.io/docs/setup/independent/install-kubeadm/#check-required-ports) * [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/#supported-component-upgrade-order) # etcd Developed by CoreOS, etcd is an open-source distributed key-value store that serves as a backbone of distributed systems. It is based on the Raft consensus algorithm and shines in areas such as service discovery, sharing configuration information, and ensuring consistency (such as database master selection, distributed locks, etc.). ## What makes etcd so special? * Basic key-value storage * Listening mechanism * Expiry and renewal mechanisms for keys, used for monitoring and service discovery * Atomic Compare-and-Swap and Compare-and-Delete operations, used for distributed locks and leader elections ## How does etcd achieve consistency? Etcd achieves consistency in its operations, thanks to the RAFT protocol. The election process is as follows: 1. Initially, all nodes are in a 'follower' state and are assigned an election timeout. If they don't receive a heartbeat from the leader within this timeout, they switch to a 'candidate' state, ask other nodes in the cluster to vote for them, and initiate an election. 2. When a candidate receives votes from over half of the nodes in the cluster, it becomes the leader, starts receiving and saving client data, and syncs its logs with other follower nodes. If consensus isn't reached, candidates wait for a random period (between 150ms - 300ms), before initiating another vote. 3. A leader node maintains its position by sending heartbeats to follower nodes at regular intervals. 4. If, at any point, a follower node does not receive a heartbeat from the leader within the election timeout, it will switch to a 'candidate' state and initiate a new election. Each time this happens, the term of the newly elected leader is incremented by 1. Replication of logs occurs in the following manner: When the leader node receives a log (transaction request) from a client, it first appends this log to its own Log, then syncs this entry with other followers via a heartbeat. Followers, on receiving the log, record it and send an acknowledgment (ACK) back to the leader. Once the leader receives ACKs from the majority (n/2+1) of followers, it sets the log as committed, appends it to the local disk, notifies the client, and in the next heartbeat, instructs all followers to store the log in their local storage. Etcd also has safety measures that make sure every node executes the same sequence of instructions. For example, if a follower becomes unavailable when the current leader commits a Log, that follower might later be elected as the Leader and may overwrite the already committed Log with a new one. This could lead to nodes executing different sequences, which is where Safety steps in. Safety ensures that any elected Leader must contain the previously committed Log. Some key safety measures include: 1. Election Safety: Only one Leader can be elected in a term. 2. Leader Completeness: The Leader's log must be complete. If a Log is committed in term1, then the Leaders in any future terms (term2, term3...) must contain that Log. This is verified by the Term during the election phase. Etcd also has protocols for handling faults: 1. When a Leader fails: Other nodes, which have not received a heartbeat, initiate a new election. When the original Leader recovers, due to lower stepping numbers, it automatically becomes a follower, and its logs are overwritten by the new Leader's logs. 2. When a follower node becomes unavailable: This is relatively easy to handle. The cluster's log content is always synced from the Leader. As soon as the unavailable node re-joins the cluster, it replicates the log from the Leader. 3. When multiple candidates exist: In the event of a conflict, each candidate randomly selects a wait interval (between 150ms - 300ms) before initiating a new vote. The candidate that receives the majority (over half) of the votes in the cluster will become the Leader. ### etcd’s wal logs While implementing Raft, etcd makes full use of the Concurrent Sequential Processes (CSP) concurrency model and channel magic in Go language. They have a wal log for finer details, which you can explore in the source code. The wal logs are binary. When parsed, the resultant data structure is LogEntry. It consists of four main fields: The first one is type, which can either be 0 for Normal or 1 for ConfChange (which represents configuration alterations within etcd itself, like the addition of new nodes). The second one term represents the tenure of the leader, which changes every time there's a change in leadership. The third field, index, is a sequentially growing number denoting the change order. The last field, data, is a binary representation of the Raft request objects' Protocol Buffers (pb) structure, which is wholly saved. etcd has a tool called etcd-dump-logs which can transform wal logs into text for viewing, aiding in the analysis of the Raft protocol. Though the Raft protocol does not concern itself with application data (the data part of it), it ensures consistency through syncing the wal logs, with each node applying the data received from the leader to its local storage. Raft is only concerned with the log's syncing status. If there's a bug in the local storage, such as one that fails to apply the data to the local, it could potentially cause a data discrepancy. ## What's the difference between etcd v2 and v3? In essence, etcd v2 and v3 are two separate applications sharing the same Raft protocol code. Their APIs are not alike, their storage methods differ, and their data is mutually exclusive. Which is to say, if you upgraded from etcd v2 to etcd v3, you can only access v2 data via v2 interface, and data created using the v3 interface can only be accessed using the v3 interface. When using etcd in Kubernetes clusters, etcd v3 is recommended as the v2 version has been deprecated as of Kubernetes v1.11. ## How does etcd v2 handle storage, watch operations, and expiration? Etcd v2 is primarily an in-memory implementation. It does not write data to disk in real-time; instead, it serializes the entire store into a JSON format and writes it to a file. The data in memory is organised in a simple tree structure. For example, the following data is stored in etcd as shown in the figure. ``` /nodes/1/name node1 /nodes/1/ip 192.168.1.1 ``` The store has a global currentIndex which increments by 1 with every change; each event is then linked to this currentIndex. When a client invokes the watch interface (and includes the 'wait' parameter), if the request parameters contain a waitIndex that is lower than the currentIndex, it fetches those events from the EventHistory table which have an index greater than or equal to the waitIndex and are associated with the watch key. If there is data, it is returned immediately. If the history table does not contain any data, or if the request does not contain a waitIndex, then the request is placed into the WatchHub. Each key has an associated list of watchers. Any changes generate an event that is placed in the EventHistory table and notifies the relevant watcher associated with the key. Similarly, the etcd v2 expiration mechanism sets the expiration time only on individual keys, making it difficult to ensure consistent lifecycles for multiple keys. Watchers can only watch a specific key and its sub-nodes (through the recursive parameter) and cannot conduct multiple watches. It is also challenging to implement complete data synchronization through the watch mechanism due to the risk of missing changes. Therefore, most current uses involve watching for changes, then retrieving data through a get request, rather than relying on the change events from the watch operation. ## How does etcd v3 handle storage, watch operations, and expiration? Etcd v3 separates the watch and store operations and improves on the expiration mechanisms. You can set the lease on expiration time, and then link the key with the lease. This lets you link multiple keys to the same lease ID, making it easier to set a unified expiration time and implement batch renewal. This feature means that, compared to etcd v2, it is easier to use, more capable and provides a more efficient and reliable watch mechanism. With etcd v3, you can pretty much implement total data synchronization through the watch mechanism. You can also set up an etcd cluster to automatically compact data when starting up or do it manually using a command. If the change frequency is high, this is recommended, or else it could result in excessive resource usage and even errors. In etcd v3, the default backend quota is 2GB, and if you do not compact your data and the boltdb file size exceeds this limit, you will see an 'Error: etcd # kube-apiserver Kube-apiserver might seem like a tongue-twisting piece of jargon. But it is actually central to the operation of Kubernetes, a popular open-source platform used to automate the deployment, scaling, and management of applications. Here's a deep-dive into knowing what it does. Kube-apiserver plays two key roles. First, it provides the REST API interface for cluster management tasks - including authentication, authorization, data validation, and cluster state changes. Second, it acts as a hub for data exchange and communication between other Kubernetes modules. These modules can use APIs to query or modify data, with only the API Server having direct access to the etcd, the distributed database storing all Kubernetes configuration data. ## The Two Roads to the API Kube-apiserver offers both https and non-secure http API access. The former, https, is, by default, linked to port number 6443. The http API is generally accessed via '127.0.0.1' at port 8080. It's crucial to note here that the http API isn't recommended for use in production environments as it lacks any authentication protocols. A user can access these interfaces and their identical REST API formats by referring to the [Kubernetes API Reference](https://kubernetes.io/docs/reference/kubernetes-api/). Usage commonly occurs through the [kubectl](https://kubernetes.io/docs/user-guide/kubectl-overview/) command-line tool or clients developed in various programming languages available for Kubernetes. Helpful inscriptions, such as the format of each API call, become visible when activating debug log during kubectl usage, like so: ```bash $ kubectl --v=8 get pods ``` One can use `kubectl api-versions` and `kubectl api-resources` to find out about the API versions and resource objects that Kubernetes API supports, as demonstrated below: ```bash $ kubectl api-versions admissionregistration.k8s.io/v1beta1 ... $ kubectl api-resources --api-group=storage.k8s.io NAME SHORTNAMES APIGROUP NAMESPACED KIND storageclasses sc storage.k8s.io false StorageClass ... ``` ## Integration with OpenAPI and Swagger OpenAPI and Swagger API can be viewed at `/swaggerapi` and `/openapi/v2`, respectively. Once the `--enable-swagger-ui=true` command activates the Swagger UI, it becomes accessible via `/swagger-ui`. Fun fact - OpenAPI actually allows for the development of clients in various languages. For instance, the following command generates one for the Go language: ```bash git clone https://github.com/kubernetes-client/gen /tmp/gen cat >go.settings <) | | `nodefs.available` | `nodefs.available` := `node.stats.fs.available`. This includes Kubelet Volumes, logs, etc. | | `nodefs.inodesFree` | `nodefs.inodesFree` := `node.stats.fs.inodesFree`. | | `imagefs.available` | `imagefs.available` := `node.stats.runtime.imagefs.available`. This includes images and writable layers of containers. | | `imagefs.inodesFree` | `imagefs.inodesFree` := `node.stats.runtime.imagefs.inodesFree`. | These eviction thresholds can be set as a percentage or an absolute value, e.g., ```bash --eviction-hard=memory.available<500Mi,nodefs.available<1Gi,imagefs.available<100Gi --eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi" --system-reserved=memory=1.5Gi ``` These eviction signals can be divided into soft and hard eviction: * Soft Eviction: Used in conjunction with eviction grace periods (eviction-soft-grace-period and eviction-max-pod-grace-period). An eviction only takes place if the system resource reaches the soft eviction threshold and the grace period has elapsed. * Hard Eviction: An eviction is immediately executed once the system resource crosses the hard eviction threshold. Eviction actions involve reclaiming node resources and evicting user Pods: * Reclaiming Node Resources * If the imagefs threshold is configured: * If the nodefs threshold is exceeded: Delete stopped Pods * If the imagefs threshold is exceeded: Delete unused images * If the imagefs threshold is not configured: * If the nodefs threshold is exceeded: Cleanse resources in the order of deleting stopped Pods and unused images * Evicting User Pods * Eviction order: BestEffort, Burstable, Guaranteed * If the imagefs threshold is configured: * If the nodefs threshold is exceeded: Eviction based on nodefs usage (local volume + logs) * If the imagefs threshold is exceeded: Eviction based on imagefs usage (writable container layers) * If the imagefs threshold is not configured: * If the nodefs threshold is exceeded: Evicts based on total disk usage (local volume + logs + writable container layers) In addition to eviction, Kubelet supports an array of container and image garbage collection options that will eventually get replaced by eviction: | Garbage Collection Parameter | Eviction Parameter | Explanation | | ---------------------------- | -------------------------------------- | ------------------------------------------------------ | | `--image-gc-high-threshold` | `--eviction-hard` or `--eviction-soft` | Existing eviction thresholds can trigger image garbage | ## Kubelet API The [Kubelet API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go#L93-L102) can be accessed through its exposed port or `kubectl get --raw /api/v1/nodes/$NODE/proxy/`. Some commonly used APIs include: * `/metrics` - Query Kubelet metrics * `/metrics/cadvisor` - Query Cadvisor metrics (including node and container metrics) * `/metrics/resource` - Query resource usage metrics (including CPU and memory) * `/metrics/probes` - Query probe metrics * `/stats/summary` - Query node and Pod summary indicators * `/pods` - Query the list of Pods * `/logs` - Query node or container logs * `/containerLogs/{podNamespace}/{podID}/{containerName}` - Query container logs * `configz` - Query Kubelet configuration * `/run`, `/exec`, `/attach`, and # kube-proxy A service known as kube-proxy operates on each machine, attentively observing the variations in the service and endpoint within the API server. Kube-proxy configures load balancing for services via mechanisms like iptables, keeping in mind that it only supports TCP and UDP. Kube-proxy can function directly on a physical machine, or operate in the style of a static pod or daemonset. There are a few modes of operation available for kube-proxy: * userspace: This is the earliest load balancing scheme. It operates on a port in user space, forwarding all services via iptables to this port. Inside, it handles load balancing to the actual Pod. Its most significant drawback is its inefficiency, creating an evident performance bottleneck. * iptables: This is the currently recommended scheme. It brings service load balancing into existence using iptables rules alone. The trouble with this approach is that an exorbitant number of iptables rules may result in service overload, and non-incremental updates can introduce certain delays. In extensive situations, performance degradation is quite significant. * ipvs: In order to counteract the performance problems of the iptables mode, ipvs mode was introduced in version v1.11 (support for test version began in v1.8, and it was introduced to the GA in v1.11). It updates incrementally and maintains connection consistency during service updates. * winuserspace: Operating just like userspace, it only works on windows nodes. When using ipvs mode, you will need to preload the kernel modules `nf_conntrack_ipv4`, `ip_vs`, `ip_vs_rr`, `ip_vs_wrr`, `ip_vs_sh`, etc., on each Node. Check the diagram below for an illustration of Kube-proxy iptables ![](/files/lzzZdBt8YggvxuyDLPW6) The image is from [cilium/k8s-iptables-diagram](https://github.com/cilium/k8s-iptables-diagram) Now let's check out the Kube-proxy NAT Diagram ![](/files/ogN3akPwo5AE280DfMZM) The image is from [kube-proxy iptables "nat" control flow](https://docs.google.com/drawings/d/1MtWL8qRTs6PlnJrW4dh8135_S9e2SaawT410bJuoBPk/edit) In [Kube-proxy IPVS mode](https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md), you can find a detailed explanation of how different services work under IPVS mode. ![](/files/DUOWHAjkfHwUBg6Cuj3h) Note that IPVS mode also uses iptables for tasks like SNAT and IP Masquerading (MASQUERADE), and uses ipset to simplify the management of iptables rules. Let's look at how to start kube-proxy with this example: ```bash kube-proxy --kubeconfig=/var/lib/kubelet/kubeconfig --cluster-cidr=10.240.0.0/12 --feature-gates=ExperimentalCriticalPodAnnotation=true --proxy-mode=iptables ``` To better understand how kube-proxy works: kube-proxy monitors the changes in the service and endpoint within the API server. It configures load balancing (only supporting TCP and UDP) for services using proxiers such as userspace, iptables, ipvs, or winuserspace. ![](/files/gTfzE4SYfR4abOhptt7n) There are, however, some shortcomings of kube-proxy. It currently only supports TCP and UDP, and does not support HTTP routing or a health check mechanism. But these gaps can be closed by customizing an [Ingress Controller](/en/extension/ingress). # kube-dns The DNS service is one of the essentials in the Kubernetes world, and it's facilitated through kube-dns or CoreDNS as crucial extensions of the Kubernetes cluster. ## CoreDNS: Efficiency at its best Starting from version v1.11, [CoreDNS](https://coredns.io/) has been available to furnish the vital DNS services, and it took the mantle as the default DNS service from v1.13. CoreDNS checks off all the boxes when it comes to efficiency and less resource usage. Thus, the shift from using kube-dns to CoreDNS in delivering DNS services to the cluster is highly recommended. Upgrading from kube-dns to CoreDNS: Here's how you can do it: ```bash $ git clone https://github.com/coredns/deployment $ cd deployment/kubernetes $ ./deploy.sh | kubectl apply -f - $ kubectl delete --namespace=kube-system deployment kube-dns ``` For a fresh deployment, you can follow the CoreDNS extension configuration method [right here](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns). ## DNS formats supported * Service * A record: Generates `my-svc.my-namespace.svc.cluster.local`. IP resolving takes two forms * For a standard service, it resolves to Cluster IP * For a headless service, it resolves to a list of specified Pod IPs * SRV record: Generates `_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local` * Pod * A record: `pod-ip-address.my-namespace.pod.cluster.local` * Specified hostname and subdomain: `hostname.custom-subdomain.default.svc.cluster.local`. Check out an example shown below: ```yaml apiVersion: v1 kind: Pod metadata: name: busybox2 labels: name: busybox spec: hostname: busybox-2 subdomain: default-subdomain containers: - image: busybox command: - sleep - "3600" name: busybox ``` ![](/files/kWhf6FshCIgSEozQlTvi) ## Configuring Private DNS Servers and Upstream DNS Servers Beginning with Kubernetes 1.6, customization of stub domains and upstream name servers got easier by providing a ConfigMap for kube-dns. The configuration below introduces a standalone private root DNS server and two upstream DNS servers. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kube-dns namespace: kube-system data: stubDomains: | {“acme.local”: [“1.2.3.4”]} upstreamNameservers: | [“8.8.8.8”, “8.8.4.4”] ``` Upon using the above configuration, query requests will first be sent to the DNS cache layer of kube-dns (Dnsmasq server). The Dnsmasq server checks the suffix of the request first. Requests with a cluster suffix (such as: ”.cluster.local”) will be sent to kube-dns, names with a stub domain suffix (like: ”.acme.local”) will be dispatched to the configured private DNS server \[“1.2.3.4”]. Finally, requests that do not satisfy any of these suffixes will be sent to the upstream DNS \[“8.8.8.8”, “8.8.4.4”]. ![](/files/RmiHdk9Eg9QRcwd3cFxa) ## kube-dns: At the heart of Kubernetes ### Starting a kube-dns example Generally, the DNS service is deployed as an expansion. This can be done by adding the [kube-dns.yaml](https://github.com/feiskyer/kubernetes-handbook/raw/master/manifests/kubedns/kube-dns.yaml) to the `/etc/kubernetes/addons` directory of the Master node. Of course, manual deployment is also an option: ```bash kubectl apply -f https://github.com/feiskyer/kubernetes-handbook/raw/master/manifests/kubedns/kube-dns.yaml ``` This will initiate a Pod containing three containers in Kubernetes, running three DNS-related services: ```bash # kube-dns container kube-dns --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2 # dnsmasq container dnsmasq-nanny -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --log-facility=- --server=127.0.0.1#10053 # sidecar container sidecar --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local.,5,A --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local.,5,A ``` Kubernetes v1.10 also supports the Beta version of CoreDNS, which outperforms kube-dns. Deployment can be done via extension by adding [coredns.yaml](https://github.com/feiskyer/kubernetes-handbook/blob/master/manifests/kubedns/coredns.yaml) to the `/etc/kubernetes/addons` directory on the Master node. Of course, manual deployment is another option: ```bash kubectl apply -f https://github.com/feiskyer/kubernetes-handbook/raw/master/manifests/kubedns/coredns.yaml ``` ### kube-dns: Behind the scenes As shown below, kube-dns consists of three main components: * kube-dns: The heart of the DNS service, mainly composed of KubeDNS and SkyDNS * KubeDNS listens to the changes in Service and Endpoint and updates related information in SkyDNS * SkyDNS is responsible for DNS resolution, listening on ports 10053 (tcp/udp) and 10055 for metrics * kube-dns also listens on port 8081 for health checks * dnsmasq-nanny: Manages dnsmasq and restarts it when the configuration changes * The upstream of dnsmasq is SkyDNS, meaning the internal DNS resolution of the cluster is handled by SkyDNS * sidecar: Looks after health checks and provides DNS metrics (listening on port 10054) ![](/files/uU82vpETE1MCFXXxe3hv) ### An introduction to the source code The kube-dns code has been separated from Kubernetes and can now be found at . The code for kube-dns, dnsmasq-nanny, and sidecar starts from `cmd//main.go` respectively and calls `pkg/dns`, `pkg/dnsmasq`, and `pkg/sidecar` to perform respective functions. The core DNS resolution directly refers to the code in `github.com/skynetservices/skydns/server`, the specific implementation can be seen at [skynetservices/skydns](https://github.com/skynetservices/skydns/tree/master/server). ## Frequently Asked Questions **Issues with DNS Resolution in Ubuntu 18.04** Ubuntu 18.04 has been configured to activate systemd-resolved by default. This writes `nameserver 127.0.0.53` into the system's /etc/resolv.conf. As this is a local address, it can cause CoreDNS or kube-dns to fail when resolving external addresses. To fix this issue, replace the resolv.conf file generated by systemd-resolved: ```bash sudo rm /etc/resolv.conf sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf ``` Or, manually specify the path to the resolv.conf for the DNS service: ```bash --resolv-conf=/run/systemd/resolve/resolv.conf ``` ## References * [Introduction to dns-pod-service](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) * [coredns/coredns](https://github.com/coredns/coredns) # Federation In a Cloud computing environment, services can operate at different range levels: within the same host (Host, Node), spanning hosts within the same available zone (Available Zone), crossing zones within the same region (Region), served from the same Cloud service providers, or even across different cloud platforms. The design of Kubernetes (often referred to as K8s) is aimed at handling a single cluster in the same region because performance within a region can meet K8s's threshold for scheduling and connections between computation storage. However, the concept of clustering Federation is designed to offer services from K8s clusters that cross regions and service providers. Each Federation comes equipped with its distributed storage, API Server, and Controller Manager. Users are able to register K8s Clusters to a Federation's API Server. When users create or modify API objects through the Federation's API Server, the Federation API Server will create a replica of the same API object in all its registered sub K8s Clusters. When serving business requests, K8s Federation first balances the load among its own sub Clusters. For requests directed to a specific K8s Cluster, it follows the same scheduling pattern as when the K8s Cluster operates independently, providing internal load balancing within that K8s Cluster. Load balancing between Clusters is realized through domain service load balancing. ![](/files/DNOmTXjsErHP91WmT7Zz) All designs aim to reduce impact on existing K8s Cluster mechanisms. Thus, each individual K8s Cluster does not need an additional outer layer of K8s Federation, which implies that existing K8s code and mechanisms do not need to be altered due to Federation functionality. ![](/files/TzWuIcd9CPv74tt7hmPC) Federation mainly consists of three components: * federation-apiserver: similar to kube-apiserver but provides REST API across clusters * federation-controller-manager: like the kube-controller-manager, but ensures synchronization across multiple cluster states * kubefed: a command line tool for managing Federation The code for Federation is maintained at . The next section delves into the steps of deploying Federation, which includes downloading kubefed and kubectl, initializing a main cluster, customizing DNS, deploying on physical machinery and, customizing etcd storage. Once the Federation is deployed, you can use it by registering clusters other than the main cluster using the `kubefed join` command, query registered kubernetes clusters list, use annotation `federation.alpha.kubernetes.io/cluster-selector` for new object selection as of version 1.7+, and implement policy-based scheduling. Try utilizing the Federation by deploying resources such as Federated ConfigMap, Federated Service, Federated DaemonSet, and more! Ending the Federation operation is also possible by removing the cluster or the Federation. With technological advancement making cross-region, cross-service operations a reality, Federation is increasingly becoming an intrinsic part of K8s operations. It's time to embrace the future of digital transformation with Federation - at your fingertips! For further information, check out [Kubernetes federation](https://kubernetes.io/blog/2018/12/12/kubernetes-federation-evolution/) and [kubefed](https://github.com/kubernetes-sigs/kubefed). # kubeadm Kubeadm is among the tools that Kubernetes proudly recommends, and it's currently undergoing rapid iteration and development. ## System Initialization All machines need to initialize their container execution engine (like Docker or Frakti) and also kubelet. These initializations are essential since kubeadm relies on kubelet to start up the Master components such as kube-apiserver, kube-manager-controller, kube-scheduler, and kube-proxy, among others. ## Connecting with Master To initialize the master, all you have to do is run the command `kubeadm init`, like so: ```bash kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stable ``` Executing this command will autonomously: * Run a systematic status check, * Generate a token, * Launch a self-signed CA and client-side certificates, * Create a kubeconfig for kubelet to connect to the API server, * Produce Static Pod manifests for Master components and place them in the `/etc/kubernetes/manifests` directory, * Configure RBAC and set the Master node to only run the control plane components, * Establish additional services, like kube-proxy and kube-dns. ## Adjusting the Network Plugin During initialization, kubeadm remains indifferent to the network plugin. On default, kubelet is configured to use CNI plugins, requiring users to initialize the network plugin separately. ### CNI Bridge ```bash mkdir -p /etc/cni/net.d cat >/etc/cni/net.d/10-mynet.conf <<-EOF { "cniVersion": "0.3.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.244.1.0/24", "routes": [ {"dst": "0.0.0.0/0"} ] } } EOF cat >/etc/cni/net.d/99-loopback.conf <<-EOF { "cniVersion": "0.3.0", "type": "loopback" } EOF ``` ### Flannel ```bash kubectl create -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel-rbac.yml kubectl create -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml ``` ### Weave ```bash kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d'\n')" ``` ### Calico ```bash kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml ``` ## Node Addition ```bash token=$(kubeadm token list | grep authentication,signing | awk '{print $1}') kubeadm join --token $token ${master_ip} ``` This step includes the following processes: * Downloading the CA from the API server, * Generating local certificates and requesting the API Server's signature, * Finally, configuring kubelet to connect to the API Server. ## Installation Removal ```bash kubeadm reset ``` ## Helpful References * [kubeadm Setup Tool](https://kubernetes.io/docs/admin/kubeadm/) # hyperkube Hyperkube is an all-in-one binary package of Kubernetes, having the capability to power many Kubernetes services, and is often found in Docker images. With every release of Kubernetes, a Docker image incorporating Hyperkube is published simultaneously, such as `gcr.io/google_containers/hyperkube:v1.6.4`. The commands that Hyperkube supports include: * kubelet * apiserver * controller-manager * federation-apiserver * federation-controller-manager * kubectl * proxy * scheduler # kubectl kubectl is the command-line interface (CLI) of Kubernetes, being the essential management tool for Kubernetes users and administrators. Instead of listing all of its subcommands, this article will show you how to use it effectively, navigate your way around and look up any assistance you might need. * `kubectl -h` for listing subcommands * `kubectl options` for global options * `kubectl --help` for assistance with subcommands * `kubectl [command][PARAMS] -o=` to set your output format, for example json, yaml, jsonpath etc. * `kubectl explain[RESOURCE]` to display a resource’s definition ## Your first step The first step in using kubectl is to set up your Kubernetes cluster and its authentication methods, this includes: * Information about the cluster: the Kubernetes server’s address * User information: user name, password or key * Context: a combination of cluster information, user information and namespace Here’s an example: ```bash kubectl config set-credentials myself --username=admin --password=secret kubectl config set-cluster local-server --server=http://localhost:8080 kubectl config set-context default-context --cluster=local-server --user=myself --namespace=default kubectl config use-context default-context kubectl config view ``` ## Some common command patterns * Create: `kubectl run --image=` or `kubectl create -f manifest.yaml` * Check: `kubectl get ` * Update: `kubectl set` or `kubectl patch` * Delete: `kubectl delete ` or `kubectl delete -f manifest.yaml` * Check a Pod IP: `kubectl get pod -o jsonpath='{.status.podIP}'` * Execute commands inside a container: `kubectl exec -ti sh` * Check for a container's logs: `kubectl logs [-f] ` * Share a service: `kubectl expose deploy --port=80` * Decode from Base64: ```bash kubectl get secret SECRET -o go-template='{{ .data.KEY | base64decode }}' ``` Take note that `kubectl run` only supports creating resources like Pod, Replication Controller, Deployment, Job and CronJob. Specifying which resources are created depends on which parameters you pass, by default, it's a Deployment: | Resource type | Parameter | | ---------------------- | --------------------- | | Pod | `--restart=Never` | | Replication Controller | `--generator=run/v1` | | Deployment | `--restart=Always` | | Job | `--restart=OnFailure` | | CronJob | `--schedule=` | ## Command-line auto-completion For Linux systems: ```bash source /usr/share/bash-completion/bash_completion source <(kubectl completion bash) ``` For MacOS: ```bash source <(kubectl completion zsh) ``` ## Customized output columns Say, you want to check requests or limits for resources for all Pods: ```bash kubectl get pods --all-namespaces -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,"CPU(requests)":.spec.containers[*].resources.requests.cpu,"CPU(limits)":.spec.containers[*].resources.limits.cpu,"MEMORY(requests)":.spec.containers[*].resources.requests.memory,"MEMORY(limits)":.spec.containers[*].resources.limits.memory ``` ## Checking Logs `kubectl logs` are for displaying content output from programs running inside a container. It’s similar to Docker's logs command. ```bash # Return snapshot logs from pod nginx with only one container kubectl logs nginx # Return snapshot of previous terminated ruby container logs from pod web-1 kubectl logs -p -c ruby web-1 # Begin streaming the logs of the ruby container in pod web-1 kubectl logs -f -c ruby web-1 ``` > Note: kubectl can only check logs for individual containers. If you want to check logs for multiple pods simultaneously, you can use [stern](https://github.com/wercker/stern). For example: `stern --all-namespaces -l run=nginx`. ## Connect to a Running Container `kubectl attach` is used to connect to a running container. It's similar to Docker's attach command. ```bash # Get output from running pod 123456-7890, using the first container by default kubectl attach 123456-7890 # Get output from ruby-container from pod 123456-7890 kubectl attach 123456-7890 -c ruby-container # Switch to raw terminal mode, sends stdin to 'bash' in ruby-container from pod 123456-7890 # and sends stdout/stderr from 'bash' back to the client kubectl attach 123456-7890 -c ruby-container -i -t Options: -c, --container='': Container name. If omitted, the first container in the pod will be chosen -i, --stdin=false: Pass stdin to the container -t, --tty=false: Stdin is a TTY ``` ## Execute Commands Inside a Container `kubectl exec` is used to execute commands inside a running container. It's similar to Docker's exec command. > Note: For multiple-container Pods, the default container for kubectl commands can be set by kubectl.kubernetes.io/default-container annotation ```bash # Get output from running 'date' from pod 123456-7890, using the first container by default kubectl exec 123456-7890 date # Get output from running 'date' in ruby-container from pod 123456-7890 kubectl exec 123456-7890 -c ruby-container date # Switch to raw terminal mode, sends stdin to 'bash' in ruby-container from pod 123456-7890 # and sends stdout/stderr from 'bash' back to the client kubectl exec 123456-7890 -c ruby-container -i -t -- bash -il Options: -c, --container='': Container name. If omitted, the first container in the pod will be chosen -p, --pod='': Pod name -i, --stdin=false: Pass stdin to the container -t, --tty=false: Stdin is a TTY ``` ## Port Forwarding `kubectl port-forward` is used to forward a local port to a specified Pod. ```bash # Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod kubectl port-forward mypod 5000 6000 # Listen on port 8888 locally, forwarding to 5000 in the pod kubectl port-forward mypod 8888:5000 # Listen on a random port locally, forwarding to 5000 in the pod kubectl port-forward mypod :5000 # Listen on a random port locally, forwarding to 5000 in the pod kubectl port-forward mypod 0:5000 ``` Also, local ports can be forwarded to services, replica sets or deployments. ```bash # Forward to deployment kubectl port-forward deployment/redis-master 6379:6379 # Forward to replicaSet kubectl port-forward rs/redis-master 6379:6379 # Forward to service kubectl port-forward svc/redis-master 6379:6379 ``` ## API Server Proxy The `kubectl proxy` command creates an HTTP proxy to service Kubernetes APIs. ```bash $ kubectl proxy --port=8080 Starting to serve on 127.0.0.1:8080 ``` Direct access to the Kubernetes API through the proxy address `http://localhost:8080/api/` can be achieved. A list of pods can be retrieved, for example: ```bash curl http://localhost:8080/api/v1/namespaces/default/pods ``` If accessing port 8080 from a non-localhost address specified by `--address`, an unauthorized error will be received. To rectify this (recommended for non-production environments) the setting `--accept-hosts` can be adjusted: ```bash kubectl proxy --address='0.0.0.0' --port=8080 --accept-hosts='^*$' ``` ## Copying Files `kubectl cp` enables you to copy from a container or to copy files into a container. ```bash # Copy a local directory /tmp/foo_dir to a /tmp/bar_dir in a remote pod kubectl cp /tmp/foo_dir :/tmp/bar_dir # Copy a local file /tmp/foo to /tmp/bar in a remote pod in a specific container kubectl cp /tmp/foo :/tmp/bar -c # Copy local file /tmp/foo to a remote pod /tmp/bar in namespace kubectl cp /tmp/foo /:/tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally kubectl cp /:/tmp/foo /tmp/bar Options: -c, --container='': Container name. If omitted, the first container in the pod will be chosen ``` Note that file copying depends on the tar command, so the tar command must be executable within the container. ## Node Draining with kubectl Drain ```bash kubectl drain NODE [Options] ``` * Deletes pods on that NODE created by ReplicationController, ReplicaSet, DaemonSet, StatefulSet or Job * Doesn't delete mirror pods (since they can't be deleted through the API) * If there are other types of Pods (for e.g., directly created by kubectl create), if --force option isn't present, the command fails * If --force option is included in the command, it will delete Pods that were not created by ReplicationController, Job or DaemonSet Sometimes radical solutions like evicting pods is unnecessary. If you just need to make the Node not callable, you can use the `kubectl cordon` command. To reset, just type `kubectl uncordon NODE` to make the NODE schedulable again. ## Permissions Check The `kubectl auth` provides two subcommands for checking a user's authorization status: * `kubectl auth can-i` checks whether a user has permission to perform certain operations: ```bash # Check to see if I can create pods in any namespace kubectl auth can-i create pods --all-namespaces # Check to see if I can list deployments in my current namespace kubectl auth can-i list deployments.extensions # Check to see if I can do everything in my current namespace ("*" means all) kubectl auth can-i '*' '*' # Check to see if I can get the job named "bar" in namespace "foo" kubectl auth can-i list jobs.batch/bar -n foo ``` * `kubectl auth reconcile` automatically fixes problematic RBAC policies: ```bash # Reconcile rbac resources from a file kubectl auth reconcile -f my-rbac-rules.yaml ``` ## Simulating Other Users kubectl supports you to simulate other users or groups for cluster management operations: ```bash kubectl drain mynode --as=superman --as-group=system:masters ``` This is equivalent to adding following HTTP HEADER when requesting Kubernetes API: ```bash Impersonate-User: superman Impersonate-Group: system:masters ``` ## Event Inspection ```bash # Check all events kubectl get events --all-namespaces # Check events for objects named nginx kubectl get events --field-selector involvedObject.name=nginx,involvedObject.namespace=default # Check service events for nginx kubectl get events --field-selector involvedObject.name=nginx,involvedObject.namespace=default,involvedObject.kind=Service # Check events for a Pod kubectl get events --field-selector involvedObject.name=nginx-85cb5867f-bs7pn,involvedObject.kind=Pod # Sort events by time kubectl get events --sort-by=.metadata.creationTimestamp # Customize events output format kubectl get events --sort-by='.metadata.creationTimestamp' -o 'go-template={{range .items}}{{.involvedObject.name}}{{"\t"}}{{.involvedObject.kind}}{{"\t"}}{{.message}}{{"\t"}}{{.reason}}{{"\t"}}{{.type}}{{"\t"}}{{.firstTimestamp}}{{"\n"}}{{end}}' ``` ## kubectl Plugins The kubectl plugin provides a mechanism to extend kubectl, such as adding new subcommands. The plugin can be written in any language as long as it meets the following criteria: * The plugin resides in `~/.kube/plugins` or a directory specified by the `KUBECTL_PLUGINS_PATH` environment variable * The format of the plugin is 'subdirectory / executable file or script' and the subdirectory must contain a `plugin.yaml` configuration file. For example: ```bash $ tree . └── hello └── plugin.yaml 1 directory, 1 file $ cat hello/plugin.yaml name: "hello" shortDesc: "Hello kubectl plugin!" command: "echo Hello plugins!" $ kubectl plugin hello Hello plugins! ``` You can also use [krew](/en/setup/kubectl) to manage your kubectl plugins. ## Raw URIs kubectl can also be used to directly access raw URIs. For example, you can access the [Metrics API](https://github.com/kubernetes-incubator/metrics-server): * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes` * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods` * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/` * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces//pods/` ## Appendix The Kubectl Installation ```bash # OS X curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl # Linux curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl # Windows curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/windows/amd64/kubectl.exe ``` # Objects Here's an introduction to the key concepts and objects of Kubernetes. * [Autoscaling (HPA)](/en/concepts/objects/autoscaling) * [ConfigMap](/en/concepts/objects/configmap) * [CronJob](/en/concepts/objects/cronjob) * [CustomResourceDefinition](/en/concepts/objects) * [DaemonSet](/en/concepts/objects/daemonset) * [Deployment](/en/concepts/objects/deployment) * [Ingress](/en/concepts/objects/ingress) * [Job](/en/concepts/objects/job) * [LocalVolume](/en/concepts/objects/local-volume) * [Namespace](/en/concepts/objects/namespace) * [NetworkPolicy](/en/concepts/objects/network-policy) * [Node](/en/concepts/objects/node) * [PersistentVolume](/en/concepts/objects/persistent-volume) * [Pod](/en/concepts/objects/pod) * [PodPreset](/en/concepts/objects/podpreset) * [ReplicaSet](/en/concepts/objects/replicaset) * [Resource Quota](/en/concepts/objects/quota) * [Secret](/en/concepts/objects/secret) * [SecurityContext](/en/concepts/objects/security-context) * [Service](/en/concepts/objects/service) * [ServiceAccount](/en/concepts/objects/serviceaccount) * [StatefulSet](/en/concepts/objects/statefulset) * [Volume](/en/concepts/objects/volume) Now, let's take a deep dive into the fascinating world of Kubernetes. This revolutionary open-source platform was designed to automate and enhance an array of digital functions. It can manage tasks like scaling applications, rolling out updated versions, and providing an ideal framework for building distributed systems. Here's a list of key Kubernetes concepts (or objects) and an exploration of how they operate: * Autoscaling (HPA) adjusts the number of virtual machines dynamically to match current demands. * ConfigMap handles the configuration details of applications. * CronJob is the Kubernetes' version of a time-based job scheduler. * CustomResourceDefinition extends the Kubernetes API with custom resources. * DaemonSet ensures that every node operates a copy of a specific pod. * Deployment manages the updates for Pods and ReplicaSets. * Ingress is an API object that manages external access to services within a cluster. * Jobs represents one or more Pods with the intention to execute a set of commands. * LocalVolume represents locally mounted storage resources. * Namespace provides multiples virtual clusters backed by the same physical cluster. * NetworkPolicy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. * Node represents a worker machine in Kubernetes. * PersistentVolume provides storage that outlives the execution of individual pods. * Pod is the smallest and simplest unit in the Kubernetes object model that users create or deploy. * PodPreset injects additional runtime requirements into pods. * ReplicaSet aims to maintain a stable set of running pods. * Resource Quota provides constraints that limit resources consumption per namespace. * Secret stores sensitive data, such as password or keys. * SecurityContext performs per-pod or per-container security attributes. * Service is the abstraction for performing Kubernetes applications over the network. * ServiceAccount provides identifications for processes that run inside a pod. * StatefulSet manages deployment and scaling of a set of Pods. * Lastly, Volume is a directory, possibly with some data in it, which is accessible to an individual Pod. It's okay if you don't become an expert overnight. With time and practice, you'll gain a thorough understanding of these cornerstone concepts and objects of Kubernetes. Let's embark on this exciting learning journey together! # Autoscaling The Horizontal Pod Autoscaling (HPA) system offers a smart solution, enabling automatic extension of the Pod quantity based on CPU usage or an application's custom metrics. It seamlessly supports replication controllers, deployments and replica sets. * Monitor managers survey the resource usage of the metrics every 15 seconds (adjustable via `--horizontal-pod-autoscaler-sync-period`) * It can work with three types of metrics: * Predefined metrics (like Pod's CPU) are calculated as a ratio or usage rate * Custom Pod metrics are calculated as raw value amounts * Custom object metrics * Metrics can be retrieved using Heapster or the customized REST API * It is capable of managing multiple metrics Do note that the extent of our discussion here is limited to Pod's automatic scaling; to comprehend Node's automatic scaling, refer to [Cluster AutoScaler](/en/setup/addon-list/cluster-autoscaler). Before using the HPA, further, it is necessary to ensure that the [**metrics-server**](/en/setup/addon-list/metrics) is properly deployed. ## API Version Comparison Table | Kubernetes Version | Autoscaling API Version | Supported Metrics | | ------------------ | ----------------------- | ----------------- | | v1.5+ | autoscaling/v1 | CPU | | v1.6+ | autoscaling/v2beta1 | Memory and Custom | ## Examples ```bash # This segment demonstrates how to create a pod and service $ kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80 service "php-apache" created deployment "php-apache" created # Here, we create the autoscaler $ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10 deployment "php-apache" autoscaled ... ``` The snippet above walks you through an example; from creating a pod and service, generating an autoscaler, increasing loads to finally witnessing the reduction of load and automatic reduction of pod quantity. This offers an illustrative explanation of how the autoscaling functions. ## Custom Metrics The control manager can be enabled and configured with `--horizontal-pod-autoscaler-use-rest-clients` and `--master` or `--kubeconfig` respectively. Custom metrics API, such as and , should be registered in the API Server Aggregator. For reference, you can check out [k8s.io/metics](https://github.com/kubernetes/metrics) to develop your custom metrics API server. For example, HorizontalPodAutoscaler promises that each Pod will consume 50% of the CPU, 1000pps, and 10,000 requests per second: ## HPA Best Practices Looking for a smoother scaling experience? Follow these best practices: * Set CPU Requests for Containers * Adjust HPA target appropriately, aiming for 30% reserve for applications and containers * Maintain robust Pods and Nodes to avoid frequent rebuilding of Pods * Implement user request load balancing * Monitor resource usage with `kubectl top node` and `kubectl top pod` For more in-depth understanding on this topic, refer to [Ensure High Availability and Uptime With Kubernetes Horizontal Pod Autoscaler and Prometheus](https://www.weave.works/blog/kubernetes-horizontal-pod-autoscaler-and-prometheus). # ConfigMap Your application's performance can hinge on its configuration, which, naturally, may evolve along with your needs. If your application architecture is merged with your configuration, then changing certain settings would mean having to rebuild your mirror file—an inconvenient predicament. Enter the 'ConfigMap' component. Designed to separate your application and configuration, ConfigMap effectively eliminates the need to rebuild your mirror file every time you tweak a setting. ConfigMap works by storing key-value pairs of configuration data, which can either take the form of individual properties or entire configuration files. It's a lot like the 'Secret' component, except it's better suited to handling strings that don't contain sensitive information. ## API Version Compatibility Table | Kubernetes Version | Core API Version | | ------------------ | ---------------- | | v1.5+ | core/v1 | ## Creating a ConfigMap You can enlist the help of `kubectl create configmap` to create a ConfigMap from a file, directory or a key-value string. Alternatively, `kubectl create -f file` can be used to make a ConfigMap from a file. ### Create from Key-Value String ```bash $ kubectl create configmap special-config --from-literal=special.how=very configmap "special-config" created $ kubectl get configmap special-config -o go-template='{{.data}}' map[special.how:very] ``` ### Create from ENV File ```bash $ echo -e "a=b\nc=d" | tee config.env a=b c=d $ kubectl create configmap special-config --from-env-file=config.env configmap "special-config" created $ kubectl get configmap special-config -o go-template='{{.data}}' map[a:b c:d] ``` ### Create from Directory ```bash $ mkdir config $ echo a>config/a $ echo b>config/b $ kubectl create configmap special-config --from-file=config/ configmap "special-config" created $ kubectl get configmap special-config -o go-template='{{.data}}' map[a:a b:b ] ``` ### Create from Yaml/Json File ```yaml apiVersion: v1 kind: ConfigMap metadata: name: special-config namespace: default data: special.how: very special.type: charm ``` ```bash $ kubectl create -f config.yaml configmap "special-config" created ``` ## Using ConfigMap You can incorporate ConfigMap into your 'Pod' via three different methods, either by setting environment variables, setting container command line parameters, or by directly mounting files or directories in 'Volume'. > **Note** > > * ConfigMap must be created before a Pod can reference it > * Invalid keys will be automatically ignored when using `envFrom` > * A Pod can only use ConfigMap within the same namespace First, you'll need to create a ConfigMap: ```bash $ kubectl create configmap special-config --from-literal=special.how=very --from-literal=special.type=charm $ kubectl create configmap env-config --from-literal=log_level=INFO ``` ### Use as Environment Variable ```yaml apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: ["/bin/sh", "-c", "env"] env: - name: SPECIAL_LEVEL_KEY valueFrom: configMapKeyRef: name: special-config key: special.how - name: SPECIAL_TYPE_KEY valueFrom: configMapKeyRef: name: special-config key: special.type envFrom: - configMapRef: name: env-config restartPolicy: Never ``` Once the Pod ends, it'd output: ``` SPECIAL_LEVEL_KEY=very SPECIAL_TYPE_KEY=charm log_level=INFO ``` ### Use as Command Line Arguments To use ConfigMap as command line arguments, you'd have to store the ConfigMap data in environment variables and reference these through `$(VAR_NAME)`. ```yaml apiVersion: v1 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: ["/bin/sh", "-c", "echo $(SPECIAL_LEVEL_KEY) $(SPECIAL_TYPE_KEY)" ] env: - name: SPECIAL_LEVEL_KEY valueFrom: configMapKeyRef: name: special-config key: special.how - name: SPECIAL_TYPE_KEY valueFrom: configMapKeyRef: name: special-config key: special.type restartPolicy: Never ``` Once the Pod ends, it'd output: ``` very charm ``` ### Mount ConfigMap as File or Directory in Volume Directly You can directly mount the created ConfigMap unto a Pod’s /etc/config directory. Here, each key-value pair would generate a file—with the key as the filename and the value as the content. ```yaml apiVersion: v1 kind: Pod metadata: name: vol-test-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: ["/bin/sh", "-c", "cat /etc/config/special.how"] volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: special-config restartPolicy: Never ``` Once the Pod ends, it'd output: ``` very ``` You can also mount this key, special.how, to a relative path /keys/special.level within the /etc/config directory. Any file with the same name would be directly overwritten, while all other keys are left unmounted. ```yaml apiVersion: v1 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: ["/bin/sh","-c","cat /etc/config/keys/special.level"] volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: special-config items: - key: special.how path: keys/special.level restartPolicy: Never ``` Once the Pod ends, it'd output: ``` very ``` Additionally, ConfigMap supports mounting multiple keys in the same directory or multiple directories. For instance, in the example below, special.how and special.type are doubly mounted to /etc/config and special.how is also mounted to /etc/config2. ```yaml apiVersion: v1 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: ["/bin/sh","-c","sleep 36000"] volumeMounts: - name: config-volume mountPath: /etc/config - name: config-volume2 mountPath: /etc/config2 volumes: - name: config-volume configMap: name: special-config items: - key: special.how path: keys/special.level - key: special.type path: keys/special.type - name: config-volume2 configMap: name: special-config items: - key: special.how path: keys/special.level restartPolicy: Never ``` ```bash # ls /etc/config/keys/ special.level special.type # ls /etc/config2/keys/ special.level # cat /etc/config/keys/special.level very # cat /etc/config/keys/special.type charm ``` ### Using Subpath to Mount ConfigMap as Individual File to Directory In general, congfigmap will mount its content as a file after first overwriting the mounted directory. If you want to avoid overwriting files under the original folder, and just want to mount each key from configmap as a file to the directory, you can use the subpath parameter. ```yaml apiVersion: v1 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: nginx command: ["/bin/sh","-c","sleep 36000"] volumeMounts: - name: config-volume mountPath: /etc/nginx/special.how subPath: special.how volumes: - name: config-volume configMap: name: special-config items: - key: special.how path: special.how restartPolicy: Never ``` ```bash root@dapi-test-pod:/# ls /etc/nginx/ conf.d fastcgi_params koi-utf koi-win mime.types modules nginx.conf scgi_params special.how uwsgi_params win-utf root@dapi-test-pod:/# cat /etc/nginx/special.how very root@dapi-test-pod:/# ``` ## Immutable ConfigMap > Immutable ConfigMap went stable in v1.21.0. When a cluster includes a large number of ConfigMap and Secret, a myriad of watch events can intensely boost the load on kube-apiserver and hasten the spread of configuration errors throughout the entire cluster. In this scenario, marking ConfigMap and Secret that don't require frequent modifications as `immutable: true` can sidestep this issue. Principal benefits of immutable ConfigMap include: * Safeguarding your application from the adverse effects of accidental updates. * Amping up your cluster performance by drastically curbing the pressure on kube-apiserver, as Kubernetes ceases to monitor operations on immutable ConfigMap. ```yaml apiVersion: v1 kind: ConfigMap metadata: ... data: ... immutable: true ``` ## Further Reading * [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) # CronJob Imagine a virtual timekeeper, ticking along to the Linux system's crontab, triggering particular tasks to run at the precise time designated. This is the idea behind 'CronJob'. ## API Version Cheat Sheet | Kubernetes Version | Batch API Version | Activated By Default? | | ------------------ | ----------------- | --------------------- | | v1.5-v1.7 | batch/v2alpha1 | No | | v1.8-v1.20 | batch/v1beta1 | Yes | | v1.21+ | batch/v1 | Yes | A word of caution: when executing APIs that aren't activated by default, users must configure `--runtime-config=batch/v2alpha1` in the kube-apiserver. ## CronJob Specs * `.spec.schedule` outlines the schedule of task execution, akin to the [Cron](https://en.wikipedia.org/wiki/Cron) format. * `.spec.jobTemplate` lists the tasks that need running, and mirrors the [Job](/en/concepts/objects/job) format. * `.spec.startingDeadlineSeconds` specifies the deadline for initiating tasks. * `.spec.concurrencyPolicy` delineates the policy for task concurrency, providing three options: Allow, Forbid, and Replace. ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: hello spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: hello image: busybox imagePullPolicy: IfNotPresent command: - /bin/sh - -c - date; echo Hello from the Kubernetes cluster restartPolicy: OnFailure ``` ``` $ kubectl create -f cronjob.yaml cronjob "hello" created ``` You can also use `kubectl run` to create a CronJob: ``` kubectl run hello --schedule="*/1 * * * *" --restart=OnFailure --image=busybox -- /bin/sh -c "date; echo Hello from the Kubernetes cluster" ``` ``` $ kubectl get cronjob NAME SCHEDULE SUSPEND ACTIVE LAST-SCHEDULE hello */1 * * * * False 0 $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE hello-1202039034 1 1 49s $ pods=$(kubectl get pods --selector=job-name=hello-1202039034 --output=jsonpath={.items..metadata.name} -a) $ kubectl logs $pods Mon Aug 29 21:34:09 UTC 2016 Hello from the Kubernetes cluster # When deleting a cronjob, it will also delete its created jobs and pods and stop the creation of new jobs. $ kubectl delete cronjob hello cronjob "hello" deleted ``` ## Additional Resources * [Cron Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) # CustomResourceDefinition The novel feature of the Kubernetes API, CustomResourceDefinition (CRD), offers a seamless way to extend the Kubernetes API without changing the existing code. Effectively, it is a replacement and upgrade for the older ThirdPartyResources (TPR), which was deprecated starting from version v1.8. ## A glance at the API version compatibility table | Kubernetes Versions | Compatible CRD API Versions | | ------------------- | ---------------------------- | | v1.8+ | apiextensions.k8s.io/v1beta1 | ## A dive into CRD through an example Here’s an illustration of creating a CRD, thereby deploying a tailor-made API endpoint at `/apis/stable.example.com/v1/namespaces//crontabs/…`. Let’s break down the sample code. It starts off by specifying the API version and type of resource (kind). The metadata section requires a unique name that aligns with the spec fields provided below. Following the metadata are the spec fields, which provide information about the group name to be used for REST API, versions of the REST API, and the scope. It also includes the names of the custom resources, where 'plural' is used in the URL, 'singular' acts as an alias on the CLI and for display, 'kind' is the CamelCased singular type which is used in your resource manifests, and 'shortNames', which allow shorter strings to match the resource on the CLI. With this API, we can now proceed to create specific CronTab objects. ## Finalizer: a life-jacket for controllers Finalizer works as a life-preserver for controllers to implement asynchronous pre-deletion hooks. It can be specified in the metadata with `metadata.finalizers`. Once specified, any attempt from the client side to delete the object only sets the `metadata.deletionTimestamp` instead of executing the deletion. This will trigger the ongoing CRD controllers, perform some pre-deletion housecleaning activities, remove their own finalizer from the list, and then launch a new delete operation. Only then, the targeted object will be officially deleted. ## Validation: Keeping Standards High From v1.8, the schema-based validation based on [OpenAPI v3 schema](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject) was introduced, which allows us to verify user submissions for compliance. To use this feature, the `--feature-gates=CustomResourceValidation=true` needs to be configured in the kube-apiserver. For instance, the CRD below expects: * `spec.cronSpec` to be a string matching a regular expression * `spec.replicas` to be an integer between 1 and 10 Any deviations from these rules will result in a validation failure error. ## Subresources From v1.10, CRD started supporting the status and scale subresources (Beta), and from v1.11, those are enabled by default. ## Categorizing CRDs Categories are used to group CRD objects, allowing an all-at-once query of all objects belonging to that category with `kubectl get `. ## CRD Controllers Usually, when extending Kubernetes API with CRD, there's also a need to implement a new resource controller to keep track of changes in the new resource and carry out further handling. The [sample-controller](https://github.com/kubernetes/sample-controller) offers an example of a CRD controller, including details like how to register a resource `Foo`, how to create, delete, and query `Foo` objects, and how to track changes in `Foo` resource objects. ## Kubebuilder: The Friendly Neighborhood Framework As we see from the examples above, building a CRD controller from scratch is no mean task. Getting in-depth knowledge of Kubernetes API aside, integrating RBAC, building images, and continuous integration and deployment demand substantial efforts. Here’s when [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder) comes to the rescue. It provides an intuitive framework for CRD controllers and helps generate the resource files needed for image building, continuous integration, and continuous deployment directly. ### Installing Kubebuilder ### How to use #### Starting a project #### Creating an API After this, you need to adjust the `pkg/apis/ship/v1beta1/sloop_types.go` and `pkg/controller/sloop/sloop_controller.go` as per your business requirements. #### Running Test Locally Subsequently, with the help of `ships.k8s.io/v1beta1`, a `Sloop` kind resource can be created. #### Building Images and Deploying Controllers #### Documentation and Testing ## References * [Extend the Kubernetes API with CustomResourceDefinitions](https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) * [CustomResourceDefinition API](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#customresourcedefinition-v1beta1-apiextensions-k8s-io) # DaemonSet A DaemonSet ensures a specific container copy runs on each Node - a way commonly used to deploy cluster logs, monitors, or other system management applications. Stellar examples include: * Log collection systems, like fluentd or logstash. * System monitors such as Prometheus Node Exporter, collectd, New Relic agent, or Ganglia gmond. * System programs like kube-proxy, kube-dns, glusterd, and ceph. ## API version compatibility | Kubernetes version | Deployment version | | ------------------ | ------------------ | | v1.5-v1.6 | extensions/v1beta1 | | v1.7-v1.15 | apps/v1beta1 | | v1.8-v1.15 | apps/v1beta2 | | v1.9+ | apps/v1 | There's an example of using Fluentd to collect logs: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: fluentd-logging spec: selector: matchLabels: name: fluentd-elasticsearch template: metadata: labels: name: fluentd-elasticsearch spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd-elasticsearch image: gcr.io/google-containers/fluentd-elasticsearch:1.20 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers ``` ## Rolling update From version 1.6 onwards, DaemonSets support rolling updates. You can set your update strategy with `.spec.updateStrategy.type`. Two strategies are currently supported: * OnDelete: The default strategy. After updating the template, a new Pod will only be created once the old one has been manually deleted. * RollingUpdate: After the DaemonSet template has been updated, the old Pod is automatically removed and a new one is created. The RollingUpdate strategy enables you to set: * `.spec.updateStrategy.rollingUpdate.maxUnavailable`, defaulting to 1 * `spec.minReadySeconds`, defaulting to 0 ### Rollback From version 1.7 onwards, support for rollback is included. ```bash # Search through historical versions $ kubectl rollout history daemonset # Search for detailed information of a specific historical version $ kubectl rollout history daemonset --revision=1 # Rollback $ kubectl rollout undo daemonset --to-revision= # Search for rollback status $ kubectl rollout status ds/ ``` ## Specifying Node DaemonSet ignores a Node's unschedulable status. There are two ways to ensure a Pod only runs on specified Node nodes: * nodeSelector: Only schedules on Nodes that match the specific label. * nodeAffinity: A more feature-rich Node selector that, for instance, supports set operations. * podAffinity: Schedules on the Node where the Pod meeting conditional criteria is located. ### nodeSelector example First, label the node: ```bash kubectl label nodes node-01 disktype=ssd ``` Then specify `disktype=ssd` as nodeSelector in DaemonSet: ```yaml spec: nodeSelector: disktype: ssd ``` ### nodeAffinity example NodeAffinity currently supports both requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution, which represent mandatory and preferred conditions. The following example represents scheduling on a Node that contains the label `kubernetes.io/e2e-az-name` with a value of e2e-az1 or e2e-az2, and it's preferred the Node also carries the label `another-node-label-key=another-node-label-value`. ```yaml apiVersion: v1 kind: Pod metadata: name: with-node-affinity spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/e2e-az-name operator: In values: - e2e-az1 - e2e-az2 preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: another-node-label-key operator: In values: - another-node-label-value containers: - name: with-node-affinity image: gcr.io/google_containers/pause:2.0 ``` ### podAffinity example PodAffinity selects Nodes based on Pod labels, scheduling only on the Node where the Pod meeting the conditions resides. It supports podAffinity and podAntiAffinity. This feature can be quite convoluted. Take the following example: * It'll schedule on any "Node that contains at least one running Pod tagged with `security=S1`". * It improves its chances of not being scheduled on the "Nodes containing at least one running Pod tagged with `security=S2`". ```yaml apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: security operator: In values: - S1 topologyKey: failure-domain.beta.kubernetes.io/zone podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: security operator: In values: - S2 topologyKey: kubernetes.io/hostname containers: - name: with-pod-affinity image: gcr.io/google_containers/pause:2.0 ``` ## Static Pod Besides using DaemonSet, you can operate specific Pods on each server with Static Pod. This requires the kubelet to specify the manifest directory when launching: ```bash kubelet --pod-manifest-path=/etc/kubernetes/manifests ``` Then place the needed Pod definition file into the specified manifest directory. Note: Static Pods cannot be deleted through the API Server. But, you can automate the deletion of the corresponding Pod by eliminating the manifest file. # Deployment ## Quick Overview Deployments offer a declarative definition for Pods and ReplicaSets, making application management more straightforward as compared to the former ReplicationControllers. ## A Handy Version Guide | Kubernetes Version | Deployment Version | | ------------------ | ------------------ | | v1.5-v1.6 | extensions/v1beta1 | | v1.7-v1.15 | apps/v1beta1 | | v1.8-v1.15 | apps/v1beta2 | | v1.9+ | apps/v1 | For example, you can define a simple nginx application as follows: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 ``` Scale up: ``` kubectl scale deployment nginx-deployment --replicas 10 ``` If your cluster supports horizontal pod autoscaling, you can even set automatic expansion for Deployment: ``` kubectl autoscale deployment nginx-deployment --min=10 --max=15 --cpu-percent=80 ``` Updating images is also straightforward: ``` kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1 ``` Roll back: ``` kubectl rollout undo deployment/nginx-deployment ``` Typical use cases for Deployment include: * Define Deployment to create Pod and ReplicaSet * Roll up and roll back applications * Scale up and scale down * Pause and resume Deployment ## A Brief Explanation of the Deployment Concept Deployment provides a declarative update for Pod and ReplicaSet (the next generation of Replication Controller). You only need to describe what you want the target state to be in the Deployment, and the Deployment controller will help you change the actual state of the Pod and ReplicaSet to your target state. You can define a brand new Deployment, or create a new one to replace the old Deployment. For example: * Use Deployment to create a ReplicaSet. ReplicaSet creates pods in the background. Check the startup status to see if it is successful or not. * Then, declare the new state of the Pod by updating the Deployment's PodTemplateSpec field. This creates a new ReplicaSet, and Deployment will move the pod from the old ReplicaSet to the new ReplicaSet at a controlled rate. * If the current state is unstable, roll back to the previous Deployment revision. Every rollback updates the Deployment's revision. * Scale the Deployment to meet higher loads. * Pause Deployment to apply multiple fixes to PodTemplateSpec and then resume operation. * Determine whether the launch is hung based on the status of Deployment. * Clear unnecessary old ReplicaSet. ## Creating a Deployment The following is an example of Deployment, which creates a ReplicaSet to start 3 nginx pods. Download the sample file and perform the command: ```bash $ kubectl create -f docs/user-guide/nginx-deployment.yaml --record deployment "nginx-deployment" created ``` Setting the `—record` flag of kubectl to `true` can record the command that creates or upgrades the resource in the annotation. This will be useful in the future, for example, to see which commands were executed in each Deployment revision. Executing `get` immediately afterwards will give the following result: ```bash $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 3 0 0 0 1s ``` The output indicates that our desired number of replicas is 3 (based on the configuration in the deployment's `.spec.replicas`). The current number of replicas ( `.status.replicas`) is 0, the newest number of replicas ( `.status.updatedReplicas`) is 0, and the available number of replicas ( `.status.availableReplicas`) is 0. A few seconds later, performing the `get` command again will give the following output: ```bash $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 3 3 3 3 18s ``` As you can see, Deployment has created 3 replicas, all of which are updated (contain the latest pod template), and are available (based on Deployment's `.spec.minReadySeconds` declaration, the minimum number of pods in ready status). Executing `kubectl get rs` and `kubectl get pods` will display the created ReplicaSets (RS) and Pods. ```bash $ kubectl get rs NAME DESIRED CURRENT READY AGE nginx-deployment-2035384211 3 3 0 18s ``` You may notice that the name of a ReplicaSet is always `-`. ```bash $ kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS nginx-deployment-2035384211-7ci7o 1/1 Running 0 18s app=nginx,pod-template-hash=2035384211 nginx-deployment-2035384211-kzszj 1/1 Running 0 18s app=nginx,pod-template-hash=2035384211 nginx-deployment-2035384211-qqcnn 1/1 Running 0 18s app=nginx,pod-template-hash=2035384211 ``` The newly created ReplicaSet will ensure that there are always 3 nginx pods present. **Note:** You must specify the correct pod template label (`app = nginx`) in the Deployment selector. Do not mix it up with other controllers, including other Deployments, ReplicaSets, ReplicationController, and so on. Although **Kubernetes itself does not prevent you from doing this**, if you do, these controllers will fight each other and may result in incorrect behavior. ## Updating Deployment **Note:** Only when the Deployment pod template (such as `.spec.template`) is updated, which includes updating labels or container images in Deployment, will it trigger a rollout. Other updates, such as scaling up the Deployment, do not trigger a rollout. Assuming we now want to use the `nginx:1.9.1` image instead of the original `nginx:1.7.9` image. ```bash $ kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1 deployment "nginx-deployment" image updated ``` We can use the `edit` command to edit the Deployment. We modify `.spec.template.spec.containers[0].image`, changing `nginx:1.7.9` to `nginx:1.9.1`. ```bash $ kubectl edit deployment/nginx-deployment deployment "nginx-deployment" edited ``` To see the status of the rollout, # Ingress This article will help you make sense of terms common to Kubernetes that you may often encounter used interchangeably elsewhere, in order to prevent confusion. * Node: a server in a Kubernetes cluster; * Cluster: a group of servers managed by Kubernetes; * Edge router: a router that routes data packets between a local area network and the internet, with the additional function as a firewall protecting the local network; * Cluster network: a specific implementation of networking that adheres to Kubernetes' [networking model](https://kubernetes.io/docs/admin/networking/), for example, [flannel](https://github.com/coreos/flannel#flannel) and [OVS](https://github.com/openvswitch/ovn-kubernetes); * Service: A Kubernetes service is a group of pods identified by label selectors [Service](https://kubernetes.io/docs/user-guide/services/). Unless stated otherwise, the virtual IP of a service is only accessible within the cluster. ## What is Ingress? Typically, service and pod IP addresses are only accessible within the cluster. External requests must be routed via a load balancer that directs them to a NodePort exposed on a node, which is then handled by the kube-proxy via an edge router. This process either forwards the requests to the relevant pod or discards them, like the illustration below. ``` internet | ------------ [Services] ``` Ingress is simply a set of rules that help route requests entering the cluster, as shown in the following figure. ![image-20190316184154726](/files/m3B3jTK53AortoW4IO3p) Ingress offers load balancing, public URLs, SSL termination, and HTTP routing for services outside the cluster. To set these Ingress rules, a cluster administrator needs to deploy an [Ingress controller](/en/extension/ingress). The controller listens for changes in Ingress and services, and based on the rules, it configures load balancing and makes the necessary access provisions. ## Ingress Format ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress spec: rules: - http: paths: - path: /testpath backend: serviceName: test servicePort: 80 ``` Each Ingress rule needs to be configured. At present, Kubernetes only supports HTTP rules. The above example shows that when a request is made to '/testpath', it gets routed to the service 'test' on port 80. ## API Version Table | Kubernetes Version | Extension Version | | ------------------ | ------------------------- | | v1.5-v1.17 | extensions/v1beta1 | | v1.8-v1.18 | networking.k8s.io/v1beta1 | | v1.19+ | networking.k8s.io/v1 | ## Types of Ingress Based on the configuration of the Ingress Spec, Ingress can be divided into the following types: ### Single-service Ingress Single-service Ingress refers to an Ingress that only points to one backend service without any rules. ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress spec: backend: serviceName: testsvc servicePort: 80 ``` > Note: A single service can be exposed externally by setting `Service.Type=NodePort` or `Service.Type=LoadBalancer`. ### Multi-service Ingress Routing-to-multi-service Ingress refers to different backend services being routed according to the request path. ``` foo.bar.com -> 178.91.123.132 -> / foo s1:80 / bar s2:80 ``` The following Ingress defines the above: ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test spec: rules: - host: foo.bar.com http: paths: - path: /foo backend: serviceName: s1 servicePort: 80 - path: /bar backend: serviceName: s2 servicePort: 80 ``` After creating the ingress with `kubectl create -f`: ```bash $ kubectl get ing NAME RULE BACKEND ADDRESS test - foo.bar.com /foo s1:80 /bar s2:80 ``` ### Virtual Host Ingress Virtual host Ingress refers to different backend services being routed based on different names but sharing the same IP address. ``` foo.bar.com --| |-> foo.bar.com s1:80 | 178.91.123.132 | bar.foo.com --| |-> bar.foo.com s2:80 ``` The following Ingress routes a request based on the [Host Header](https://tools.ietf.org/html/rfc7230#section-5.4) : ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test spec: rules: - host: foo.bar.com http: paths: - backend: serviceName: s1 servicePort: 80 - host: bar.foo.com http: paths: - backend: serviceName: s2 servicePort: 80 ``` > Note: A backend service that has no default rule definition is called the default backend service, which can easily handle 404 pages. ### TLS Ingress TLS Ingress obtains TLS private keys and certificates (named 'tls.crt' and 'tls.key') via Secret to perform TLS termination. If the TLS configuration part of Ingress specifies different hosts, these hosts will be reused on multiple same ports based on the host name specified by the SNI TLS extension—if the Ingress controller supports SNI. Define a secret containing 'tls.crt' and 'tls.key': ```yaml apiVersion: v1 data: tls.crt: base64 encoded cert tls.key: base64 encoded key kind: Secret metadata: name: testsecret namespace: default type: Opaque ``` The secret is referenced in Ingress: ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: no-rules-map spec: tls: - secretName: testsecret backend: serviceName: s1 servicePort: 80 ``` Take note, different Ingress controllers support different TLS functionalities. Please refer to the documentation on [nginx](https://kubernetes.github.io/ingress-nginx/), [GCE](https://github.com/kubernetes/ingress-gce) or any other Ingress controller to learn about their TLS support. ## Updating Ingress You can update Ingress with the `kubectl edit ing name` command: ```bash $ kubectl get ing NAME RULE BACKEND ADDRESS test - 178.91.123.132 foo.bar.com /foo s1:80 $ kubectl edit ing test ``` This opens an editor containing the existing IngressSpec yaml file. After editing and saving, it updates the Kubernetes API server, triggering the Ingress Controller to reconfigure load balancing: ```yaml spec: rules: - host: foo.bar.com http: paths: - backend: serviceName: s1 servicePort: 80 path: /foo - host: bar.baz.com http: paths: - backend: serviceName: s2 servicePort: 80 path: /foo .. ``` After the update: ```bash $ kubectl get ing NAME RULE BACKEND ADDRESS test - 178.91.123.132 foo.bar.com /foo s1:80 bar.baz.com /foo s2:80 ``` Of course, you can also update it with the `kubectl replace -f new-ingress.yaml` command, where new-ingress.yaml is the modified Ingress yaml. ## Ingress Controller The normal operation of Ingress requires the running of an Ingress Controller in the cluster. The Ingress Controller is different from other controllers that automatically start when the cluster is created as part of kube-controller-manager—it requires users to choose an Ingress Controller that suits their cluster, or implement one themselves. Ingress Controller is deployed as a Kubernetes Pod and runs as a daemon, constantly watching the /ingress interface of Apiserver to update Ingress resources to meet Ingress requests. For example, you can use the [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx): ```bash helm install stable/nginx-ingress --name nginx-ingress --set rbac.create=true ``` Other Ingress Controllers available: * [traefik ingress](/en/extension/ingress/service-discovery-and-load-balancing) gives a practical example of a Traefik Ingress Controller * [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx) provides a detailed example of an Nginx Ingress Controller * [kubernetes/ingress-gce](https://github.com/kubernetes/ingress-gce) provides an example of an Ingress Controller for GCE ## Ingress Class Before Ingress Class, to choose a specific Controller for Ingress required adding a special annotation (like kubernetes.io/ingress.class: nginx). But with IngressClass, cluster administrators can pre-create supported Ingress types, which can then be referenced directly in Ingress. ```yaml apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: external-lb spec: controller: example.com/ingress-controller parameters: apiGroup: k8s.example.com kind: IngressParameters name: external-lb ``` ## References * [Kubernetes Ingress Resource](https://kubernetes.io/docs/concepts/services-networking/ingress/) * [Kubernetes Ingress Controller](https://github.com/kubernetes/ingress/tree/master) * [Using NGINX Plus to Load Balance Kubernetes Services](http://dockone.io/article/957) * [Load Balancing Kubernetes with Ingress Controller using NGINX and NGINX Plus](http://www.cnblogs.com/276815076/p/6407101.html) * [Kubernetes Ingress Controller-Træfɪk](https://doc.traefik.io/traefik/providers/kubernetes-ingress/) * [Kubernetes 1.2 and simplifying advanced networking with Ingress](https://kubernetes.io/blog/2016/03/kubernetes-1-2-and-simplifying-advanced-networking-with-ingress/) # Job The key component of Kubernetes system that handles batch handling of short-lived, one-off tasks is known as **Job.** This essential function ensures that one or more Pods successfully execute given tasks. ## API Version Compatibility | Kubernetes Version | Batch API Version | Default Activation | | ------------------ | ----------------- | ------------------ | | v1.5+ | batch/v1 | Yes | ## Job Varieties Kubernetes supports several types of Jobs: * Non-parallel Job: typically creates a single Pod until it ends successfully * Fixed completion Job: Work by setting `.spec.completions`, creating multiple Pods until the number of `.spec.completions` Pods ends successfully. * Parallel Job with a task queue: This type of Job sets `.spec.Parallelism` but doesn't set `.spec.completions`. When all Pods are finished and at least one succeeds, the Job is considered successful. The types of Jobs are categorized based on the settings of `.spec.completions` and `.spec.Parallelism`: | Job Type | Use Example | Behavior | Completions | Parallelism | | ----------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------- | ----------- | ----------- | | One-time Job | Database migration | Creates a single Pod until it ends successfully | 1 | 1 | | Fixed completion Job | Pod processing work queue | Creates a Pod in sequence until the `completions` finish successfully | 2+ | 1 | | Fixed completion Parallel Job | Multiple Pods process work queue simultaneously | Creates multiple Pods in sequence until the `completions` finish successfully | 2+ | 2+ | | Parallel Job | Multiple Pods process the work queue simultaneously | Creates one or more Pods until one ends successfully | 1 | 2+ | ## Job Controller Job Controller is in charge of creating Pods according to the Job Spec and continually monitoring the status of the Pods until they successfully complete the task. If a failure occurs, the decision to create a new Pod and retry the task is determined by the `restartPolicy` (supports only `OnFailure` and `Never`, and doesn't support `Always`). ![Job](/files/3Y7ow20HqR13x7h284P4) ## Job Spec Format * The `spec.template` format is the same as Pod * RestartPolicy only supports `Never` or `OnFailure` * When a single Pod, the job ends by default after the Pod runs successfully * `.spec.completions` flags the number of Pods that need to run successfully for the Job to end, and defaults to 1 * `.spec.parallelism` flags the number of Pods running in parallel and defaults to 1 * `spec.activeDeadlineSeconds` flags the maximum retry time for failed Pods. After this time, they will not continue to retry An example outlining this format: ```yaml apiVersion: batch/v1 kind: Job metadata: name: pi spec: template: metadata: name: pi spec: containers: - name: pi image: perl command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] restartPolicy: Never ``` \[...example content...] ## Indexed Job When Job is used to run distributed tasks, an independent system is typically needed to assign tasks among different worker Pods of the Job. A newly added feature in Kubernetes v1.21, **Indexed Job**, assigns a numerical index to each task and exposes it to each Pod via the annotation `batch.kubernetes.io/job-completion-index`. This feature is enabled by setting `completionMode: Indexed` in the Job spec. ## Pod Auto-Cleanup The TTL Controller is used to automatically clean up Pods that have finished running or are in a failed state. The TTL of a Pod after stopping can be set with `.spec.ttlSecondsAfterFinished`. This feature requires the system time on all nodes (including control nodes) in the cluster to be synchronized. ## Job Pausing and Resuming Beginning from v1.21, the function to pause and resume Jobs is enabled via `.spec.suspend`: ```yaml apiVersion: batch/v1 kind: Job metadata: name: myjob spec: suspend: true parallelism: 1 completions: 5 template: spec: ... ``` \[...example content...] ## Bare Pods Bare Pods are directly created Pods that are not managed by ReplicaSets or ReplicationControllers. These Pods are not rebooted automatically after Node restart, but the Job can make a new Pod to continue the task. Hence, it's recommended to replace bare Pods with Jobs, even for applications that require a single Pod only. ## References * [Jobs - Run to Completion](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) # LocalVolume > Heads-up: This feature is only supported in v1.7 and onwards, and was upgraded to beta in v1.10. LocalVolume is your bridge to local storage devices - be it a disk, partition, or even a humble directory. It's most at home in high-performance, high-reliability environments like distributed storage and databases. It can work smoothly with both block devices and file systems, and you can point it to the right place using `spec.local.path`. Don't sweat about space restrictions for file systems; Kubernetes won't impose any limits. However, LocalVolumes can only play with statically provisioned Persistent Volumes (PVs). Compared to [HostPath](/en/concepts/objects/volume#hostPath), these data volumes are always available for use, since they always land on a specified node thanks to NodeAffinity. Community developers have also crafted a [local-volume-provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume/provisioner), for automating the creation and cleanup of local data volumes. ## Example Below is a StorageClass: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer ``` And here's how you create a local data volume on a hostname named `example-node`: ```yaml # For Kubernetes v1.10 apiVersion: v1 kind: PersistentVolume metadata: name: example-local-pv spec: capacity: storage: 100Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /mnt/disks/ssd1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - example-node ``` ```yaml # For Kubernetes v1.7-1.9 apiVersion: v1 kind: PersistentVolume metadata: name: example-local-pv annotations: "volume.alpha.kubernetes.io/node-affinity": '{ "requiredDuringSchedulingIgnoredDuringExecution": { "nodeSelectorTerms": [ { "matchExpressions": [ { "key": "kubernetes.io/hostname", "operator": "In", "values": ["example-node"] } ]} ]} }', spec: capacity: storage: 5Gi accessModes: - AccessModeReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /mnt/disks/ssd1 ``` Creation of PVC: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: example-local-claim spec: accessModes: - AccessModeReadWriteOnce resources: requests: storage: 5Gi storageClassName: local-storage ``` Pod creation, referencing PVC: ```yaml kind: Pod apiVersion: v1 metadata: name: mypod spec: containers: - name: myfrontend image: nginx volumeMounts: - mountPath: "/var/www/html" name: mypd volumes: - name: mypd persistentVolumeClaim: claimName: example-local-claim ``` ## Limitations * As of the moment, you can't bind multiple PVCs of local data volumes to one Pod (but it's on the to-do list for v1.9). * Scheduling conflicts might happen, especially when there's a shortage of CPU or memory resources (v1.9 aims to tackle this). * The external Provisioner has some trouble detecting the size of a mount point right after starting up (the v1.9 update is expected to bring Mount Propagation feature to solve this). ## Best Practices * For optimal IO isolation, consider allocating a separate disk for each storage volume. * It's advisable to allocate separate partitions for each storage volume to isolate storage space. * Avoid creating Nodes under the same name to prevent new Nodes from identifying PVs already bound to the old Nodes. * Instead of file paths, it's recommended to use UUIDs to eliminate mismatch issues. * For block storage without file systems, opt for unique IDs, like `/dev/disk/by-id/`, to circumvent block device path mismatch problems. ## Additional Readings * For a more in-depth look, check out the [Local Persistent Storage User Guide](https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume). # Namespace Think of a Namespace as a virtual cluster or compartment housing a collection of related resources and objects. This concept allows you to group and categorize entities like deployments, pods, services, and replication controllers based on projects or user groups. By default, these all belong to the 'default' namespace. However, 'nodes', 'persistent volumes', and namespaces themselves are not subordinate to any namespace. You might find Namespaces being put to use to isolate users. For instance, Kubernetes' built-in services typically run in the `kube-system` namespace. ## Mastering Namespace Operations > The Kubernetes command-line tool `kubectl` lets you specify a namespace using the `--namespace` or shorter `-n` option. If you don't specify one, it assumes 'default'. To view resources across all namespaces, set `--all-namespace=true`. ### Searching ```bash $ kubectl get namespaces NAME STATUS AGE default Active 11d kube-system Active 11d ``` Note: Keep an eye on the status - it'll indicate if a namespace is "Active" or in the process of being "Terminated". During the deletion process, the namespace status changes to "Terminating". ### Creating ```bash (1) Go ahead and create it directly from the command line: $ kubectl create namespace new-namespace (2) Or play it traditional and create it via a file: $ cat my-namespace.yaml apiVersion: v1 kind: Namespace metadata: name: new-namespace $ kubectl create -f ./my-namespace.yaml ``` Note: Make sure your namespace name matches this regular expression `[a-z0-9]([-a-z0-9]*[a-z0-9])?` and doesn't exceed 63 characters in length. ### Deleting ```bash $ kubectl delete namespaces new-namespace ``` Take heed: 1. Deleting a namespace automatically takes out all the resources belonging to that namespace as well. 2. The `default` and `kube-system` namespaces are off-limits for deletion. 3. While a PersistentVolume doesn't belong to any namespace, a PersistentVolumeClaim is tied to a specific namespace. 4. The namespace association of an Event depends on its source object. 5. With version v1.7 came the `kube-public` namespace for storing public information, usually in the form of ConfigMaps. ```bash $ kubectl get configmap -n=kube-public NAME DATA AGE cluster-info 2 29d ``` ## For Further Reading * [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) * [Share a Cluster with Namespaces](https://kubernetes.io/docs/tasks/administer-cluster/namespaces/) # NetworkPolicy ## NetworkPolicy for Microservices With the rise of microservices, an increasing number of cloud service platforms are becoming dependent on network communication between numerous modules. Introduced in Kubernetes 1.3, Network Policy provides policy-based network control to isolate applications and reduce attack surfaces. It employs label selectors to simulate traditional segmented networks, controlling traffic between them and managing incoming traffic from external sources. When working with Network Policy, keep in mind: * Version 1.6 and earlier require `extensions/v1beta1/networkpolicies` to be enabled in kube-apiserver. * As of version 1.7, Network Policy has reached General Availability (GA) with the API version `networking.k8s.io/v1`. * Version 1.8 introduced support for **Egress** and **IPBlock**. * Version 1.21 adds **endPort** support for setting the port range (requires configuration `--feature-gates=NetworkPolicyEndPort=true`). * Network plugins such as Calico, Romana, Weave Net, and trireme that support Network Policy are needed. Refer [here](/en/extension/network-policy) for more details. ### API Version Chart | Kubernetes Version | Networking API Version | | :----------------: | :--------------------: | | v1.5 - v1.6 | extensions/v1beta1 | | v1.7+ | networking.k8s.io/v1 | ### Network Policies #### Namespace Isolation By default, all Pods can freely communicate with each other. Each Namespace can configure an independent network policy to isolate traffic between Pods. On v1.7+ versions, creating a Network Policy that matches all Pods serves as the default network policy, such as the default rejection of all Ingress communication between Pods. On the flip side, v1.6 uses Annotations to isolate traffic between all the Pods in a namespace from all Pods' external traffic to the namespace and traffic between Pods within the namespace. #### Pod Isolation It is possible to manage traffic between Pods using label selectors, including namespaceSelector and podSelector. For instance, the following Network Policy does the following: * Allows Pods with the `role=frontend` label in the default namespace to access TCP port 6379 of Pods with the `role=db` label in the default namespace. * Allows all Pods in namespaces with the `project=myprojects` label to access TCP port 6379 of Pods with the `role=db` label in the default namespace. ### Quick Example To see Network Policy in action, let's utilize calico as an instance. First, configure kubelet to use the CNI network plugin. Install the calico network plugin. Then, deploy an nginx service. At this point, nginx can be accessed by other Pods. When we enable the DefaultDeny Network Policy for the default namespace, other Pods (including those outside of the namespace) can't reach nginx anymore: At last, by implementing a network policy that allows access from Pods labelled with `access=true`, only authorized Pods are able to communicate with the nginx service. ## Use Cases #### Blocking Access to Certain Services The network policy denies every other Pod from sending traffic to the appointed service. #### Allowing Only Certain Pods to Access Services The network policy allows only specified Pods to send traffic to the appointed service. #### Prohibit Intercommunication Among Pods in the Same Namespace Disables the ability for Pods within the same namespace to communicate with each other. #### Preventing Other Namespaces From Accessing Services The network policy restricts all Pods outside the namespace from reaching an assigned service. #### Allowing Only Specific Namespace to Access Services The network policy allows only certain namespaces to send traffic to the designated service. #### Enabling External Access to Service The applied network policy allows traffic from the external network to reach a specific service within the Kubernetes cluster. ## Unsupported Use Cases While Network Policy covers a wide spectrum of applications, there are certain scenarios it does not support, including: * Forcing intra-cluster traffic to pass through a common gateway. * Situations related to Transport Layer Security (TLS). * Policies specific to nodes. * Policies that identify targets based on names. * Generating network security event logs. * Blocking localhost access or access requests from the hosting node. ### Further Reading Here are some resources that provide more information on Network Policies in Kubernetes: * [Kubernetes network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) * [Declare Network Policy](https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/) * [Securing Kubernetes Cluster Networking](https://ahmet.im/blog/kubernetes-network-policy/) * [Kubernetes Network Policy Recipes](https://github.com/ahmetb/kubernetes-networkpolicy-tutorial) # Node A Node is the actual host where a Pod runs. This could be a physical or a virtual machine. For Pod management, each Node must at least run a container runtime (like `docker` or `rkt`), `kubelet`, and the `kube-proxy` service. ![node](/files/4lTaspgB0zNEUSQh0J5f) ## Managing Nodes Unlike other resources such as Pods and Namespace, Kubernetes doesn't create a Node. It only manages resources on a Node. Although you can create a Node object using a Manifest (as shown in the yaml below), Kubernetes merely checks if such a Node indeed exists. If the check fails, Pod scheduling doesn't proceed. ```yaml kind: Node apiVersion: v1 metadata: name: 10-240-79-157 labels: name: my-first-k8s-node ``` The Node Controller conducts this check. The Node Controller is responsible for: * Maintaining Node status * Synchronizing Node with Cloud Provider * Assigning container CIDR to Node * Deleting Pods on the Node with `NoExecute` taint By default, kubelet registers itself with the master during startup and creates the Node resource. ## Node Status Each Node includes the following status information: * Address: Including hostname, public IP, and private IP * Conditions: They include OutOfDisk, Ready, MemoryPressure, and DiskPressure * Capacity: Available resources on the Node, including CPU, memory, and the total number of Pods * Info: It includes kernel version, container engine version, OS type, etc. ## Taints and Tolerations Taints and Tolerations ensure Pods are not scheduled on inappropriate Nodes. Taint is applied to a Node, and toleration is applied to a Pod (Toleration is optional). For example, you can use the taint command to add taints to node1: ```bash kubectl taint nodes node1 key1=value1:NoSchedule kubectl taint nodes node1 key1=value2:NoExecute ``` For the specific usage of Taints and Tolerations, please refer to the [scheduler section](). ## Node Maintenance Mode Marking a Node as unschedulable does not affect the Pods running on it. This feature is very useful when maintaining a Node: ```bash kubectl cordon $NODENAME ``` ## Graceful Node Shutdown When `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are configured, Kubelet will detect Node shutdown status based on systemd events, and automatically terminate the running Pods on it (ShutdownGracePeriodCriticalPods needs to be less than ShutdownGracePeriod). Note, both parameters are configured as 0 by default, which means the graceful shutdown feature is off by default. For example, if ShutdownGracePeriod is set to 30s, and ShutdownGracePeriodCriticalPods is set to 10s, the Kubelet delays the Node shutdown by 30 seconds. During the shutdown, the first 20 (30-10) seconds are reserved to terminate regular Pods, while the last 10 seconds are saved to terminate critical Pods. ## Forced Node Shutdown In circumstances where the Node experiences an anomaly, Kubelet may not have a chance to detect and perform a graceful shutdown. In such scenarios, StatefulSet cannot create a new Pod with the same name, if the Pod uses a volume, then VolumeAttachments will not be deleted from the original shutdown Node, hence these Pods' volumes are unable to be mounted on new running Nodes. Forced Node shutdown is specially designed to solve these problems. Users can manually add `node.kubernetes.io/out-of-service` taint to a Node with `NoExecute` or `NoSchedule` effect, marking it as incapable of providing services. If the `NodeOutOfServiceVolumeDetach` feature is enabled on kube-controller-manager, and respective toleration is not set on Pods, these Pods will be forcibly deleted and volume detachment operation will immediately proceed for terminated Pods on that Node. As a result, Pods that were on incapable Nodes can quickly recover on other Nodes. ## References * [Kubernetes Node](https://kubernetes.io/docs/concepts/architecture/nodes/) * [Taints and Tolerations](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature) # PersistentVolume PersistentVolume (PV) and PersistentVolumeClaim (PVC) offer a handy tool to manage persistent volumes: PV provides network storage resources, and PVC requests these resources. As such, setting a persistent workflow involves configuring the underlying file system or cloud data volume, creating a persistent data volume, and then creating a PVC to link your Pods with the data volume. With PV and PVC, Pods and data volumes can be decoupled, meaning that Pods don't need to know the exact file system or the persistent engine that supports them. ## Volume Lifecycle The lifecycle of a volume goes through five stages: 1. Provisioning - the creation of PVs, can be done directly (statically) or dynamically using a StorageClass. 2. Binding - Assigning PVs to PVCs 3. Using - Pods can use the volume via the PVC, and stop the deletion of a PVC that is in use via the admission control of StorageObjectInUseProtection (PVCProtection for versions 1.9 and earlier). 4. Releasing - Pods release the volume and delete the PVC 5. Reclaiming - The PV is retrieved. It can be retained for future use, or it can be directly deleted from the cloud storage. Finally, both the PV and the backend storage are deleted. Based on these 5 stages, we have four volume statuses: * Available * Bound * Released (PVC unbound, but reclaim policy not yet executed) * Failed ## API Version Comparison Chart | Kubernetes Version | PV / PVC Version | StorageClass Version | | ------------------ | ---------------- | ---------------------- | | v1.5-v1.6 | core/v1 | storage.k8s.io/v1beta1 | | v1.7+ | core/v1 | storage.k8s.io/v1 | ## PV A PersistentVolume (PV) is a piece of network storage residing within the cluster. Similar to a Node, it's a resource of the cluster. PV shares similarities with Volume but has a lifecycle that's independent of Pods. Let's take the example of an NFS-based PV: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle nfs: path: /tmp server: 172.17.0.2 ``` ## StorageClass In the existence of many volumes, creating an NFS Volume manually is not very convenient. Kubernetes also provides a [StorageClass](https://kubernetes.io/docs/user-guide/persistent-volumes/#storageclasses) to dynamically create PVs, saving administrators time and encapsulating different types of storage for PVCs to select. ## PVC While PV is a storage resource, a PersistentVolumeClaim (PVC) is a request for a PV. PVCs are like Pods: Pods consume Node resources and PVCs consume PV resources. They can request specific storage sizes and access modes, just as Pods can request CPU and memory resources. ## Expanding PV Space Starting from v1.8, Kubernetes supports expanding PV space. It allows you to expand the size of a PV without losing data or restarting containers. Please note that the current implementation only supports PVs that don't need to adjust the file system size and supports a few storage types. ## Block Storage (Raw Block Volume) Starting from v1.9, Kubernetes has added a new Raw Block Volume feature. Raw Block Volume enables raw block devices to be used as K8s volumes. Note that before you use this feature, you need to enable the BlockVolume feature for kube-apiserver, kube-controller-manager, and kubelet. ## StorageObjectInUseProtection Starting from v1.11, when you enable the admission control StorageObjectInUseProtection, PVs and PVCs that are in use will not be deleted immediately after the delete command is issued. Instead, they'll go through a graceful termination process once the users delete the objects. More detailed information can be found in the reference section below. ## Reference Documentation [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) [Dynamic Volume Provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) [Kubernetes CSI Documentation](https://kubernetes-csi.github.io/docs/) [Volume Snapshots Documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) # Pod Pod is a group of closely related containers that share IPC and Network namespaces, and it is the basic unit for scheduling in Kubernetes. The design concept of Pod is to support multiple containers sharing network and file system within a Pod, which can be combined to provide services through inter-process communication and file sharing. ![pod](/files/8s7rIluTRpIBSzXYg0lb) Characteristics of Pod: * Contains multiple containers that share IPC and Network namespaces, allowing direct communication via localhost. * All containers within a Pod have access to shared Volumes, enabling access to shared data. * No fault tolerance: Once directly created Pods are scheduled and bound to Nodes, they will not be rescheduled even if the Node fails (instead, they will be automatically deleted). Therefore, it is recommended to use controllers such as Deployment or DaemonSet for fault tolerance. * Graceful termination: When a Pod is deleted, its processes inside receive SIGTERM signals first and wait for a certain period of time (grace period) before being forcefully stopped if they are still running. * Privileged containers (configured through SecurityContext) have permissions to modify system configurations (widely used in network plugins). > Kubernetes v1.8+ supports sharing PID namespace between containers. It requires docker >= 1.13.1 with kubelet configured as `--docker-disable-shared-pid=false`. > > In Kubernetes v1.10+, `--docker-disable-shared-pid` has been deprecated. To enable PID namespace sharing, set ShareProcessNamespace in v1.PodSpec as true as shown below: > > ```yaml > spec: > shareProcessNamespace: true > ``` ## Pod Definition Describe the running environment and desired state of a Pod and its containers through YAML or JSON. For example, a simple nginx pod can be defined as follows: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 ``` > In production environments, it is recommended to use controllers such as Deployment, StatefulSet, Job, or CronJob to create Pods instead of directly creating Pods. ### Docker Image Support Currently, Kubernetes only allows the use of Docker images to create containers but it does not support all the behaviors defined by [Dockerfile](https://docs.docker.com/engine/reference/builder/). This can be seen in the table below: | Dockerfile Directive | Description | Supported | Notes | | -------------------- | -------------------------------------------------- | --------- | ---------------------------------------------- | | ENTRYPOINT | Launch command | Yes | containerSpec.command | | CMD | Argument list for command | Yes | containerSpec.args | | ENV | Environment variables | Yes | containerSpec.env | | EXPOSE | Port exposed outside | No | Use containerSpec.ports.containerPort instead | | VOLUME | Data volume | Yes | Use volumes and volumeMounts | | USER | Process running user and user group | Yes | securityContext.runAsUser/supplementalGroups | | WORKDIR | Working directory | Yes | containerSpec.workingDir | | STOPSIGNAL | Signal sent to process when stopping the container | Yes | SIGKILL | | HEALTHCHECK | Health check | No | Use livenessProbe and readinessProbe instead | | SHELL | SHELL to run the launch command | No | Use default SHELL for the image launch command | ## Pod Lifecycle Kubernetes abstracts the status of Pod using `PodStatus.Phase` (but doesn't directly reflect all container statuses). Possible Phases include: * Pending: The Pod has been created by the API Server, but one or more containers have not yet been created, including the process of downloading the image over the network. * Running: All containers in the Pod have been created and scheduled on the Node, but at least one container is still running or starting. * Succeeded: After the Pod is scheduled on the Node, all run successfully and will not restart. * Failed: All containers in the Pod have been terminated, but at least one container has exited with failure (i.e., the exit code is not 0 or was terminated by the system). * Unknown: The status is unknown because the Pod cannot be obtained normally due to some reasons, usually because the apiserver cannot communicate with the kubelet. The `restartPolicy` in PodSpec can be set to determine whether to restart the exited Pod. The options include `Always`, `OnFailure`, and `Never`. For instance: * Single container Pod, when the container exits successfully, different `restartPolicy` actions are: * Always: Restart Container; Pod `phase` remains Running. * OnFailure: Pod `phase` becomes Succeeded. * Never: Pod `phase` becomes Succeeded. * Single container Pod, when the container exits on failure, different `restartPolicy` actions are: * Always: Restart Container; Pod `phase` remains Running. * OnFailure: Restart Container; Pod `phase` remains Running. * Never: Pod `phase` becomes Failed. * 2-container Pod, when one container is running and the other exits on failure, different `restartPolicy` actions are * Always: Restart Container; Pod `phase` remains Running. * OnFailure: Restart Container; Pod `phase` remains Running. * Never: Do not restart Container; Pod `phase` remains Running. * 2-container Pod, when one container stops and the other exits on failure, different `restartPolicy` actions are * Always: Restart Container; Pod `phase` remains Running. * OnFailure: Restart Container; Pod `phase` remains Running. * Never: Pod `phase` becomes Failed. * Single container Pod, when the container is memory-poor (OOM), different `restartPolicy` actions are * Always: Restart Container; Pod `phase` remains Running. * OnFailure: Restart Container; Pod `phase` remains Running. * Never: Record the failure event; Pod `phase` becomes Failed. * Pod is still running, but disk access is unavailable * Terminate all containers * Pod `phase` becomes Failed * If the Pod is managed by a controller, a new one will be recreated and scheduled on another node. * Pod is running, but Node is inaccessible due to network partition failure, * Node controller waits for Node event timeout * Node controller sets Pod `phase` to Failed. * If the Pod is managed by a controller, a new one will be recreated and scheduled on another Node. ## Using Volume The Volume can provide persistent storage for containers, like this: ```yaml apiVersion: v1 kind: Pod metadata: name: redis spec: containers: - name: redis image: redis volumeMounts: - name: redis-storage mountPath: /data/redis volumes: - name: redis-storage emptyDir: {} ``` Refer to [Volume](/en/concepts/objects/volume) for more methods to mount storage volumes. ## Private Image When using a private image, you need to create a docker registry secret and reference it in the container. Create docker registry secret: ```bash kubectl create secret docker-registry regsecret --docker-server= --docker-username= --docker-password= --docker-email= ``` When referencing the docker registry secret, there are two optional ways: The first is to reference the secret directly in the Pod description file: ```yaml apiVersion: v1 kind: Pod metadata: name: private-reg spec: containers: - name: private-reg-container image: dregistry.azurecr.io/acr-auth-example imagePullSecrets: - name: acr-auth ``` The second is to add the secret to the service account and then reference it through the service account (usually the default service account of a namespace): ```bash $ kubectl get secrets myregistrykey $ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "myregistrykey"}]}' $ kubectl get serviceaccounts default -o yaml apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: 2015-08-07T22:02:39Z name: default namespace: default selfLink: /api/v1/namespaces/default/serviceaccounts/default uid: 052fb0f4-3d50-11e5-b066-42010af0d7b6 secrets: - name: default-token-uudge imagePullSecrets: - name: myregistrykey ``` ## RestartPolicy Three types of RestartPolicy are supported * Always: When the container is inoperative, Kubelet restarts the container automatically. This is the default RestartPolicy value. * OnFailure: When the container terminates and the exit code is not 0, Kubelet restarts. * Never: No matter what the circumstance, Kubelet will not restart the container. Note, the restart here means a local restart on the Node where the Pod is located and will not be scheduled on other nodes. ## Environment Variables Environment variables provide important resources for containers, including basic information about the container and Pod, and information about services in the cluster: (1) hostname The `HOSTNAME` environment variable saves the Pod's hostname. (2) Basic information of the container and Pod The name, namespace, IP of the Pod, as well as the resource limits for the container, etc. can be obtained and stored in environment variables using the [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/). ```yaml apiVersion: v1 kind: Pod metadata: name: test spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: ["sh", "-c"] args: - env resources: requests: memory: "32Mi" cpu: "125m" limits: memory: "64Mi" cpu: "250m" env: - name: MY_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: MY_POD_SERVICE_ACCOUNT valueFrom: fieldRef: fieldPath: spec.serviceAccountName - name: MY_CPU_REQUEST valueFrom: resourceFieldRef: containerName: test-container resource: requests.cpu restartPolicy: Never ``` (3) Information about services in the cluster The environment variables of the container can also reference the information of all services created before the container runs. For example, the default Kubernetes service corresponds to the following environment variables: ```bash KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1 KUBERNETES_SERVICE_HOST=10.0.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT=tcp://10.0.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_PORT=443 ``` Due to the limitations of environment variable creation order (the environment variable does not include services created later), it is recommended to use [DNS](/en/concepts/objects/pod) to resolve services. ## Pulling Strategy of Images Kubernetes supports three image pull policies: * Always: The system will pull the image from the repository regardless of whether it exists locally or not. If the image has changed, it overrides the existing one. Otherwise, it leaves the local image unchanged. * Never: The system only uses local images and won't pull from the repository. If the local image is missing, the Pod will fail to run. * IfNotPresent: The system will only pull from the repository if the local image doesn't exist. This is the default value for the ImagePullPolicy. Important Notes: * The default setting is `IfNotPresent`, but for images with the `:latest` tag, the default is `Always`. * Docker verifies during the image pull process. If the MD5 hash of the image hasn't changed, it won't pull the image data. * In a production environment, the usage of the `:latest` tag should be avoided as much as possible, while in a development environment, the `:latest` tag can be used to automatically pull the latest images. ## DNS Access Strategy By setting the dnsPolicy parameter, you can control how the containers in a pod access DNS. * ClusterFirst: It prioritizes queries based on the cluster domain suffix (for example, `default.svc.cluster.local`) via kube-dns (the default policy) * Default: It prioritizes queries from the DNS configured within the Node. ## Using the Host's IPC (Inter-Process Communication) Namespace By setting `spec.hostIPC` to true, your Pod can use the host's IPC namespace. By default, this is set to false. ## Using the Host's Network Namespace By setting `spec.hostNetwork` to true, your Pod can use the host's network namespace. By default, this is set to false. ## Using the Host's PID (Process ID) Space By setting `spec.hostPID` to true, your Pod can use the host's PID namespace. By default, this is set to false. ```yaml apiVersion: v1 kind: Pod metadata: name: busybox1 labels: name: busybox spec: hostIPC: true hostPID: true hostNetwork: true ``` ## Setting the Pod's hostname The hostname of a Pod is set by the `spec.hostname` parameter. If it's not specified, the value of the `metadata.name` parameter is used as the Pod's hostname. ## Setting the Pod's Subdomain The `spec.subdomain` parameter can be used to set a Pod's subdomain. By default, this is blank. For instance, to specify the hostname as busybox-2 and subdomain as default-subdomain, the full domain name becomes `busybox-2.default-subdomain.default.svc.cluster.local`. This can also be shortened to `busybox-2.default-subdomain.default`: ```yaml apiVersion: v1 kind: Pod metadata: name: busybox2 labels: name: busybox spec: hostname: busybox-2 subdomain: default-subdomain containers: - image: busybox command: - sleep - "3600" name: busybox ``` Important Notes: * By default, the DNS generates an A record for the Pod in the format `pod-ip-address.my-namespace.pod.cluster.local`, like `1-2-3-4.default.pod.cluster.local` * In the above example, you also need to create a headless service named `default-subdomain` (the same name as subdomain) in the default namespace. Otherwise, other Pods won't be able to access this Pod via the full domain name - only the Pod itself would be able to do so. ```yaml kind: Service apiVersion: v1 metadata: name: default-subdomain spec: clusterIP: None selector: name: busybox ports: - name: foo # Actually, no port is needed. port: 1234 targetPort: 1234 ``` Be aware, you must set at least one service port for the headless service (headless `spec.ports`), even if it seems like it is not needed, to enable the full domain name function for communication between Pods. ## Setting the Pod's DNS Options ```yaml apiVersion: v1 kind: Pod metadata: namespace: default name: dns-example spec: containers: - name: test image: nginx dnsPolicy: "None" dnsConfig: nameservers: - 1.2.3.4 searches: - ns1.svc.cluster.local - my.dns.search.suffix options: - name: ndots value: "2" - name: edns0 ``` ## Resource Limits Kubernetes uses cgroups to limit the computational resources of containers, such as CPU and memory, including requests (scheduling the pod to a node with enough resources. If the criteria cannot be met, the scheduling will fail) and limits: * `spec.containers[].resources.limits.cpu`: The CPU limit, which can briefly exceed, and the container won't be stopped * `spec.containers[].resources.limits.memory`: The memory limit, which can't be exceeded; if it is, the container may be terminated or moved to another machine with enough resources * `spec.containers[].resources.limits.ephemeral-storage`: The limit for ephemeral storage (container writable layer, logs, and EmptyDir, etc); when this limit is exceeded, the Pod will be evicted * `spec.containers[].resources.requests.cpu`: CPU request, is the basis for scheduling CPU resources, which can exceed * `spec.containers[].resources.requests.memory`: Memory request, is the basis for scheduling memory resources, which can exceed; but if it does, the container may be among the first to be cleaned up when the Node's memory is insufficient * `spec.containers[].resources.requests.ephemeral-storage`: Request for ephemeral storage (container writable layer, logs, and EmptyDir, etc); it's used as a basis for scheduling container storage A few important points: * The unit for CPU is the number of CPUs. `millicpu (m)` is used to represent situations when you have less than one CPU, eg. `500m = 500millicpu = 0.5cpu`, and one CPU is equivalent to * a vCPU on AWS * a Core on GCP * a vCore on Azure * a hyper-threading on a physical machine * The units for memory include `E, P, T, G, M, K, Ei, Pi, Ti, Gi, Mi, Ki`, etc. * Starting from v1.10, you can set the `kubelet --cpu-manager-policy=static` for Guaranteed Pods (meaning that requests.cpu and limits.cpu are equal) to bind CPU (through cpuset cgroups). ## Health Check To ensure that the container is functioning properly after being deployed, Kubernetes offers two probes (Probes) to detect the status of the container: * LivenessProbe: Detects whether the application is healthy. If it's unhealthy, it deletes and recreates the container. * ReadinessProbe: Checks whether the application has started and is serving normally. If it's not fully functional, it won't receive traffic from Kubernetes Services, i.e., the Pod will be removed from the Service endpoint. Kubernetes supports three methods for executing probes: * exec: Executes a command in the container. If the command exit code is `0`, it means the probe succeeded; otherwise, it generally indicates a failure * tcpSocket: Performs a TCP check on the specified container IP and port. If the port is open, it means the probe succeeded; otherwise, it suggests a failure * httpGet: Performs an HTTP Get request on the specified container IP, port, and path. If the returned status code is in the `[200,400)` range, it means the probe succeeded, otherwise, it suggests a failure ## Init Container A Pod can have multiple containers, and while applications run inside these containers, there can be one or more Init containers that start before the application containers. Init containers perform their tasks of initialization before all the other containers run (run-to-completion). If a Pod has multiple Init containers specified, they will run sequentially one by one. Every Init container must succeed before the next one can run. When all the Init containers have completed their tasks successfully, Kubernetes starts the Pod and runs the application containers in the usual manner. As Init containers can have separate images from the application containers, there are several benefits to setting init container startup related code: * They can contain and run utility tools, which are not recommended to be included in the application's container image on security grounds. * They can contain utility tools and customized codes for installation, which is not allowed in the application image. For instance, there is no need to create an image FROM another image, you just need to use tools like sed, awk, python, or dig during the installation process. * Application images can separate and offload the tasks of creating and deploying roles, thereby obviating the need to build a separate image to combine them. * They use Linux Namespace, so they have a distinct filesystem view from the application containers. This means they can access Secrets that are denied to application containers. * They complete their tasks before the application containers start, and while application containers run concurrently. Hence, Init containers offer a simple way to block or delay the launch of application containers until all the preconditions have been met. The maximum value between the following two is selected as the resource calculation for Init containers: * The maximum value of resource usage among all Init containers * The sum of resource usage of all containers in the Pod Init containers' restart strategy: * If the Init container's execution fails and the Pod's restartPolicy is set to Never, the Pod will be in a failed state. Otherwise, it will keep retrying all Init containers until each has succeeded. * If the Pod abruptly exits, when it is pulled again, the Init containers will also be re-executed. Hence, the tasks performed in the Init containers should be idempotent (they can be applied multiple times without changing the result beyond the initial application). ## Container Lifecycle Hooks Container Lifecycle Hooks listen to specific events in the container's lifecycle and execute the registered callback functions when these events occur. Kubernetes supports two kinds of hooks: * postStart: Executes immediately after a container is created, but note that it is performed asynchronously and cannot be guaranteed to run before the ENTRYPOINT. If it fails, the container will be killed, and whether it is restarted depends on the RestartPolicy * preStop: Executes prior to the termination of a container, often used for resource cleanup. If this fails, the container will also be killed The callback function of the hook supports two methods: * exec: Executes a command in the container. If the command's exit status code is `0`, it means the execution was successful, otherwise it indicates a failure * httpGet: Sends a GET request to a specified URL. If the returned HTTP status code is in the `[200, 400)` window, it means the request was successful, otherwise it indicates a failure ## Using Capabilities By default, containers are run in a non-privileged manner, for example, they cannot create virtual network adapters, or configure virtual networks from within. Kubernetes provides a mechanism to alter [Capabilities](http://man7.org/linux/man-pages/man7/capabilities.7.html), allowing you to add to or remove from containers as needed. For example, the configuration below adds the `CAP_NET_ADMIN` and removes the `CAP_KILL` from the container. ```yaml apiVersion: v1 kind: Pod metadata: name: cap-pod spec: containers: - name: friendly-container image: "alpine:3.4" command: ["/bin/sleep", "3600"] securityContext: capabilities: add: - NET_ADMIN drop: - KILL ``` ## Limiting Network Bandwidth You can limit the network bandwidth of a Pod by adding the `kubernetes.io/ingress-bandwidth` and `kubernetes.io/egress-bandwidth` annotations to the Pod ```yaml apiVersion: v1 kind: Pod metadata: name: qos annotations: kubernetes.io/ingress-bandwidth: 3M kubernetes.io/egress-bandwidth: 4M spec: ... ``` ## Scheduling Pods to Specific Nodes You can schedule Pods to preferred nodes through nodeSelector, nodeAffinity, podAffinity, and Taints & tolerations. You can also set the nodeName parameter to schedule the Pod to a specific node. For example, with nodeSelector, you can first label the node: ```bash kubectl label nodes disktype=ssd ``` Next, specify that you want the Pod to run only on nodes with the `disktype=ssd` label: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: env: test spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent nodeSelector: disktype: ssd ``` For the usage methods of nodeAffinity, podAffinity, and Taints & tolerations, please refer to the [Scheduler chapter](/en/concepts/components/scheduler). ## Custom hosts By default, the containers' `/etc/hosts` is automatically generated by kubelet, and only includes localhost and podName. Modifying `/etc/hosts` directly within the container is not recommended, as it will be overwritten when the Pod starts or restarts. From v1.7 onwards, you can add to hosts' content through `pod.Spec.HostAliases`, for example: ```yaml apiVersion: v1 kind: Pod metadata: name: hostaliases-pod spec: hostAliases: - ip: "127.0.0.1" hostnames: - "foo.local" - "bar.local" - ip: "10.1.2.3" hostnames: - "foo.remote" - "bar.remote" containers: - name: cat-hosts image: busybox command: - cat args: - "/etc/hosts" ``` ## HugePages ```yaml apiVersion: v1 kind: Pod metadata: generateName: hugepages-volume- spec: containers: - image: fedora:latest command: - sleep - inf name: example volumeMounts: - mountPath: /hugepages name: hugepage resources: limits: hugepages-2Mi: 100Mi volumes: - name: hugepage emptyDir: medium: HugePages ``` ## Sysctls ```yaml apiVersion: v1 kind: Pod metadata: name: sysctl-example spec: securityContext: sysctls: - name: kernel.shm_rmid_forced value: "0" - name: net.ipv4.route.min_pmtu value: "552" - name: kernel.msgmax value: "65536" ... ``` ## Pod Timezone ```yaml apiVersion: v1 kind: Pod metadata: name: sh namespace: default spec: containers: - image: alpine stdin: true tty: true volumeMounts: - mountPath: /etc/localtime name: time readOnly: true volumes: - hostPath: path: /etc/localtime type: "" name: time ``` ## References * [What is Pod?](https://kubernetes.io/docs/concepts/workloads/pods/pod/) * [Kubernetes Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) * [DNS Pods and Services](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) # PodPreset The life of a Kubernetes pod can get a lot easier with PodPreset – a stellar utility that enables the injection of additional information such as environment variables and storage volumes into Pods with specified labels. This means that you no longer need to set up repetitive information for each Pod in your templates! Even better – you can prevent them from being tampered with the PodPreset by adding the annotation `podpreset.admission.kubernetes.io/exclude: "true"` to your Pods. ## Aligning API Versions | Kubernetes Version | API Version | Default Status | | ------------------ | ------------------------ | -------------- | | v1.6+ | settings.k8s.io/v1alpha1 | No | ### Activating PodPreset * Activate API with `kube-apiserver --runtime-config=settings.k8s.io/v1alpha1=true` * Enable admission control with `--enable-admission-plugins=..,PodPreset` ## Diving into PodPreset Examples Suppose you're using a PodPreset to add environment variables and storage volumes: ```yaml kind: PodPreset apiVersion: settings.k8s.io/v1alpha1 metadata: name: allow-database namespace: myns spec: selector: matchLabels: role: frontend env: - name: DB_PORT value: "6379" volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {} ``` And you submit a Pod: ```yaml apiVersion: v1 kind: Pod metadata: name: website labels: app: website role: frontend spec: containers: - name: website image: ecorp/website ports: - containerPort: 80 ``` After going through the `PodPreset` admission control, the Pod automatically acquires additional environment variables and storage volumes: ```yaml apiVersion: v1 kind: Pod metadata: name: website labels: app: website role: frontend annotations: podpreset.admission.kubernetes.io/allow-database: "resource version" spec: containers: - name: website image: ecorp/website volumeMounts: - mountPath: /cache name: cache-volume ports: - containerPort: 80 env: - name: DB_PORT value: "6379" volumes: - name: cache-volume emptyDir: {} ``` ## Checking Out ConfigMap Examples When dealing with ConfigMaps: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: etcd-env-config data: number_of_members: "1" initial_cluster_state: new initial_cluster_token: DUMMY_ETCD_INITIAL_CLUSTER_TOKEN discovery_token: DUMMY_ETCD_DISCOVERY_TOKEN discovery_url: http://etcd_discovery:2379 etcdctl_peers: http://etcd:2379 duplicate_key: FROM_CONFIG_MAP REPLACE_ME: "a value" ``` And using PodPresets: ```yaml kind: PodPreset apiVersion: settings.k8s.io/v1alpha1 metadata: name: allow-database namespace: myns spec: selector: matchLabels: role: frontend env: - name: DB_PORT value: 6379 - name: duplicate_key value: FROM_ENV - name: expansion value: $(REPLACE_ME) envFrom: - configMapRef: name: etcd-env-config volumeMounts: - mountPath: /cache name: cache-volume - mountPath: /etc/app/config.json readOnly: true name: secret-volume volumes: - name: cache-volume emptyDir: {} - name: secret-volume secretName: config-details ``` Upon submitting a Pod and applying `PodPreset` admission control, your Pod now automatically includes ConfigMap environment variables: ```yaml apiVersion: v1 kind: Pod metadata: name: website labels: app: website role: frontend annotations: podpreset.admission.kubernetes.io/allow-database: "resource version" spec: containers: - name: website image: ecorp/website volumeMounts: - mountPath: /cache name: cache-volume - mountPath: /etc/app/config.json readOnly: true name: secret-volume ports: - containerPort: 80 env: - name: DB_PORT value: "6379" - name: duplicate_key value: FROM_ENV - name: expansion value: $(REPLACE_ME) envFrom: - configMapRef: name: etcd-env-config volumes: - name: cache-volume emptyDir: {} - name: secret-volume secretName: config-details ``` ## Example: Changing Pod Time Zone This powerful utility even allows you to change the time zone for all Pods labelled `tz: shanghai` to Shanghai time zone, as shown in the given example: ```yaml kind: PodPreset apiVersion: settings.k8s.io/v1alpha1 metadata: name: tz-shanghai namespace: default spec: selector: matchLabels: tz: shanghai volumeMounts: - mountPath: /etc/localtime name: tz-config volumes: - name: tz-config hostPath: path: /usr/share/zoneinfo/Asia/Shanghai ``` This demonstrates how PodPreset carries the potential to greatly simplify Kubernetes usage, so it's definitely time to give it a try! # ReplicaSet Imagine setting up a number of identical versions of your application (let's call these 'twins') to ensure constant service even in case of an occasional hiccup! Initially, Kubernetes had something termed as ReplicationController (also known as 'rc') just for this purpose. It ensured that the number of 'twins' for your application remained constant. If a twin behaved poorly or exited prematurely, rc would instantly replace it with a new one; if there were extra twins that are of no use anymore, it would quietly retire them. ReplicationController was a superhero, assisting with maintaining the twin-count, flexibly scaling up and down, smoothly upgrading versions, and tracking multiple versions of your app. But every superhero needs an upgrade! So, in newer versions of Kubernetes, we have ReplicaSet (or 'rs'). Don't be fooled by the name change, ReplicaSet is essentially the same superhero as ReplicationController, with a small upgrade - it supports set-based selectors (while ReplicationController only supported equality-based selectors). While you could use ReplicaSet standalone, it's recommended to let Deployment manage it. This way, you don't have to worry about any compatibility issues (like ReplicaSet not supporting rolling-update, which Deployment does). Plus, Deployment comes with extra perks like version tracking, rolling back, pausing upgrades, and more. You can find a detailed introduction and usage guide of Deployment [here](/en/concepts/objects/deployment). ## API Versions: A Comparative Study | Kubernetes Version | Deployment Version | | ------------------ | ------------------ | | v1.5-v1.6 | extensions/v1beta1 | | v1.7-v1.15 | apps/v1beta1 | | v1.8-v1.15 | apps/v1beta2 | | v1.9+ | apps/v1 | ## Glimpse of a ReplicationController ```yaml apiVersion: v1 kind: ReplicationController metadata: name: nginx spec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 ``` ## ReplicaSet In Action ```yaml apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: frontend # these labels can be applied automatically # from the labels in the pod template if not set # labels: # app: guestbook # tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 # selector can be applied automatically # from the labels in the pod template if not set, # but we are specifying the selector here to # demonstrate its usage. selector: matchLabels: tier: frontend matchExpressions: - {key: tier, operator: In, values: [frontend]} template: metadata: labels: app: guestbook tier: frontend spec: containers: - name: php-redis image: gcr.io/google_samples/gb-frontend:v3 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns # If your cluster config does not include a dns service, then to # instead access environment variables to find service host # info, comment out the 'value: dns' line above, and uncomment the # line below. # value: env ports: - containerPort: 80 ``` # Resource Quota Resource quotas are mechanisms created to constrain the usage of resources by users. This process operates as follows: * Resource Quotas apply to Namespaces and each Namespace can have a maximum of one `ResourceQuota` object. * Once the computational resources quota has been initiated, computational resource requests or limits must be configured when creating the container (default values can also be set using the [LimitRange](https://kubernetes.io/docs/tasks/administer-cluster/cpu-memory-limit/) function). * New resources cannot be created if the user exceeds their quota. ## Activating Resource Quota Function * Firstly, configure admittance control `--admission-control=ResourceQuota` when launching API Server. * Secondly, create a `ResourceQuota` object in the namespace. ## Types of Resource Quotas * Computational resources, including CPU and memory * CPU, limits.cpu, requests.cpu * Memory, limits.memory, requests.memory * Storage resources, including total storage and specific storage class total * Requests.storage: total storage resources, such as 500Gi * Persistentvolumeclaims: pvc number * .storageclass.storage.k8s.io/requests.storage * .storageclass.storage.k8s.io/persistentvolumeclaims * Requests.ephemeral-storage and limits.ephemeral-storage (requires v1.8+) * Object count, meaning the number of creatable objects * Pods, replicationcontrollers, configmaps, secrets * Resourcequotas, persistentvolumeclaims * Services, services.loadbalancers, services.nodeports Computational resource example: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources spec: hard: pods: "4" requests.cpu: "1" requests.memory: 1Gi limits.cpu: "2" limits.memory: 2Gi ``` Object count example: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: object-counts spec: hard: configmaps: "10" persistentvolumeclaims: "4" replicationcontrollers: "20" secrets: "10" services: "10" services.loadbalancers: "2" ``` ## LimitRange By default, no CPU or memory limits exist for any container in Kubernetes. LimitRange is used to add a resource limit to the Namespace, consisting of minimum, maximum, and default resources. For example, ```yaml apiVersion: v1 kind: LimitRange metadata: name: mylimits spec: limits: - max: cpu: "2" memory: 1Gi min: cpu: 200m memory: 6Mi type: Pod - default: cpu: 300m memory: 200Mi defaultRequest: cpu: 200m memory: 100Mi max: cpu: "2" memory: 1Gi min: cpu: 100m memory: 3Mi type: Container ``` ```bash $ kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/limits.yaml --namespace=limit-example limitrange "mylimits" created $ kubectl describe limits mylimits --namespace=limit-example Name: mylimits Namespace: limit-example Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Pod cpu 200m 2 - - - Pod memory 6Mi 1Gi - - - Container cpu 100m 2 200m 300m - Container memory 3Mi 1Gi 100Mi 200Mi - ``` ## Quota Ranges Several ranges can be specified when creating each quota | Scope | Description | | -------------- | ----------------------------------------------------------------------- | | Terminating | Pod with podSpec.ActiveDeadlineSeconds>=0 | | NotTerminating | Pod with podSpec.activeDeadlineSeconds=nil | | BestEffort | Pod where all containers' requests and limits are not set (Best-Effort) | | NotBestEffort | Opposite of BestEffort | # Secret Kubernetes provides an object called 'Secret' that deals with the challenge of configuring sensitive data like passwords, tokens, keys, etc. without exposing this valuable data within images or Pod Specs. Secrets can be utilized either as volumes or as environment variables. ## The Various Types of Secrets Secrets in Kubernetes come in three different types: * Opaque: This is a Secret that is formatted in base64 encoding and used to store sensitive elements like passwords, keys, etc. However, it only offers weak encryption security as the data can be decoded back to the original form using base64 --decode. * `kubernetes.io/dockerconfigjson`: This type of Secret is used to maintain authentication information of a private Docker registry. * `kubernetes.io/service-account-token`: This variety is referred to by service accounts. When a service account is created, Kubernetes will automatically generate a paired secret. If a Pod utilizes a service account, the matching secret will be automatically mounted to the directory: `/run/secrets/kubernetes.io/serviceaccount` within the Pod. Note: A service account enables a Pod to access the Kubernetes API. ## API Version Corresponding Chart | Kubernetes Version | Core API Version | | ------------------ | ---------------- | | v1.5+ | core/v1 | ## Opaque Secret The data for this type is a map requiring the value to be in base64 encoding format: ```bash $ echo -n "admin" | base64 YWRtaW4= $ echo -n "1f2d1e2e67df" | base64 MWYyZDFlMmU2N2Rm ``` secrets.yml ``` apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque data: password: MWYyZDFlMmU2N2Rm username: YWRtaW4= ``` Create a secret: `kubectl create -f secrets.yml`. ```bash # kubectl get secret NAME TYPE DATA AGE default-token-cty7p kubernetes.io/service-account-token 3 45d mysecret Opaque 2 7s ``` Deceased: The default-token-cty7p is the default secret created when creating a cluster, which is referenced by serviceaccount/default. If you are creating a secret from a file, you can use a simpler kubectl command, such as creating a TLS secret: ```bash $ kubectl create secret generic helloworld-tls \ --from-file=key.pem \ --from-file=cert.pem ``` ## Using Opaque Secrets Once a secret is created, there are two ways to use it: * As a Volume * As an Environment Variable ### Mounting Secrets into Volumes ``` apiVersion: v1 kind: Pod metadata: labels: name: db name: db spec: volumes: - name: secrets secret: secretName: mysecret containers: - image: gcr.io/my_project_id/pg:v1 name: db volumeMounts: - name: secrets mountPath: "/etc/secrets" readOnly: true ports: - name: cp containerPort: 5432 hostPort: 5432 ``` Here's the information within the Pod: ```bash # ls /etc/secrets password username # cat /etc/secrets/username admin # cat /etc/secrets/password 1f2d1e2e67df ``` ### Exporting Secrets to Environment Variables ``` apiVersion: extensions/v1beta1 kind: Deployment metadata: name: wordpress-deployment spec: replicas: 2 strategy: type: RollingUpdate template: metadata: labels: app: wordpress visualize: "true" spec: containers: - name: "wordpress" image: "wordpress" ports: - containerPort: 80 env: - name: WORDPRESS_DB_USER valueFrom: secretKeyRef: name: mysecret key: username - name: WORDPRESS_DB_PASSWORD valueFrom: secretKeyRef: name: mysecret key: password ``` ### Mounting A Specific Key of the Secret ``` apiVersion: v1 kind: Pod metadata: labels: name: db name: db spec: volumes: - name: secrets secret: secretName: mysecret items: - key: password mode: 511 path: tst/psd - key: username mode: 511 path: tst/usr containers: - image: nginx name: db volumeMounts: - name: secrets mountPath: "/etc/secrets" readOnly: true ports: - name: cp containerPort: 80 hostPort: 5432 ``` After creating the Pod successfully, you can see the following in the corresponding directory: ```bash # kubectl exec db ls /etc/secrets/tst psd usr ``` To be continued... # SecurityContext The primary goal of Security Context is to restrict the behavior of untrustworthy containers, shielding the system and other containers from their potential impact. There are three methods provided by Kubernetes to configure Security Context: * Container-level Security Context: Applied solely to the specified container * Pod-level Security Context: Implemented on all containers and Volume within the Pod * Pod Security Policies (PSP): Applied across all Pods and Volumes within the cluster ## The Nitty-Gritty of Container-level Security Context [Container-level Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) only applies to the assigned container, impacting the Volume not. For instance, setting a container to run in privileged mode can be done like this: ```yaml apiVersion: v1 kind: Pod metadata: name: hello-world spec: containers: - name: hello-world-container # The container definition # ... securityContext: privileged: true ``` ## Digging into Pod-level Security Context [Pod-level Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) is applied to all containers inside a Pod, and it also influences the Volume, including fsGroup and selinuxOptions. ```yaml apiVersion: v1 kind: Pod metadata: name: hello-world spec: containers: # specification of the pod's containers # ... securityContext: fsGroup: 1234 supplementalGroups: [5678] seLinuxOptions: level: "s0:c123,c456" ``` ## Understanding Pod Security Policies (PSP) Pod Security Policies (PSP) serve as cluster-level Pod security strategies, automatically setting the Security Context for Pods and Volumes within the cluster. Operating PSP requires the API Server to enable `extensions/v1beta1/podsecuritypolicy`, and to configure the `PodSecurityPolicy` admission controller. > Note: Due to a lack of flexibility, an imperfect authentication model, and cumbersome configuration updates, PodSecurityPolicy was officially [deprecated](https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) in v1.21 and will be removed from the codebase in v1.25. Users currently using PodSecurityPolicy are suggested to migrate to [Open Policy Agent](https://www.openpolicyagent.org/). ### API Version Comparison Table | Kubernetes Version | Extension Version | | ------------------ | ------------------ | | v1.5-v1.15 | extensions/v1beta1 | | v1.10+ | policy/v1beta1 | | v1.21 | deprecated | ### Supported Controls | Control | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | privileged | Operate privileged containers | | defaultAddCapabilities | Capabilities that can be added to the container | | requiredDropCapabilities | Capabilities that will be deleted from the container | | allowedCapabilities | Allowed list of Capabilities | | volumes | Control which volumes a container can use | | hostNetwork | Allows the use of the host network | | hostPorts | Allowed host port list | | hostPID | Use the host PID namespace | | hostIPC | Use the host IPC namespace | | seLinux | SELinux Context | | runAsUser | user ID | | supplementalGroups | Allowed supplementary user group | | fsGroup | volume FSGroup | | readOnlyRootFilesystem | Read-only root file system | | allowedHostPaths | Allowed list of paths for the hostPath plugin | | allowedFlexVolumes | Allowed list of flexVolume plugins | | allowPrivilegeEscalation | Allow container processes to set [`no_new_privs`](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt) | | defaultAllowPrivilegeEscalation | Default permission for privilege escalation | ### Example To restrict a container's host port range to 8000-8080, you can do this: ```yaml apiVersion: extensions/v1beta1 kind: PodSecurityPolicy metadata: name: permissive spec: seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny hostPorts: - min: 8000 max: 8080 volumes: - '*' ``` To allow only the use of lvm and cifs etc. flexVolume plugins: ```yaml apiVersion: extensions/v1beta1 kind: PodSecurityPolicy metadata: name: allow-flex-volumes spec: fsGroup: rule: RunAsAny runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - flexVolume allowedFlexVolumes: - driver: example/lvm - driver: example/cifs ``` ## A Closer Look at SELinux SELinux (Security-Enhanced Linux) is an implementation of mandatory access control. It operates under the principle of least privilege, using Linux Security Modules within the Linux kernel. Emmy award-winning SELinux was primarily developed by the United States National Security Agency, and was released to the open source developer community on December 22, 2000. The security policy for processes can be set using runcon, while the - Z parameter in ls and ps can inspect the security policy applied to files or processes. ### How to Enable or Disable SELinux? You can edit the / etc/selinux/config file: * Enable: SELINUX=enforcing * Disable: SELINUX=disabled Or use the command for temporary changes: * Enable: setenforce 1 * Disable: setenforce 0 To check SELinux status: ``` $ getenforce ``` ### Example ```yaml apiVersion: v1 kind: Pod metadata: name: hello-world spec: containers: - image: gcr.io/google_containers/busybox:1.24 name: test-container command: - sleep - "6000" volumeMounts: - mountPath: /mounted_volume name: test-volume restartPolicy: Never hostPID: false hostIPC: false securityContext: seLinuxOptions: level: "s0:c2,c3" volumes: - name: test-volume emptyDir: {} ``` This generates the following `HostConfig.Binds` for the docker container: ``` /var/lib/kubelet/pods/f734678c-95de-11e6-89b0-42010a8c0002/volumes/kubernetes.io~empty-dir/test-volume:/mounted_volume:Z /var/lib/kubelet/pods/f734678c-95de-11e6-89b0-42010a8c0002/volumes/kubernetes.io~secret/default-token-88xxa:/var/run/secrets/kubernetes.io/serviceaccount:ro,Z /var/lib/kubelet/pods/f734678c-95de-11e6-89b0-42010a8c0002/etc-hosts:/etc/hosts ``` The appropriate Volume also has SELinux properly set: ``` $ ls -Z /var/lib/kubelet/pods/f734678c-95de-11e6-89b0-42010a8c0002/volumes drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c2,c3 kubernetes.io~empty-dir drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c2,c3 kubernetes.io~secret ``` ## Additional Reading * [Kubernetes Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) # Service From the outset, Kubernetes incorporated mechanisms for container service discovery and load balancing, establishing the Service resource and, in conjunction with kube-proxy and cloud provider, adapted it to different application scenarios. With the explosive growth of Kubernetes users and increasingly diverse user scenarios, some new load balancing mechanisms have emerged. Currently, the load balancing mechanisms in Kubernetes can be roughly classified into the following categories, each with its specific application scenario: * Service: Directly uses Service to provide internal cluster load balancing, and leverages the LB provided by the cloud provider for external access. * Ingress Controller: Still uses Service for internal cluster load balancing, but external access is enabled via customized Ingress Controller. * Service Load Balancer: Runs the load balancer directly in the container, implementing Bare Metal's Service Load Balancer. * Custom Load Balancer: Customized load balancing replaces kube-proxy and is usually used when deploying Kubernetes physically, facilitating the connection to existing external services in the company. ## Service ![](/files/cygrIlrq2QSklX2GKcnt) A Service is an abstraction of a group of Pods providing the same functionality, providing them with a unified access point. With Service, applications can easily implement service discovery and load balancing, as well as zero-downtime upgrades. Service selects service backends through labels, usually working with Replication Controller or Deployment to ensure the normal operation of backend containers. The Pod IPs and port lists matching these labels form endpoints, with kube-proxy responsible for load balancing the service IP to these endpoints. There are four types of Service: * ClusterIP: Default type, automatically assigns a virtual IP that can only be accessed internally by the cluster. * NodePort: Based on ClusterIP, binds a port for the Service on each machine, allowing access to the service through `:NodePort`. If kube-proxy has set `--nodeport-addresses=10.240.0.0/16` (supported by v1.10), then this NodePort is only valid for IPs set within this range. * LoadBalancer: Based on NodePort, creates an external load balancer with the help of the cloud provider, redirecting requests to `:NodePort`. * ExternalName: Redirects the service to a specified domain (set through `spec.externlName`) via DNS CNAME record. Requires kube-dns version 1.7 or later. Additionally, existing services can be added to the Kubernetes cluster in Service format. When creating a Service, do not specify a Label selector. Instead, manually add an endpoint after the Service is created. ### Service Definition A Service is defined through yaml or json, such as the below example that defines a service called nginx, which forwards the service's port 80 to port 80 of the Pod labeled `run=nginx` in the default namespace. ```yaml apiVersion: v1 kind: Service metadata: labels: run: nginx name: nginx namespace: default spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: run: nginx sessionAffinity: None type: ClusterIP ``` ```bash # service automatically allocated Cluster IP 10.0.0.108 $ kubectl get service nginx NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx 10.0.0.108 80/TCP 18m # automatically created endpoint $ kubectl get endpoints nginx NAME ENDPOINTS AGE nginx 172.17.0.5:80 18m # Service automatically associates endpoint $ kubectl describe service nginx Name: nginx Namespace: default Labels: run=nginx Annotations: Selector: run=nginx Type: ClusterIP IP: 10.0.0.108 Port: 80/TCP Endpoints: 172.17.0.5:80 Session Affinity: None Events: ``` When the service requires multiple ports, each port must be given a name. ```yaml kind: Service apiVersion: v1 metadata: name: my-service spec: selector: app: MyApp ports: - name: http protocol: TCP port: 80 targetPort: 9376 - name: https protocol: TCP port: 443 targetPort: 9377 ``` ### Protocol Service, Endpoints, and Pod support three types of protocols: * TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream transport layer communication protocol. * UDP (User Datagram Protocol) is a connectionless transport layer protocol used for unreliable information delivery services. * SCTP (Stream Control Transmission Protocol) is used to transmit SCN (Signaling Communication Network) narrow band signaling messages over IP networks. ### API Version Comparison Table | Kubernetes Version | Core API Version | | ------------------ | ---------------- | | v1.5+ | core/v1 | ### Services Without Specified Selectors When creating a Service, you can also choose not to specify Selectors, used to forward the service to services outside the Kubernetes cluster (instead of to Pods). At present, two methods are supported: (1) Custom endpoints, that is, create a service and endpoint of the same name and set the IP and port of the external service in the endpoint. ```yaml kind: Service apiVersion: v1 metadata: name: my-service spec: ports: - protocol: TCP port: 80 targetPort: 9376 --- kind: Endpoints apiVersion: v1 metadata: name: my-service subsets: - addresses: - ip: 1.2.3.4 ports: - port: 9376 ``` (2) Forwarding via DNS, specifying externalName in the service definition. In this case, the DNS service will create a CNAME record for `..svc.cluster.local`, with its value set to `my.database.example.com`. Moreover, the service will not be automatically assigned a Cluster IP and needs to be accessed through the service's DNS. ```yaml kind: Service apiVersion: v1 metadata: name: my-service namespace: default spec: type: ExternalName externalName: my.database.example.com ``` Note: The IP address of the Endpoints cannot be 127.0.0.0/8, 169.254.0.0/16 or 224.0.0.0/24, nor can it be the clusterIP of other services in Kubernetes. ### Headless Service A Headless Service is one that does not require a Cluster IP. This is specified when creating a service by setting `spec.clusterIP=None`. This includes two types: * No Selectors specified but an externalName is set (See above 2), handled by the CNAME record. * Selectors specified, with a DNS A record setting the backend endpoint list. ```yaml apiVersion: v1 kind: Service metadata: labels: app: nginx name: nginx spec: clusterIP: None ports: - name: tcp-80-80-3b6tl port: 80 protocol: TCP targetPort: 80 selector: app: nginx sessionAffinity: None type: ClusterIP --- apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: app: nginx name: nginx namespace: default spec: replicas: 2 revisionHistoryLimit: 5 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx:latest imagePullPolicy: Always name: nginx resources: limits: memory: 128Mi requests: cpu: 200m memory: 128Mi dnsPolicy: ClusterFirst restartPolicy: Always ``` ```bash # Query the created nginx service $ kubectl get service --all-namespaces=true NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE default nginx None 80/TCP 5m kube-system kube-dns 172.26.255.70 53/UDP,53/TCP 1d $ kubectl get pod NAME READY STATUS RESTARTS AGE IP NODE nginx-2204978904-6o5dg 1/1 Running 0 14s 172.26.2.5 10.0.0.2 nginx-2204978904-qyilx 1/1 Running 0 14s 172.26.1.5 10.0.0.8 $ dig @172.26.255.70 nginx.default.svc.cluster.local ;; ANSWER SECTION: nginx.default.svc.cluster.local. 30 IN A 172.26.1.5 nginx.default.svc.cluster.local. 30 IN A 172.26.2.5 ``` Note: Some of the information queried in the dig command is omitted. ## Preserving Source IP Different types of Service handle the source IP differently: * ClusterIP Service: Using iptables mode, the source IP within the cluster is retained (no SNAT). If the client and server pod are on the same Node, the source IP is the IP address of the client pod; if on different Nodes, the source IP depends on how the network plugin handles it. For instance, when using flannel, the source IP is the node flannel IP address. * NodePort Service: By default, the source IP would undergo SNAT, and the server pod would see the source IP as Node IP. To avoid this, the service can be set to `spec.ExternalTrafficPolicy=Local` (for versions 1.6-1.7, set Annotation `service.beta.kubernetes.io/external-traffic=OnlyLocal`), allowing the service to proxy requests for local endpoints only (if there are no local endpoints, the packets are directly dropped), thus retaining the source IP. * LoadBalancer Service: By default, the source IP would undergo SNAT, and the server pod sees the source IP as Node IP. After setting `service.spec.ExternalTrafficPolicy=Local`, the Node without local endpoints will be automatically removed from the cloud platform load balancer, thus retaining the source IP. ## Internal Network Policy By default, Kubernetes considers all Endpoints IP in the cluster to be Service backends. By setting `.spec.internalTrafficPolicy=Local`, kube-proxy will only load balance for local Endpoints on the Node. ```yaml apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: MyApp ports: - protocol: TCP port: 80 targetPort: 9376 internalTrafficPolicy: Local ``` Note, with the internal network policy enabled, even if other Nodes have functioning Endpoints, if there are no Pods running locally on the Node, the Service will be inaccessible. ## How it Works kube-proxy is responsible for load balancing the service to the backend Pod, as shown in the diagram: ![service-flow](/files/J31cr5x3APJzkjWAzWiE) ## Ingress Although Service solves the problems of service discovery and load balancing, it still has some limitations in use, for example: * It only supports layer 4 load balancing and lacks layer 7 functionality. * For external access, NodePort type requires additional load balancing at the external, whereas LoadBalancer requires Kubernetes to run on a supported cloud provider. Ingress is a newly introduced resource designed to address these limitations, mainly used to expose services outside the cluster and allows customizing service access policies. For instance, if you want to access different services through different subdomains via a load balancer: ``` foo.bar.com --| |-> foo.bar.com s1:80 | 178.91.123.132 | bar.foo.com --| |-> bar.foo.com s2:80 ``` You can define Ingress like this: ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test spec: rules: - host: foo.bar.com http: paths: - backend: serviceName: s1 servicePort: 80 - host: bar.foo.com http: paths: - backend: serviceName: s2 servicePort: 80 ``` Note the Ingress itself does not automatically create a load balancer, an ingress controller needs to be operating within the cluster managing the load balancer according to the Ingress definition. The community currently provides reference implementations for nginx and gce. Traefik offers an easy-to-use Ingress Controller, the use of which is explained at . For a more in-depth introduction to Ingress and Ingress Controller, see [ingress](/en/concepts/objects/ingress). ## Service Load Balancer Before the introduction of Ingress, [Service Load Balancer](https://github.com/kubernetes/contrib/tree/master/service-loadbalancer) was the recommended method to address the limitations of Service. Service Load Balancer runs haproxy in containers and monitors changes in service and endpoints, providing layer 4 and layer 7 load balancing services through container IP. The community provided Service Load Balancer supports four load balancing protocols: TCP, HTTP, HTTPS and SSL TERMINATION, and also supports ACL access control. > Note: Service Load Balancer is no longer recommended for use. The use of [Ingress Controller](/en/concepts/objects/ingress) is recommended instead. ## Custom Load Balancer Despite Kubernetes offering a variety of load balancing mechanisms, in practice, some complex scenarios are unsupported, such as: * Connecting to existing load balancing equipment * In multi-tenant network situations, the container network and host network are isolated, rendering `kube-proxy` dysfunctional. At these times, components can be customized and used to replace kube-proxy for load balancing. The basic idea is to monitor changes in service and endpoints in Kubernetes and configure the load balancer according to these changes, seen in weave flux, nginx plus, kube2haproxy, and more. ## External Cluster Access to Service A Service's ClusterIP is an internal virtual IP address in Kubernetes that cannot be accessed directly from outside. But what if it's necessary to access these services from the outside? There are several ways: * Use NodePort service to bind a port on each machine, allowing access to the service through `:NodePort`. * Use LoadBalancer service to create an external load balancer with the help of Cloud Provider, redirecting requests to `:NodePort`. This method is only applicable to Kubernetes clusters running on the cloud platform. For clusters deployed on physical machines, [MetalLB](https://github.com/google/metallb) can implement similar functionality. * Create L7 load balancing atop Service through Ingress Controller and open it to the public. * Use [ECMP](https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing) to route the Service ClusterIP network segment to each Node, allowing direct access through ClusterIP and even direct use of kube-dns outside the cluster. This method is applied in situations of depolyment on physical machines. ## References * * * * * \[ # ServiceAccount Think of service accounts as a way for the processes within your Pod to smoothly interact with the Kubernetes API and other external services. You see, they're distinct from user accounts: * User accounts are designed for humans. On the other hand, service accounts are custom-made for processes within a Pod that want to interact with the Kubernetes API. * User accounts transcend namespaces, while service accounts are restrained by their respective namespaces. * Each namespace automatically conjures a default service account. * The token controller keeps an eye out for any freshly spawned service accounts, creating a corresponding [secret](/en/concepts/objects/secret) for each one. * With the ServiceAccount Admission Controller activated * Every newly created Pod is automatically assigned a `spec.serviceAccountName` set to default (unless another ServiceAccount is specified). * It double-checks if the service account \[20] referenced by the Pod exists; if it doesn't, the creation process is denied. * If a Pod hasn't specified ImagePullSecrets, the service account's ImagePullSecrets are added to the Pod. * Each container brought to life will have a token and ‘ca.crt’ from its service account mounted on `/var/run/secrets/kubernetes.io/serviceaccount/`. > Heads up: Starting with v1.24.0, ServiceAccount won’t spawn Secrets automatically. If you’re keen on retaining this feature, configure your kube-controller-manager to `LegacyServiceAccountTokenNoAutoGeneration=false`. ```bash $ kubectl exec nginx-3137573019-md1u2 ls /var/run/secrets/kubernetes.io/serviceaccount ca.crt namespace token ``` > Pro Tip: Head to for an in-depth look at your token (like PAYLOAD, SIGNATURE, etc.). ## To create a Service Account: ```bash $ kubectl create serviceaccount jenkins serviceaccount "jenkins" created $ kubectl get serviceaccounts jenkins -o yaml apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: 2017-05-27T14:32:25Z name: jenkins namespace: default resourceVersion: "45559" selfLink: /api/v1/namespaces/default/serviceaccounts/jenkins uid: 4d66eb4c-42e9-11e7-9860-ee7d8982865f secrets: - name: jenkins-token-l9v7v ``` The corresponding secret gets generated automatically: ```bash kubectl get secret jenkins-token-l9v7v -o yaml apiVersion: v1 data: ca.crt: (APISERVER CA BASE64 ENCODED) namespace: ZGVmYXVsdA== token: (BEARER TOKEN BASE64 ENCODED) kind: Secret metadata: annotations: kubernetes.io/service-account.name: jenkins kubernetes.io/service-account.uid: 4d66eb4c-42e9-11e7-9860-ee7d8982865f creationTimestamp: 2017-05-27T14:32:25Z name: jenkins-token-l9v7v namespace: default resourceVersion: "45558" selfLink: /api/v1/namespaces/default/secrets/jenkins-token-l9v7v uid: 4d697992-42e9-11e7-9860-ee7d8982865f type: kubernetes.io/service-account-token ``` ## To add ImagePullSecrets: ```yaml apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: 2015-08-07T22:02:39Z name: default namespace: default selfLink: /api/v1/namespaces/default/serviceaccounts/default uid: 052fb0f4-3d50-11e5-b066-42010af0d7b6 secrets: - name: default-token-uudge imagePullSecrets: - name: myregistrykey ``` ## Giving Authorization While service accounts smoothly enable service authentications, they remain apathetic towards authorization matters. To make them more useful, pair them with [RBAC](https://kubernetes.io/docs/admin/authorization/#a-quick-note-on-service-accounts) for granting Service Account access: * Set up both `--authorization-mode=RBAC` and `--runtime-config=rbac.authorization.k8s.io/v1alpha1` * Enable `--authorization-rbac-super-user=admin` * Define your Role, ClusterRole, RoleBinding, or ClusterRoleBinding Here's an example: ```yaml # This role allows to read pods in the "default" namespace kind: Role apiVersion: rbac.authorization.k8s.io/v1alpha1 metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # The empty API group "" specifies the core API Group. resources: ["pods"] verbs: ["get", "watch", "list"] nonResourceURLs: [] --- # This role binding allows "default" to read pods in the "default" namespace kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1alpha1 metadata: name: read-pods namespace: default subjects: - kind: ServiceAccount # Can be "User", "Group", or "ServiceAccount" name: default roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` # StatefulSet StatefulSet is specifically designed to address issues related to stateful services (while Deployments and ReplicaSets cater to stateless services), making it an essential tool when dealing with * Stable, persistent storage allowing Pods to access the same persistent data even after rescheduling, using Persistent Volumes Claims (PVC) to implement. * Stable networking IDs, meaning that the PodName and HostName remain unchanged even after a Pod is rescheduled, achieved through Headless Service (a Service without a Cluster IP). * Sequential Deployment and Scaling, indicating an orderly deployed or scaled Pod, guided by a predefined sequence (from 0 to N-1, where all preceding Pods have to be in Running and Ready status before the next Pod is run), implemented using init containers. * An ordered scale-down and deletion (from N-1 to 0). From these application scenarios, we can deduce that a StatefulSet consists of: * A Headless Service defining a networking ID (DNS domain). * volumeClaimTemplates used for creating PersistentVolumes. * The StatefulSet explicitly defining the application. Each Pod within a StatefulSet follows this DNS format: `statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local`, where: * `serviceName` refers to the name of the Headless Service. * `0..N-1` is the sequence number of the Pod, starting from 0 and continuing to N-1. * `statefulSetName` is the name of the StatefulSet. * `namespace` indicates the namespace where the service resides. The Headless Service and StatefulSet need to be in the same namespace. * `.cluster.local` is the Cluster Domain. ## API version comparison table | Kubernetes Version | Deployment Version | | ------------------ | ------------------ | | v1.5-v1.6 | extensions/v1beta1 | | v1.7-v1.15 | apps/v1beta1 | | v1.8-v1.15 | apps/v1beta2 | | v1.9+ | apps/v1 | ## Simple Example Let's consider a simple example using an nginx service [web.yaml](https://github.com/feiskyer/kubernetes-handbook/tree/549e0e3c9ba0175e64b2d4719b5a46e9016d532b/concepts/web.txt): \[Code snippet omitted for brevity] You can perform other operations as well: ```bash # Scaling up $ kubectl scale statefulset web --replicas=5 # Scaling down $ kubectl patch statefulset web -p '{"spec":{"replicas":3}}' # Image updating (currently, direct image updates are unsupported, patchwork is used to achieve it indirectly) $ kubectl patch statefulset web --type='json' -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"gcr.io/google_containers/nginx-slim:0.7"}]' # Deleting a StatefulSet and Headless Service $ kubectl delete statefulset web $ kubectl delete service nginx # After the StatefulSet is deleted, the PVC will remain. If the data is no longer needed, it should be removed as well $ kubectl delete pvc www-web-0 www-web-1 ``` ## Updating a StatefulSet From v1.7 and onwards, Kubernetes supports automatic updating of StatefulSets via the `spec.updateStrategy` setting. Currently, two strategies are supported: * OnDelete: When `.spec.template` is updated, old Pods aren't deleted immediately. Instead, users need to manually delete these old Pods, after which new Pods are automatically created. This is the default update strategy and is compatible with the behavior in versions v1.6 and earlier. * RollingUpdate: When `.spec.template` is updated, old Pods are automatically deleted, and new Pods are created simultaneously. In the update process, these Pods go through deletion, creation, and stabilization into Ready status one at a time in reverse order, before moving on to the next Pod update. ### Partitions RollingUpdate also supports Partitions, which can be set using `.spec.updateStrategy.rollingUpdate.partition`. Once partition is set, only Pods with a sequence number equal to or greater than the partition will be rolled out for `.spec.template` updates, while the remaining Pods are left unchanged (even when deleted, they will be recreated using the previous version). \[Example and code snippet omitted for brevity] ## Pod Management Policies From v1.7 and onwards, you can set the Pod management policy using `.spec.podManagementPolicy`, with two options available: * OrderedReady: This default policy sequentially creates each Pod and waits for it to be Ready before creating the next one. * Parallel: Pods are created or deleted simultaneously without waiting for other Pods to reach Ready status before launching all Pods. ### Parallel Example \[Code snippet omitted for brevity] You can observe that all Pods are created simultaneously. \[Code snippet omitted for brevity] ## Zookeeper Another example showing the StatefulSet's powerful functions is [zookeeper.yaml](https://github.com/feiskyer/kubernetes-handbook/tree/549e0e3c9ba0175e64b2d4719b5a46e9016d532b/concepts/zookeeper.txt). \[Code snippet omitted for brevity] ```bash kubectl create -f zookeeper.yaml ``` Detailed usage instructions can be found at the [zookeeper stateful application](https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/) tutorial. ## Caveats for StatefulSets 1. Recommended for use in Kubernetes v1.9 or later. 2. All Pod Volumes must either use PersistentVolumes or be pre-created by an administrator. 3. To ensure data safety, deleting a StatefulSet does not delete the Volumes. 4. A StatefulSet requires a Headless Service to define the DNS domain. This should be created before the StatefulSet. # Volume Just as we know that our apps and data aren't destined to last forever, in the Docker universe, the lifecycle of container data is fundamentally ephemeral. Once a container bites the dust, so does its data. Recognizing the importance of data permanence, Docker created a clever system called 'Volumes' to persist container data. Similarly, Kubernetes has embraced and even improved upon Docker's concept, offering its own robust version of volumes. Kubernetes volumes, accompanied by a plethora of plugins, help tremendously in ensuring data permanence and sharing data between containers. However, a critical distinction lies between Docker and Kubernetes volumes. Unlike Docker, Kubernetes volumes are intrinsically tied to the lifecycle of a pod. * Regardless of whether a container is brought back from the verge of oblivion by Kubelet, the volume's data will always remain unharmed. * It's only when a Pod is deleted that the volume is cleaned up. Whether the data is also deleted depends on the type of volume being used. For an emptyDir volume, the data gets lost, but for a Persistent Volume (PV), it's preserved. ## Different Flavors of Kubernetes Volumes As of now, Kubernetes offers the following types of volumes: * emptyDir * hostPath * gcePersistentDisk * awsElasticBlockStore * nfs * iscsi * flocker * glusterfs * rbd * cephfs * gitRepo * secret * persistentVolumeClaim * downwardAPI * azureFileVolume * azureDisk * vsphereVolume * Quobyte * PortworxVolume * ScaleIO * FlexVolume * StorageOS * local Remember, not all volume types are 'persistent'. For instance, emptyDir, secret, gitRepo volumes disppear along with the pod. ## API Version Compatibility Chart | Kubernetes Version | Core API Version | | ------------------ | ---------------- | | v1.5+ | core/v1 | ## Getting into the Nuances of Different Volume Types ### emptyDir When a Pod is assigned an emptyDir type volume, the emptyDir is created as soon as the Pod is scheduled on the node. As long as the Pod runs on the node, the emptyDir remains. However, if the Pod is removed from the node, the emptyDir is also removed, causing the data to be lost permanently. ```yaml apiVersion: v1 kind: Pod metadata: name: test-pd spec: containers: - image: gcr.io/google_containers/test-webserver name: test-container volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {} ``` ### hostPath The hostPath volume allows mounting of the node's filesystem within the pod – perfectly suited when a pod needs to use files on the node. ```yaml apiVersion: v1 kind: Pod metadata: name: test-pd spec: containers: - image: gcr.io/google_containers/test-webserver name: test-container volumeMounts: - mountPath: /test-pd name: test-volume volumes: - name: test-volume hostPath: path: /data ``` ### NFS Short for Network File System, NFS provides a way to easily mount its system to a Pod in Kubernetes. Notably, NFS ensures permanent data storage and supports concurrent write operations. ```yaml volumes: - name: nfs nfs: # FIXME: use the right hostname server: 10.254.234.223 path: "/" ``` \[...] For the details and examples of all the other volume types, please visit the Kubernetes examples page at \[]. ## The Art of Volume Snapshotting Although in a pre-alpha state, the concept of volume snapshots was introduced to Kubernetes in version 1.8. However, its implementation is not in the core Kubernetes but rests in [kubernetes-incubator/external-storage](https://github.com/kubernetes-incubator/external-storage/tree/master/snapshot). > Stay tuned for a detailed discussion and examples of volume snapshotting on our upcoming posts! ## Volume Mount Propagation Introduced with version v1.9, Mount Propagation is quickly scaling its way through the beta version in Kubernetes v1.10. Mount Propagation is used to handle the mounting issues of the same volume across different containers or even Pods. By adjusting the `Container.volumeMounts.mountPropagation` setting, you can assign different types of propagation to the volume. It provides three options: * None: private mount * HostToContainer: Where new mounts made inside the host directory are visible inside the container, equivalent to Linux kernel's rslave. * Bidirectional: Where new mounts made inside the host or container's directory are visible in the opposite party, equivalent to Linux kernel's rshared. The privileged containers can only use the bi-directional type. Note: * The enabling of the Mount Propagation feature is required first. * If not set, the default is 'private' for v1.9 and v1.10, whereas for v1.11, it defaults to 'HostToContainer'. * Docker's systemd configuration file must be set to `MountFlags=shared`. ## Byte-Sized Guides to Other Volume Types * [iSCSI Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/iscsi) * [cephfs Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/cephfs) * [Flocker Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/flocker) * [GlusterFS Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/glusterfs) * [RBD Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/rbd) * [Secret Volume Example](/en/concepts/objects/secret) * [downwardAPI Volume Example](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) * [AzureFile Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_file) * [AzureDisk Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_disk) * [Quobyte Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/quobyte) * [Portworx Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/portworx) * [ScaleIO Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/scaleio) * [StorageOS Volume Example](https://github.com/kubernetes/examples/tree/master/staging/volumes/storageos) # Setup Guidance This chapter provides a handy guide for deploying Kubernetes clusters, installing the kubectl client, and recommended configurations. The [Kubernetes-The-Hard-Way](https://github.com/feiskyer/kubernetes-handbook/blob/en/setup/k8s-hard-way/README.md) guide provides detailed steps to deploy a highly available Kubernetes cluster in Ubuntu virtual machines in Google Cloud Engine (GCE). These steps are also suitable for other operating systems like CentOS, and other public cloud platforms such as AWS and Azure. When deploying a cluster within China, it's common to encounter difficulties in pulling images or experiencing slow pull speeds. A solution to this problem is to use domestic images. You can refer to the [domestic image list](/en/appendix/mirrors) for options. Generally speaking, after the deployment, you need to run a series of tests to verify the deployment's success. [Sonobuoy](https://github.com/heptio/sonobuoy) can simplify this validation process by running a series of tests to ensure your cluster is functioning correctly. Its usage methods are: * Online use via the [Sonobuoy Scanner tool](https://scanner.heptio.com/) (which requires the cluster to be publicly accessible) * Or use it as a command line tool. ```bash # Install $ go get -u -v github.com/heptio/sonobuoy # Run $ sonobuoy run $ sonobuoy status $ sonobuoy logs $ sonobuoy retrieve . # Cleanup $ sonobuoy delete ``` ## Version Dependencies | Dependencies | v1.13 | v1.12 | | ---------------------------------------------- | ----- | ----- | | ... | | | | *Remaining table contents omitted for brevity* | | | ## Deployment Methods * [1. Single Machine Deployment](/en/setup/single) * [2. Cluster Deployment](/en/setup/cluster) * [kubeadm](/en/setup/cluster/kubeadm) * [kops](/en/setup/cluster/kops) * [Kubespray](/en/setup/cluster/kubespray) * [Azure](/en/setup/cluster/azure) * [Windows](/en/setup/cluster/windows) * [LinuxKit](/en/setup/cluster/k8s-linuxkit) * [Frakti](/en/extension/cri/frakti) * [kubeasz](https://github.com/gjmzj/kubeasz) * [3. kubectl Client](/en/setup/kubectl) * [4. Additional Components](/en/setup/addon-list) * [Addon-manager](/en/setup/addon-list/addon-manager) * [DNS](/en/setup/index) * [Dashboard](/en/setup/addon-list/dashboard) * [Monitoring](/en/setup/addon-list/monitor) * [Logging](/en/setup/addon-list/logging) * [Metrics](/en/setup/addon-list/metrics) * [GPU](/en/setup/index) * [Cluster Autoscaler](/en/setup/addon-list/cluster-autoscaler) * [ip-masq-agent](/en/setup/addon-list/ip-masq-agent) * [Heapster (retired)](https://github.com/kubernetes-retired/heapster) * [5. Recommended Configurations](/en/setup/kubernetes-configuration-best-practice) * [6. Version Support](/en/setup/upgrade) # kubectl Install Ready to get kubectl onto your machine? You've come to the right place. ## How to Install ### For OSX Users Option one: You can get kubectl on to your OSX machine with one command using Homebrew: ```bash brew install kubectl ``` Option two: Feel like using `curl` instead? No problem: ```bash curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl ``` ### For Linux Users Just input the following command: ```bash curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl ``` ### For Windows Users Command one, for good measure: ```bash curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/windows/amd64/kubectl.exe ``` Or, if you prefer, you can use Chocolatey to install: ```bash choco install kubernetes-cli ``` ## How to Use For a deep dive into using kubectl, check out our [kubectl guide](/en/concepts/components/kubectl). ## kubectl Plugins Ever heard of krew? It's something you can use to manage kubectl plugins. [krew](https://github.com/kubernetes-sigs/krew) is a handy tool that lets you manage kubectl plugins, sort of like apt or yum. It allows you to search for, install, and manage kubectl plugins. ### Installing krew Use the following command: ```bash ( set -x; cd "$(mktemp -d)" && curl -fsSLO "https://storage.googleapis.com/krew/v0.2.1/krew.{tar.gz,yaml}" && tar zxvf krew.tar.gz && ./krew-"$(uname | tr '[:upper:]' '[:lower:]')_amd64" install \ --manifest=krew.yaml --archive=krew.tar.gz ) ``` Once you've got it installed, add the krew binary to your PATH: ```bash export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" ``` Then, you should be able to verify your install with a kubectl command: ```bash $ kubectl plugin list The following kubectl-compatible plugins are available: /home//.krew/bin/kubectl-krew ``` ### How to Use krew Before your first use, update the plugin index using the following command: ```bash kubectl krew update ``` Here's how you can use krew: ```bash kubectl krew search # show all plugins kubectl krew install ssh-jump # install a plugin named "ssh-jump" kubectl ssh-jump # use the plugin kubectl krew upgrade # upgrade installed plugins kubectl krew remove ssh-jump # uninstall a plugin ``` After you install your plugins, you'll see a list of external tools that the plugin depends on. You'll need to install these manually. ```bash Installing plugin: ssh-jump CAVEATS: \ | This plugin needs the following programs: | * ssh(1) | * ssh-agent(1) | | Please follow the documentation: https://github.com/yokawasa/kubectl-plugin-ssh-jump / Installed plugin: ssh-jump ``` Finally, you can use the plugin by typing `kubectl `: ```bash kubectl ssh-jump -u -i ~/.ssh/id_rsa -p ~/.ssh/id_rsa.pub ``` ### How to Upgrade krew Here's your command: ```bash kubectl krew upgrade ``` ## Further Reading * [Here's krew's GitHub page](https://github.com/kubernetes-sigs/krew) # Single Machine ## minikube The simplest way to create a Kubernetes cluster (single machine version) is by using [minikube](https://github.com/kubernetes/minikube). If you are operating in China's network environment, you can also consider utilizing AllInOne deployment from [kubeasz](https://github.com/gjmzj/kubeasz). Begin by downloading kubectl: ```bash curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl chmod +x kubectl ``` Next, install minikube (for MacOS as an example): ```bash # install minikube $ brew cask install minikube $ curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit $ sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/ ``` For Windows users: ```bash choco install minikube choco install kubernetes-cli ``` Finally, launch minikube: ```bash # start minikube. # HTTP proxy needed in China $ minikube start --docker-env HTTP_PROXY=http://proxy-ip:port --docker-env HTTPS_PROXY=http://proxy-ip:port --vm-driver=hyperkit ``` ### Utilizing calico Minikube supports configuration using the CNI (Container Network Interface) plugins, which enables an easy access to a variety of community-provided network plugins, like calico which also supports Network Policy. Start minikube with the command below: ```bash minikube start --docker-env HTTP_PROXY=http://proxy-ip:port \ --docker-env HTTPS_PROXY=http://proxy-ip:port \ --network-plugin=cni \ --host-only-cidr 172.17.17.1/24 \ --extra-config=kubelet.ClusterCIDR=192.168.0.0/16 \ --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \ --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 ``` Then, install the calico network plugin: ```bash kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml curl -O -L https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml sed -i -e '/nodeSelector/d' calico.yaml sed -i -e '/node-role.kubernetes.io\/master:""/d' calico.yaml sed -i -e 's/10\.96\.232/10.0.0/' calico.yaml kubectl apply -f calico.yaml ``` ## Developer Mode ### local-up-cluster.sh Minikube/localkube only offers the formal release versions. However, if you're looking to deploy a master or developer version, you can start a local cluster using `hack/local-up-cluster.sh`: ```bash cd $GOPATH/src/k8s.io/kubernetes hack/local-up-cluster.sh ``` Then, open another terminal to configure kubectl: ```bash cd $GOPATH/src/k8s.io/kubernetes cluster/kubectl.sh get pods cluster/kubectl.sh get services cluster/kubectl.sh get replicationcontrollers cluster/kubectl.sh run my-nginx --image=nginx --port=80 ``` ### Kind Use [kind](https://github.com/kubernetes-sigs/kind) to operate a Kubernetes cluster via Docker containers: ```bash $ go get sigs.k8s.io/kind # ensure that Kubernetes is cloned in $(go env GOPATH)/src/k8s.io/kubernetes # build a node image $ kind build node-image # create a cluster with kind build node-image $ kind create cluster --image kindest/node:latest ``` ## Reference Documents * [Running Kubernetes Locally via Minikube](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) * # Feature Gates Feature Gates are configurations used in Kubernetes to enable experimental features. Different components such as kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy can have their functionality enabled through the `--feature-gates` option. | Feature | Default | Stage | Since | Until | | ----------------------------------------- | ------- | ---------- | ----- | ----- | | `Accelerators` | `false` | Alpha | 1.6 | 1.10 | | `AdvancedAuditing` | `false` | Alpha | 1.7 | 1.7 | | `AdvancedAuditing` | `true` | Beta | 1.8 | 1.11 | | `AdvancedAuditing` | `true` | GA | 1.12 | - | | `AffinityInAnnotations` | `false` | Alpha | 1.6 | 1.7 | | `AllowExtTrafficLocalEndpoints` | `false` | Beta | 1.4 | 1.6 | | `AllowExtTrafficLocalEndpoints` | `true` | GA | 1.7 | - | | `APIListChunking` | `false` | Alpha | 1.8 | 1.8 | | `APIListChunking` | `true` | Beta | 1.9 | | | `APIResponseCompression` | `false` | Alpha | 1.7 | | | `AppArmor` | `true` | Beta | 1.4 | | | `AttachVolumeLimit` | `false` | Alpha | 1.11 | | | `BlockVolume` | `false` | Alpha | 1.9 | | | `CPUManager` | `false` | Alpha | 1.8 | 1.9 | | `CPUManager` | `true` | Beta | 1.10 | | | `CRIContainerLogRotation` | `false` | Alpha | 1.10 | 1.10 | | `CRIContainerLogRotation` | `true` | Beta | 1.11 | | | `CSIBlockVolume` | `false` | Alpha | 1.11 | 1.11 | | `CSIPersistentVolume` | `false` | Alpha | 1.9 | 1.9 | | `CSIPersistentVolume` | `true` | Beta | 1.10 | | | `CustomPodDNS` | `false` | Alpha | 1.9 | 1.9 | | `CustomPodDNS` | `true` | Beta | 1.10 | | | `CustomResourceSubresources` | `false` | Alpha | 1.10 | | | `CustomResourceValidation` | `false` | Alpha | 1.8 | 1.8 | | `CustomResourceValidation` | `true` | Beta | 1.9 | | | `DebugContainers` | `false` | Alpha | 1.10 | | | `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 | | `DevicePlugins` | `true` | Beta | 1.10 | | | `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 | | `DynamicKubeletConfig` | `true` | Beta | 1.11 | | | `DynamicProvisioningScheduling` | `false` | Alpha | 1.11 | 1.11 | | `DynamicVolumeProvisioning` | `true` | Alpha | 1.3 | 1.7 | | `DynamicVolumeProvisioning` | `true` | GA | 1.8 | | | `EnableEquivalenceClassCache` | `false` | Alpha | 1.8 | | | `ExpandInUsePersistentVolumes` | `false` | Alpha | 1.11 | | | `ExpandPersistentVolumes` | `false` | Alpha | 1.8 | 1.10 | | `ExpandPersistentVolumes` | `true` | Beta | 1.11 | | | `ExperimentalCriticalPodAnnotation` | `false` | Alpha | 1.5 | | | `ExperimentalHostUserNamespaceDefaulting` | `false` | Beta | 1.5 | | | `GCERegionalPersistentDisk` | `true` | Beta | 1.10 | | | `HugePages` | `false` | Alpha | 1.8 | 1.9 | | `HugePages` | `true` | Beta | 1.10 | | | `HyperVContainer` | `false` | Alpha | 1.10 | | | `Initializers` | `false` | Alpha | 1.7 | | | `KubeletConfigFile` | `false` | Alpha | 1.8 | 1.9 | | `KubeletPluginsWatcher` | `false` | Alpha | 1.11 | 1.11 | | `KubeletPluginsWatcher` | `true` | Beta | 1.12 | | | `LocalStorageCapacityIsolation` | `false` | Alpha | 1.7 | 1.9 | | `LocalStorageCapacityIsolation` | `true` | Beta | 1.10 | | | `MountContainers` | `false` | Alpha | 1.9 | | | `MountPropagation` | `false` | Alpha | 1.8 | 1.9 | | `MountPropagation` | `true` | Beta | 1.10 | 1.11 | | `MountPropagation` | `true` | GA | 1.12 | | | `PersistentLocalVolumes` | `false` | Alpha | 1.7 | 1.9 | | `PersistentLocalVolumes` | `true` | Beta | 1.10 | | | `PodPriority` | `false` | Alpha | 1.8 | | | `PodReadinessGates` | `false` | Alpha | 1.11 | | | `PodReadinessGates` | `true` | Beta | 1.12 | | | `PodShareProcessNamespace` | `false` | Alpha | 1.10 | | | `PodShareProcessNamespace` | `true` | Beta | 1.12 | | | `PVCProtection` | `false` | Alpha | 1.9 | 1.9 | | `ReadOnlyAPIDataVolumes` | `true` | Deprecated | 1.10 | | | `ResourceLimitsPriorityFunction` | `false` | Alpha | 1.9 | | | `RotateKubeletClientCertificate` | `true` | Beta | 1.7 | | | `RotateKubeletServerCertificate` | `false` | Alpha | 1.7 | | | `RunAsGroup` | `false` | Alpha | 1.10 | | | `RuntimeClass` | `false` | Alpha | 1.12 | | | `SCTPSupport` | `false` | Alpha | 1.12 | | | `ServiceNodeExclusion` | `false` | Alpha | 1.8 | | | `StorageObjectInUseProtection` | `true` | Beta | 1.10 | 1.10 | | `StorageObjectInUseProtection` | `true` | GA | 1.11 | | | `StreamingProxyRedirects` | `true` | Beta | 1.5 | | | `SupportIPVSProxyMode` | `false` | Alpha | 1.8 | 1.8 | | `SupportIPVSProxyMode` | `false` | Beta | 1.9 | 1.9 | | `SupportIPVSProxyMode` | `true` | Beta | 1.10 | 1.10 | | `SupportIPVSProxyMode` | `true` | GA | 1.11 | | | `SupportPodPidsLimit` | `false` | Alpha | 1.10 | | | `Sysctls` | `true` | Beta | 1.11 | | | `TaintBasedEvictions` | `false` | Alpha | 1.6 | | | `TaintNodesByCondition` | `false` | Alpha | 1.8 | | | `TaintNodesByCondition` | `true` | Beta | 1.12 | | | `TokenRequest` | `false` | Alpha | 1.10 | 1.11 | | `TokenRequest` | `True` | Beta | 1.12 | | | `TokenRequestProjection` | `false` | Alpha | 1.11 | 1.11 | | `TokenRequestProjection` | `True` | Beta | 1.12 | | | `TTLAfterFinished` | `false` | Alpha | 1.12 | | | `VolumeScheduling` | `false` | Alpha | 1.9 | 1.9 | | `VolumeScheduling` | `true` | Beta | 1.10 | | | `VolumeSubpathEnvExpansion` | `false` | Alpha | 1.11 | | | `ScheduleDaemonSetPods` | `true` | Beta | 1.12 | | ## References * [Kubernetes Feature Gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) # Best Practice This document aims at summarizing and highlighting the best practices found in user guides, quick-start documents, and examples. It is constantly updated and if you think there are useful best practices not included in this document, feel free to submit a Pull Request. ## General Configuration Tips * When defining a configuration file, specify the latest stable API version. * Save configuration files in a version control system before deploying them to the cluster. This allows for quick rollback when necessary and makes it easier to quickly create a cluster. * Use YAML format instead of JSON for configuration files. They are interchangeable in most scenarios, but YAML is more user-friendly. * Try to keep related objects in the same configuration file, it's easier to manage than splitting them into multiple files. See the configuration in [guestbook-all-in-one.yaml](https://github.com/kubernetes/examples/blob/master/guestbook/all-in-one/guestbook-all-in-one.yaml) for reference. * Specify the configuration file directory when using the `kubectl` command. * Avoid specifying unnecessary default configurations, this helps to keep the configuration files simple and reduces configuration errors. * Placing a description of the resource objects in an annotation can improve introspection. ## Bare Pods vs Replication Controllers and Jobs * If there are other options to replace "bare pods" (such as pods not bound to a [replication controller](https://kubernetes.io/docs/user-guide/replication-controller)), use them instead. * Bare pods will not be rescheduled in case of a node failure. * Replication Controllers will always recreate pods, except in scenarios where [`restartPolicy: Never`](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy) is explicitly specified. [Job](https://kubernetes.io/docs/concepts/jobs/run-to-completion-finite-workloads/) objects also apply. ## Services * It's generally best to create a [service](https://kubernetes.io/docs/concepts/services-networking/service/) before creating its related [replication controllers](https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/). This ensures that the service's environment variables are set up at container startup time. For new applications, it's recommended to access the service by its DNS name (rather than through environment variables). * Unless necessary (e.g. running a node daemon), don't use Pods with configured `hostPort` (used to specify the port number exposed on the host). When you bind a `hostPort` to a Pod, it can be difficult for the pod to be scheduled due to port conflicts. If you need to access ports for debugging purposes, you can use [kubectl proxy and apiserver proxy](https://kubernetes.io/docs/tasks/extend-kubernetes/http-proxy-access-api/) or [kubectl port-forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/). You can expose services to the external world using [Service](https://kubernetes.io/docs/concepts/services-networking/service/). If you do need to expose a Pod's port to the host, consider using a [NodePort](https://kubernetes.io/docs/user-guide/services/#type-nodeport) service. * For the same reason as `hostPort`, avoid using `hostNetwork`. * If you don't need kube-proxy's load balancing, consider using [headless services](https://kubernetes.io/docs/user-guide/services/#headless-services) (ClusterIP set to None). ## Utilizing Labels * Use [labels](https://kubernetes.io/docs/user-guide/labels/) to specify the semantic attributes of an application or Deployment. This allows you to select the suitable object group for the scenario, such as `app: myapp, tire: frontend, phase: test, deployment: v3`. * A service can be configured to span multiple deployments by simply omitting the release-related labels in its label selector. * Note that the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) object no longer needs to manage the version name of the replication controller. The Deployment object describes the desired state of the object, and if changes to the spec are applied, the Deployment controller will change the actual state to the desired state at a controlled rate. * Use labels for debugging. Kubernetes replication controllers and services use labels to match pods, allowing you to remove a pod from a controller or service by removing its relevant label. The controller will create a new pod to replace the removed one. This is a useful way to debug a previously "live" pod in an isolated environment. ## Container Images * The default container image pull policy is `IfNotPresent`, meaning the Kubelet will not pull from the image repository if the image is already present locally. If you want to always pull images from the repository, set the image pull policy in the yaml file to `Always` (`imagePullPolicy: Always`) or specify the image tag as `:latest`. * If the image tag is not set to `:latest`, for example `myimage:v1`, and the image with that tag has been updated, the Kubelet will not pull that image. You can generate a new tag (for example `myimage:v2`) after each image update and specify that version in the configuration file. * You can use the image digest to ensure the container always uses the same version of the image. * **Note:** In a production environment, avoid using the `:latest` tag when deploying containers. This makes it difficult to trace which version is running and how to rollback in case of failure. ## Using kubectl * Use `kubectl create -f ` or `kubectl apply -f `. Kubectl will automatically look for all files with the extensions `.yaml`, `.yml`, and `.json` in the directory and pass them to the `create` or `apply` command. * Using label selectors with `kubectl get` or `kubectl delete` can operate on a group of objects in bulk. * Use `kubectl run` and `expose` commands to quickly create a Deployment and Service with a single container, for example: ```bash kubectl run hello-world --replicas=2 --labels="run=load-balancer-example" --image=gcr.io/google-samples/node-hello:1.0 --port=8080 kubectl expose deployment hello-world --type=NodePort --name=example-service kubectl get pods --selector="run=load-balancer-example" --output=wide ``` ## References * [Configuration Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/) # Version Support ## Kubernetes version support In Kubernetes, versions are denoted as **x.y.z**, where x refers to the major version, y to the minor version, and z to the patch version. This versioning follows [Semantic Versioning](http://semver.org/), which means: * Major version: for incompatible changes to the API. * Minor version: for backward-compatible additions. * Patch version: for backward-compatible bug fixes. The Kubernetes project only maintains the latest three minor versions, each kept in a separate release branch. Serious issues and security fixes discovered in upstream versions are backported to these release branches, which are maintained by [Patch Releases](https://kubernetes.io/releases/patch-releases/). Minor versions are generally released every three months, so each release branch is typically maintained for about nine months. **Version support for different components** In Kubernetes, not all components necessarily share the same version. However, there are basic limitations when deploying mixed versions of components. **kube-apiserver** In [highly-available (HA) clusters](https://kubernetes.io/docs/setup/independent/high-availability/), the version gap between instances of kube-apiserver can't exceed one minor version. For instance, if the latest kube-apiserver version is 1.13, other instances of kube-apiserver can only be version 1.13 or 1.12. **kubelet** The kubelet version can't be higher than the kube-apiserver version, and it can only lag behind the kube-apiserver by up to two minor versions. For example: * if the `kube-apiserver` version is **1.13** * then `kubelet` can be version **1.13**, **1.12**, or **1.11** Furthermore, for a high-availability cluster: * if `kube-apiserver` versions are **1.13** and **1.12** * then `kubelet` can be version **1.12**, or **1.11** (but **1.13** would not be supported, as it would be higher than kube-apiserver's **1.12**) **kube-controller-manager, kube-scheduler, and cloud-controller-manager** The versions of `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` can't be higher than the kube-apiserver version. Typically, they should have the same version as the kube-apiserver, although they can also run with a variant of one minor version. For instance: * If `kube-apiserver` version is **1.13** * Then `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` can be versions **1.13** and **1.12** Similarly, for a high-availability cluster: * If `kube-apiserver` versions are **1.13** and **1.12** * Then `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` can be version **1.12** (but, **1.13** would not be supported, as it would be higher than apiserver's **1.12**) **kubectl** kubectl can differ from kube-apiserver by one minor version, such as: * if the `kube-apiserver` version is **1.13** * then `kubectl` can be versions **1.14**, **1.13**, and **1.12** **Version upgrade order** When upgrading from version 1.n to 1.(n+1), the following upgrade order must be followed. **kube-apiserver** Prerequisites for upgrade include: * In single-node clusters, kube-apiserver is version 1.n; in HA clusters, kube-apiserver is either version 1.n or 1.(n+1). * `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are all version 1.n. * The kubelet is version 1.n or 1.(n-1). * All registered injection webhooks can handle requests from the new version, for instance if ValidatingWebhookConfiguration and MutatingWebhookConfiguration have been updated to support the features introduced in version 1.(n+1). Now the kube-apiserver can be upgraded to 1.(n+1). However, it is important to note that **versions cannot hop over minor versions** during an upgrade. ### kube-controller-manager, kube-scheduler, and cloud-controller-manager The prerequisites for upgrade are: * The kube-apiserver has been upgraded to version 1.(n+1). With these conditions fulfilled, `kube-controller-manager`, `kube-scheduler` and `cloud-controller-manager` can be upgraded to version **1.(n+1)**. ### Kubelet The prerequisites for upgrade are: * The kube-apiserver has been upgraded to version 1.(n+1). * During the upgrade, the version gap between the kubelet and kube-apiserver must not exceed one minor version. Then the kubelet can be upgraded to version 1.(n+1). ## References * [Kubernetes Version and Version Skew Support Policy - Kubernetes](https://kubernetes.io/docs/setup/version-skew-policy/) # Setup Cluster ## The Architecture of a Kubernetes Cluster ![cluster](/files/8v17DGhqt1AgVA2Z6fEH) ### The etcd Cluster After obtaining a token from `https://discovery.etcd.io/new?size=3`, place `etcd.yaml` on each machine's `/etc/kubernetes/manifests/etcd.yaml` and replace `${DISCOVERY_TOKEN}`, `${NODE_NAME}`, and `${NODE_IP}`. With this, the kubelet can initiate an etcd cluster. For an etcd running outside the kubelet, refer to the [etcd cluster guide](https://etcd.io/docs/v3.5/op-guide/clustering/) for manually setting the cluster mode. ### The kube-apiserver Place `kube-apiserver.yaml` on each Master node's `/etc/kubernetes/manifests/`, and put related configurations into `/srv/kubernetes/`. This lets kubelet automatically create and launch the apiserver, which requires: * basic\_auth.csv - basic authentication username and password * ca.crt - Certificate Authority cert * known\_tokens.csv - tokens that specific entities (like the kubelet) can use to communicate with the apiserver * kubecfg.crt - Client certificate, public key * kubecfg.key - Client certificate, private key * server.cert - Server certificate, public key * server.key - Server certificate, private key After launching the apiserver, load balancing is crucial. This can be achieved via the elastic load balance service of cloud platforms or configuring master nodes with haproxy/lvs/nginx. Moreover, tools like Keepalived, OSPF, Pacemaker, etc., can ensure high availability of load balance nodes. Note: * For large-scale clusters, increase `--max-requests-inflight` (default at 400) * When using nginx, increase `proxy_timeout: 10m` ### Controller Manager and Scheduler It's important to ensure that at any given moment, only a single instance of both the controller manager and scheduler is running. This requires a leader election process, so include `--leader-elect=true` at startup, such as: ``` kube-scheduler --master=127.0.0.1:8080 --v=2 --leader-elect=true kube-controller-manager --master=127.0.0.1:8080 --cluster-cidr=10.245.0.0/16 --allocate-node-cidrs=true --service-account-private-key-file=/srv/kubernetes/server.key --v=2 --leader-elect=true ``` Placing `kube-scheduler.yaml` and `kube-controller-manager` on each Master node's `/etc/kubernetes/manifests/` and the related configuration into `/srv/kubernetes/` lets kubelet automatically create and start kube-scheduler and kube-controller-manager. ### kube-dns kube-dns can be deployed via the Deployment method. While kubeadm automatically creates it in a default setting, for large-scale clusters, you need to relax resource limits, like: ``` dns_replicas: 6 dns_cpu_limit: 100m dns_memory_limit: 512Mi dns_cpu_requests: 70m dns_memory_requests: 70Mi ``` Additionally, resources for dnsmasq need to be increased too, such as enlarging cache size to 10000, increasing concurrent handling ability with `--dns-forward-max=1000`, etc. ### Data Persistence In addition to the above configurations, persistent storage is essential for a high availability Kubernetes cluster. * For clusters deployed on public cloud, consider using persistent storage provided by the cloud platform, like AWS EBS or GCE persistent disk. * For clusters deployed on physical machines, consider network storage options like iSCSI, NFS, Gluster, Ceph, or even RAID. ## Azure On Azure, you can use AKS or acs-engine to deploy a Kubernetes cluster. For detailed deployment methods, refer [here](/en/setup/cluster/azure). ## GCE On GCE, you can conveniently deploy clusters utilizing cluster scripts: ``` # gce,aws,gke,azure-legacy,vsphere,openstack-heat,rackspace,libvirt-coreos export KUBERNETES_PROVIDER=gce curl -sS https://get.k8s.io | bash cd kubernetes cluster/kube-up.sh ``` ## AWS Deploying on AWS is best done using [kops](https://kubernetes.io/docs/setup/production-environment/tools/kops/). ## Physical or Virtual Machines On Linux physical or virtual machines, we recommend using [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) or [kubespray](/en/setup/cluster/kubespray) for Kubernetes cluster deployment. # kubeadm Are you looking to deploy Kubernetes using Docker runtime? Then look no further, as kubeadm is your go-to solution. It streamlines the process of setting up a Kubernetes cluster by using automation scripts. Let's walk through it together. To kick off with the process, clone the ops repository from GitHub \[1]. From the cloned repo, run the installation script for Kubernetes. Remember to set `USE_MIRROR=true` if you're setting this up in mainland China. Make sure to note down the console output of TOKEN and MASTER IP address, you will need these for installation on other nodes. So far so good, right? Now, let's move to the nodes, repeating the above process for each of them. Don't forget to set up the TOKEN, MASTER\_IP, and CONTAINER\_CIDR (Container Network Interface, part of the Kubernetes ecosystem that assigns IP addresses to Kubernetes objects) specific to your setup. Next, we will cover detailed steps on deploying a Kubernetes cluster using kubeadm. ## System initialization Whether you're running on Ubuntu or CentOS, you first need to initialize Docker and kubelet (the primary "node agent" that runs on each worker node in Kubernetes) on all your machines. Then you run commands to install Docker and Kubernetes. ## Setting up the Master Having done the initial setup, now is the time to initiate the master node. If you want to customize your Kubernetes service options (and you're the type of user who likes to refine things), you can do that too! We use a YAML file that lists all the configuration options to achieve this. If you choose to create kubeadm configuration file, specify the path of this YAML file while initializing the master: ```bash kubeadm init --config ./kubeadm.yaml ``` ## Network Plugin Configuration We offer configuration details for multiple options: CNI bridge, Flannel, Weave and Calico. Remember to set `--pod-network-cidr` according to the network plugin. ## Adding the Node To add a node to your cluster, run the `kubeadm join` command. Remember to replace ``, ``, and `` with your setup values. Just as with the Master, when adding a Node you have the option of customizing your Kubernetes service options. You can specify the NodeConfiguration configuration file path when you're adding the Node to the cluster: ```bash kubeadm join --config ./nodeconfig.yml --token $token ${master_ip} ``` ## Cloud Provider By default, kubeadm doesn’t include the Cloud Provider configuration. Therefore, when running on cloud platforms like Azure or AWS, you'll need to configure the Cloud Provider. ## Uninstalling To uninstall, you first need to drain and delete the respective node and then reset kubeadm. ## Upgrading Moving on to the dynamic upgrade, with support starting from kubeadm v1.8. The process involves uploading the kubeadm configuration, checking for newer versions on the master, and then issuing an upgrade command. For example, if a newer version v1.8.0 is available, execute `kubeadm upgrade apply v1.8.0` to upgrade the control plane. With manual upgrading, note that versions prior to kubeadm v1.7 don't support dynamic upgrading. ## Security Options By default, kubeadm enables automatic approval for Node client certificates. If you don't need them, you can opt to turn this off. ## References 1. [Kubeadm Reference Guide](https://kubernetes.io/docs/admin/kubeadm/) 2. [Upgrading kubeadm clusters from v1.14 to v1.15](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/) # kops [Kops](https://github.com/kubernetes/kops) is a top-tier tool for deploying Kubernetes clusters. Its versatility allows it to automatically set up high-availability Kubernetes clusters on platforms such as AWS, GCE, and VMWare vSphere. Here are some of its standout features: * Automated deployment of high-availability Kubernetes clusters. * Upgrade capability from clusters created with [kube-up](https://github.com/kubernetes/kops/blob/master/docs/upgrade_from_kubeup.md) to Kops versions. * Dry-run and automatic idempotent upgrades, based on a state synchronization model. * Auto-generation of AWS CloudFormation and Terraform configurations. * Customizable extension add-ons. * Command-line auto-completion. ## Installing kops and kubectl ```bash # on macOS brew install kubectl kops # on Linux wget https://github.com/kubernetes/kops/releases/download/1.7.0/kops-linux-amd64 chmod +x kops-linux-amd64 mv kops-linux-amd64 /usr/local/bin/kops ``` ## Launching on AWS First, you'll need to install AWS CLI and configure IAM: ```bash # install AWS CLI pip install awscli # configure iam aws iam create-group --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops aws iam create-user --user-name kops aws iam add-user-to-group --user-name kops --group-name kops aws iam create-access-key --user-name kops # configure the aws client to use your new IAM user aws configure # Use your new access and secret key here aws iam list-users # you should see a list of all your IAM users here # Because "aws configure" doesn't export these vars for kops to use, we export them now export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` Next, create a route53 domain: ```bash aws route53 create-hosted-zone --name dev.example.com --caller-reference 1 ``` Then, set up an S3 storage bucket: ```bash aws s3api create-bucket --bucket clusters.dev.example.com --region us-east-1 aws s3api put-bucket-versioning --bucket clusters.dev.example.com --versioning-configuration Status=Enabled ``` Now you're ready to deploy a Kubernetes cluster: ```bash export KOPS_STATE_STORE=s3://clusters.dev.example.com kops create cluster --zones=us-east-1c useast1.dev.example.com --yes ``` Want a high-availability cluster? No problem: ```bash kops create cluster \ --node-count 3 \ --zones us-west-2a,us-west-2b,us-west-2c \ --master-zones us-west-2a,us-west-2b,us-west-2c \ --node-size t2.medium \ --master-size t2.medium \ --topology private \ --networking kopeio-vxlan \ hacluster.example.com ``` When your needs shift, you can delete your cluster: ```bash kops delete cluster --name ${NAME} --yes ``` ## Launching on GCE ```bash # Create cluster in GCE. # This is an alpha feature. export KOPS_STATE_STORE="gs://mybucket-kops" export ZONES=${MASTER_ZONES:-"us-east1-b,us-east1-c,us-east1-d"} export KOPS_FEATURE_FLAGS=AlphaAllowGCE kops create cluster kubernetes-k8s-gce.example.com --zones $ZONES \ --master-zones $ZONES \ --node-count 3 --project my-gce-project \ --image "ubuntu-os-cloud/ubuntu-1604-xenial-v20170202" \ --yes ``` # Kubespray [Kubespray](https://github.com/kubernetes-incubator/kubespray) is a project under the Kubernetes incubator. Its mission is to provide a production-ready Kubernetes deployment solution. The project is based on Ansible Playbook to define system and Kubernetes cluster deployment tasks, with the following characteristics: * It can be deployed on AWS, GCE, Azure, OpenStack, and bare metal. * It allows for the deployment of highly available Kubernetes clusters. * It is composable, allowing users to choose Network Plugin (flannel, calico, canal, weave) for deployment. * It supports various Linux distributions (CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL7). This article will explain how to deploy Kubernetes to bare metal nodes using Kubespray. The versions will be as follows: * Kubernetes v1.7.3 * Etcd v3.2.4 * Flannel v0.8.0 * Docker v17.04.0-ce ## Node Information The operating system for the installation test environment will be Ubuntu 16.04 Server and the other details are as follows: | IP Address | Role | CPU | Memory | | --------------- | ---------------- | --- | ------ | | 192.168.121.179 | master1 + deploy | 2 | 4G | | 192.168.121.106 | node1 | 2 | 4G | | 192.168.121.197 | node2 | 2 | 4G | | 192.168.121.123 | node3 | 2 | 4G | > Here, the master is the primary control node, and the node is the work node. ## Preparatory Information * All nodes' networks can communicate with each other. * The deployment node (here, master1) can log in to other nodes without needing SSH passwords. * All nodes possess Sudoer permissions and don't require password input. * All nodes need to have Python installed. * All nodes need to set `/etc/hosts` to resolve all hosts. * Modify all nodes' `/etc/resolv.conf` ```bash $ echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf ``` * The deployment node (here, master1) installs Ansible >= 2.3.0. The process for installing Ansible on Ubuntu 16.04 is as follows: ```bash $ sudo sed -i 's/us.archive.ubuntu.com/tw.archive.ubuntu.com/g' /etc/apt/sources.list $ sudo apt-get install -y software-properties-common $ sudo apt-add-repository -y ppa:ansible/ansible $ sudo apt-get update && sudo apt-get install -y ansible git cowsay python-pip python-netaddr libssl-dev ``` ## Installing Kubespray and Preparing Deployment Information First, install kubespray-cli through pypi. Although the official sources say they have switched to a Go language version of the tool, it hasn't been updated, so we'll use the pypi version for now: ```bash $ sudo pip install -U kubespray ``` After installation, add a configuration file `~/.kubespray.yml` and include the following content: ```bash $ cat < ~/.kubespray.yml kubespray_git_repo: "https://github.com/kubernetes-incubator/kubespray.git" # Logging options loglevel: "info" EOF ``` Then use the kubespray cli to generate an inventory file: ```bash $ kubespray prepare --masters master1 --etcds master1 --nodes node1 node2 node3 ``` Add some content in the inventory.cfg: ``` $ vim ~/.kubespray/inventory/inventory.cfg [all] master1 ansible_host=192.168.121.179 ansible_user=root ip=192.168.121.179 node1 ansible_host=192.168.121.106 ansible_user=root ip=192.168.121.106 node2 ansible_host=192.168.121.197 ansible_user=root ip=192.168.121.197 node3 ansible_host=192.168.121.123 ansible_user=root ip=192.168.121.123 [kube-master] master1 [kube-node] node1 node2 node3 [etcd] master1 [k8s-cluster:children] kube-node kube-master ``` > You can also create a new `inventory` to describe the deployment nodes. After completing the above, execute the following command to deploy the Kubernetes cluster: ```bash $ time kubespray deploy --verbose -u root -k .ssh/id_rsa -n flannel Run kubernetes cluster deployment with the above command ? [Y/n]y ... master1 : ok=368 changed=89 unreachable=0 failed=0 node1 : ok=305 changed=73 unreachable=0 failed=0 node2 : ok=276 changed=62 unreachable=0 failed=0 node3 : ok=276 changed=62 unreachable=0 failed=0 Kubernetes deployed successfully ``` > The `-n` refers to the type of network plugin to be deployed, currently supporting calico, flannel, weave, and canal. ## Verifying the Cluster After Ansible has run, if no errors have occurred, you can start operating the Kubernetes, such as obtaining version information: ```bash $ kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.7.3+coreos.0", GitCommit:"9212f77ed8c169a0afa02e58dce87913c6387b3e", GitTreeState:"clean", BuildDate:"2017-04-04T00:32:53Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.7.3+coreos.0", GitCommit:"9212f77ed8c169a0afa02e58dce87913c6387b3e", GitTreeState:"clean", BuildDate:"2017-04-04T00:32:53Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} ``` Get the current cluster node status: ```bash $ kubectl get node NAME STATUS AGE VERSION master1 Ready,SchedulingDisabled 11m v1.7.3+coreos.0 node1 Ready 11m v1.7.3+coreos.0 node2 Ready 11m v1.7.3+coreos.0 node3 Ready 11m v1.7.3+coreos. ``` Check the current cluster Pod status: ```bash $ kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE dnsmasq-975202658-6jj3n 1/1 Running 0 14m dnsmasq-975202658-h4rn9 1/1 Running 0 14m dnsmasq-autoscaler-2349860636-kfpx0 1/1 Running 0 14m flannel-master1 1/1 Running 1 14m flannel-node1 1/1 Running 1 14m flannel-node2 1/1 Running 1 14m flannel-node3 1/1 Running 1 14m kube-apiserver-master1 1/1 Running 0 15m kube-controller-manager-master1 1/1 Running 0 15m kube-proxy-master1 1/1 Running 1 14m kube-proxy-node1 1/1 Running 1 14m kube-proxy-node2 1/1 Running 1 14m kube-proxy-node3 1/1 Running 1 14m kube-scheduler-master1 1/1 Running 0 15m kubedns-1519522227-thmrh 3/3 Running 0 14m kubedns-autoscaler-2999057513-tx14j 1/1 Running 0 14m nginx-proxy-node1 1/1 Running 1 14m nginx-proxy-node2 1/1 Running 1 14m nginx-proxy-node3 1/1 Running 1 14m ``` # Azure Azure's container services, specifically the Azure Container Service (AKS) and Azure Container Service (ACS), offer unique opportunities for deployment and management of containerized applications. AKS, recently debuted by Microsoft Azure, runs separately from ACS. Through AKS, users can easily deploy and manage containerized applications without needing specialist knowledge of container business processes. Even better? The AKS platform doesn't require any additional user maintenance - it offers automatic upgrades, fault repair, and scaling of resource pools as needed. What’s more, users only pay for virtual machines running their containers - AKS’s cluster management is completely free of charge. Since 2015, Microsoft Azure's ACS has supported a range of container orchestration tools, including Kubernetes, DCOS, and Dockers Swarm. What's also cool about ACS is that its core function is open source – users can check it out and download it from their Github page at . ## AKS: A Deep Dive ### Getting Started Following are simple steps to get started with AKS. The process will require the Azure CLI software to be installed on your computer. If it's not already installed, you can do that from [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). Before you can create AKS clusters, you first need to enable AKS using the following command: ```bash # Enable AKS az provider register -n Microsoft.ContainerService ``` The next step is to create a resource group to manage all the related resources: ```bash # Create Resource Group az group create --name group1 --location centralus ``` Now, you are ready to create your AKS clusters: ```bash # Create aks az aks create --resource-group group1 --name myK8sCluster --node-count 3 --generate-ssh-keys ``` It’s almost done! Once your cluster is created, install and configure kubectl: ```bash # Install kubectl az aks install-cli # Configure kubectl az aks get-credentials --resource-group=group1 --name=myK8sCluster ``` > Using version 2.0.24 of azure-cli might result in the `az aks get-credentials` command failing. You can fix this by upgrading to a newer version or reverting back to version 2.0.23. ### Connect with Dashboard ```bash # Create dashboard az aks browse --resource-group group1 --name myK8SCluster ``` ### Manually enlarge or shrink your cluster ```bash az aks scale --resource-group=group1 --name=myK8SCluster --agent-count 5 ``` ### Upgrade your cluster Get your cluster's current version and check the ones you can upgrade to: ```bash # Current version and upgradable versions az aks get-versions --name myK8sCluster --resource-group group1 --output table # Upgrade to version 1.11.3 az aks upgrade --name myK8sCluster --resource-group group1 --kubernetes-version 1.11.3 ``` The graphic below shows the process of deploying a version 1.7.7 cluster and upgrading it to version 1.8.1: ![](https://feisky.xyz/images/aks-examples.gif) ### Using Helm Other tools and services in the Kubernetes community can also be used, such as deploying the Nginx Ingress Controller with Helm: ```bash helm init --client-only helm install stable/nginx-ingress ``` ### Cluster deletion Your cluster can be deleted when no longer required: ```bash az group delete --name group1 --yes --no-wait ``` ## Looking at acs-engine Although AKS is expected to be the future of Azure's container services, many users value being able to manage their own container clusters to ensure maximum flexibility (such as customizing master services). Such users can utilize the open-source [acs-engine](https://github.com/Azure/acs-engine) for creating and managing their clusters. Acs-engine is, actually, core to ACS. It is a command line tool that assists with deployment and management of Kubernetes, Swarm, and DC/OS clusters, by transforming a container cluster descriptor file into a group of ARM (Azure Resource Manager) templates. In acs-engine, each cluster is described through a json file. For example, a Kubernetes cluster can be described as follows: ```bash { "apiVersion": "vlabs", "properties": { "orchestratorProfile": { ... }, "masterProfile": { ... }, "agentPoolProfiles": [ { ... } ], ... } ``` ## Azure's Container Registry Around the same time that AKS was launched, Azure also debuted their Azure Container Registry (ACR) services. This service hosts users' private images. ```bash # Create ACR az acr create --resource-group myResourceGroup --name --sku Basic --admin-enabled true # Login to the registry: az acr login --name # Tag your image: az acr list --resource-group myResourceGroup --query "[].{acrLoginServer:loginServer}" --output table docker tag azure-vote-front /azure-vote-front:redis-v1 # Push your image to the registry docker push /azure-vote-front:redis-v1 # List available images az acr repository list --name --output table ``` ## Virtual Kubelet Azure's container instances (ACI) offer a simplified way to run containers in Azure as ACI effectively absolves users from having to configure any virtual machines or other sophisticated services. Ideal for fast growth and resource adjustment, ACI is designed to be relatively straightforward. The [Virtual Kubelet](https://github.com/virtual-kubelet/virtual-kubelet) allows ACI to function as an unlimited Node for a Kubernetes cluster, making Node quantity a non-issue. ACI then automatically manages the cluster's resources based on the containers in operation. ![](/files/jnBHDrqxQJniE08eGut1) You can use Helm to deploy your Virtual Kubelet: ```bash RELEASE_NAME=virtual-kubelet CHART_URL=https://github.com/virtual-kubelet/virtual-kubelet/raw/master/charts/virtual-kubelet-0.4.0.tgz helm install "$CHART_URL" --name "$RELEASE_NAME" --namespace kube-system --set env.azureClientId=,env.azureClientKey=,env.azureTenantId=,env.azureSubscriptionId=,env.aciResourceGroup=,env.nodeName=aci, env.nodeOsType=,env.nodeTaint=azure.com/aci ``` ## References * [AKS – Managed Kubernetes on Azure](https://www.reddit.com/r/AZURE/comments/7d7diz/ama_aks_managed_kubernetes_on_azure/) * [Azure Container Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/) * [Azure/acs-engine Github](https://github.com/Azure/acs-engine) * [acs-engine/examples](https://github.com/Azure/acs-engine/tree/master/examples) # Windows Beginning with v1.5, Kubernetes started to introduce support for Windows nodes in its alpha version, upgrading this support to beta with the release of v1.9. Some of the key features for Windows containers are include: * Pod-level support for Windows containers (isolation=process) * Kernel load balancing based on Virtual Filtering Platform (VFP) Hyper-v Switch Extension * Windows container management via Container Runtime Interface (CRI) * Support for the use of the kubeadm command to add Windows nodes to an existing cluster * Recommended use of Windows Server Version 1803+ and Docker Version 17.06+ > Note: > > 1. Control plane services still run on Linux servers, with only Kubelet, Kube-proxy, Docker, and network plugin services running on Windows nodes. > 2. Windows Server 1803 is recommended since it fixes issues with Windows container symlinks, allowing ServiceAccount and ConfigMap to function normally. ## Downloads You can download the released binary files for Windows servers from . For instance, ```bash wget https://dl.k8s.io/v1.15.0/kubernetes-node-windows-amd64.tar.gz ``` Alternatively, you can compile from Kubernetes source code: ```bash go get -u k8s.io/kubernetes cd $GOPATH/src/k8s.io/kubernetes # Build the kubelet KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet # Build the kube-proxy KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy # You will find the output binaries under the folder _output/local/bin/windows/ ``` ## Network Plugins The following network plugins are supported in Windows Server (note that the network plugin on Windows nodes must be the same as on Linux nodes): 1. L3 routing network plugins like [wincni](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/cni/wincni.exe), where routing is configured in TOR switches, routers, or cloud services 2. [Azure VNET CNI Plugin](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) 3. [Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay](https://github.com/openvswitch/ovn-kubernetes/) 4. Flannel v0.10.0+ 5. Calico v3.0.1+ 6. [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge) 7. [win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay) For more network topology modes, please refer to [Windows container network drivers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/network-drivers-topologies). ### L3 Routing Topology ![](/files/49571VmplPN7x24xNUZv) Example configuration for the wincni network plugin: ```javascript { "cniVersion": "0.2.0", "name": "l2bridge", "type": "wincni.exe", "master": "Ethernet", "ipam": { "environment": "azure", "subnet": "10.10.187.64/26", "routes": [{ "GW": "10.10.187.66" }] }, "dns": { "Nameservers": [ "11.0.0.10" ] }, "AdditionalArgs": [{ "Name": "EndpointPolicy", "Value": { "Type": "OutBoundNAT", "ExceptionList": [ "11.0.0.0/8", "10.10.0.0/16", "10.127.132.128/25" ] } }, { "Name": "EndpointPolicy", "Value": { "Type": "ROUTE", "DestinationPrefix": "11.0.0.0/8", "NeedEncap": true } }, { "Name": "EndpointPolicy", "Value": { "Type": "ROUTE", "DestinationPrefix": "10.127.132.213/32", "NeedEncap": true } } ] } ``` ### OVS Network Topology ![](/files/ut9yNg3MBOYmiPIvsdFC) ## Deployment ### kubeadm If the master node is deployed via kubeadm, Windows nodes can also be deployed through kubeadm: ```bash kubeadm.exe join --token : --discovery-token-ca-cert-hash sha256: ``` ### Azure On Azure, it is recommended to use [acs-engine](/en/setup/cluster/azure#Windows) for automatic deployment of Master and Windows nodes. First, create a Kubernetes cluster configuration file that includes Windows, named `windows.json` ```javascript { "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorVersion": "1.11.1", "kubernetesConfig": { "networkPolicy": "none", "enableAggregatedAPIs": true, "enableRbac": true } }, "masterProfile": { "count": 3, "dnsPrefix": "kubernetes-windows", "vmSize": "Standard_D2_v3" }, "agentPoolProfiles": [ { "name": "windowspool1", "count": 3, "vmSize": "Standard_D2_v3", "availabilityProfile": "AvailabilitySet", "osType": "Windows" } ], "windowsProfile": { "adminUsername": "", "adminPassword": "" }, "linuxProfile": { "adminUsername": "azure", "ssh": { "publicKeys": [ { "keyData": "" } ] } }, "servicePrincipalProfile": { "clientId": "", "secret": "" } } } ``` You can then deploy using acs-engine: ```bash # create a new resource group. az group create --name myResourceGroup --location "centralus" # start deploy the kubernetes acs-engine deploy --resource-group myResourceGroup --subscription-id --auto-suffix --api-model windows.json --location centralus --dns-prefix # setup kubectl export KUBECONFIG="$(pwd)/_output//kubeconfig/kubeconfig.centralus.json" kubectl get node ``` ### Manual Deployment (1) Install Docker on Windows Server by following these instructions: [Install Docker](https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server) ``` Install-Module -Name DockerMsftProvider -Repository PSGallery -Force Install-Package -Name Docker -ProviderName DockerMsftProvider Restart-Computer -Force ``` (2) Download kubelet.exe and kube-proxy.exe based on the earlier download section. (3) Copy the Node spec file (kube config) from the Master node. (4) Configure the CNI network plugin and base images. ``` wget https://github.com/Microsoft/SDN/archive/master.zip -o master.zip Expand-Archive master.zip -DestinationPath master mkdir C:/k/ mv master/SDN-master/Kubernetes/windows/* C:/k/ rm -recurse -force master,master.zip ``` ``` docker pull microsoft/windowsservercore:1709 docker tag microsoft/windowsservercore:1709 microsoft/windowsservercore:latest cd C:/k/ docker build -t kubeletwin/pause . ``` (5) Use [start-kubelet.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubelet.ps1) to start kubelet.exe, and use [start-kubeproxy.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubeproxy.ps1) to start kube-proxy.exe ```bash ./start-kubelet.ps1 -ClusterCidr 192.168.0.0/16 ./start-kubeproxy.ps1 ``` (6) If you are using the Host-Gateway network plugin, you will also need to use [AddRoutes.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/AddRoutes.ps1) to add static routes. For a detailed step-by-step guide, you can refer to [this](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows.md). ## Running Windows Containers To schedule a container on a Windows node, use the NodeSelector `beta.kubernetes.io/os: windows`, for example: ```yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: iis spec: replicas: 1 template: metadata: labels: app: iis spec: nodeSelector: beta.kubernetes.io/os: windows containers: - name: iis image: microsoft/iis resources: limits: memory: "128Mi" cpu: 2 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: labels: app: iis name: iis namespace: default spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: iis type: NodePort ``` Running a DaemonSet: ```yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: my-DaemonSet labels: app: foo spec: template: metadata: labels: app: foo spec: containers: - name: foo image: microsoft/windowsservercore:1709 nodeSelector: beta.kubernetes.io/os: windows ``` ## Known Issues ### Secrets and ConfigMaps can only be used as environmental variables This is a known issue with versions 1709 and earlier and can be fixed by upgrading to version 1803. ### Volume Support Local, emptyDir, hostPath, AzureDisk, AzureFile, and flexvolume are currently the only types of volumes supported by Windows containers. It's important to note that the format for the Volume's path needs to be `mountPath: "C:\\etc\\foo"` or `mountPath: "C:/etc/foo"`. ```yaml apiVersion: v1 kind: Pod metadata: name: hostpath-pod spec: containers: - name: hostpath-nano image: microsoft/nanoserver:1709 stdin: true tty: true volumeMounts: - name: blah mountPath: "C:\\etc\\foo" readOnly: true nodeSelector: beta.kubernetes.io/os: windows volumes: - name: blah hostPath: path: "C:\\AzureData" ``` ```yaml apiVersion: v1 kind: Pod metadata: name: empty-dir-pod spec: containers: - image: microsoft/nanoserver:1709 name: empty-dir-nano stdin: true tty: true volumeMounts: - mountPath: /cache name: cache-volume - mountPath: C:/scratch name: scratch-volume volumes: - name: cache-volume emptyDir: {} - name: scratch-volume emptyDir: {} nodeSelector: beta.kubernetes.io/os: windows ``` ### Image Version Matching In `Windows Server version 1709`, you must use images with the 1709 tag, such as * microsoft/aspnet:4.7.1-windowsservercore-1709 * microsoft/windowsservercore:1709 * microsoft/iis:windowsservercore-1709 Likewise, for `Windows Server version 1803`, you must use images with the 1803 tag. For `Windows Server 2016`, you need to use images with the ltsc2016 tag, such as `microsoft/windowsservercore:ltsc2016`. ## Setting CPU and Memory Starting from v1.10, Kubernetes supports setting CPU and memory for Windows containers: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: iis spec: replicas: 3 template: metadata: labels: app: iis spec: containers: - name: iis image: microsoft/iis resources: limits: memory: "128Mi" cpu: 2 ports: - containerPort: 80 ``` ## Hyper-V Containers Starting from v1.10, containers with Hyper-V isolation are supported (Alpha). Before use, the kubelet needs to be configured to enable the `HyperVContainer` feature switch. Then you can specify a container for Hyper-V isolation using Annotation `experimental.windows.kubernetes.io/isolation-type=hyperv`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: iis spec: replicas: 3 template: metadata: labels: app: iis annotations: experimental.windows.kubernetes.io/isolation-type: hyperv spec: containers: - name: iis image: microsoft/iis ports: - containerPort: 80 ``` ### Other Known Issues * Only Windows Server 1709 or later versions support running multiple containers in a Pod (only Process isolation is supported) * StatefulSet is not currently supported * Automatic expansion of Windows Server Container Pods (Horizontal Pod Autoscaling) is not yet supported * The OS version of the Windows container needs to match the Host OS version; otherwise, the container will not be able to start * When using L3 or Host GW networks, you cannot access Kubernetes Services directly from the Windows Node (there is no such issue when using OVS/OVN) * On Window Server running on VMWare Fusion kubelet.exe may fail to start (this has been fixed in [#57124](https://github.com/kubernetes/kubernetes/pull/57124)) * The Weave network plugin is not currently supported * Calico network plugin only supports Policy-Only mode * For .NET containers that need to use `:` as an environment variable, you can replace `:` in the environment variable with `__` (see [here](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration#configuration-by-environment) for reference) ## Appendix: Docker EE Installation Method To install Docker EE stable version: ``` Install-Module -Name DockerMsftProvider -Repository PSGallery -Force Install-Package -Name docker -ProviderName DockerMsftProvider Restart-Computer -Force ``` To install Docker EE preview version: ``` Install-Module DockerProvider Install-Package -Name Docker -ProviderName DockerProvider -RequiredVersion preview ``` To upgrade Docker EE version: ``` # Check the installed version Get-Package -Name Docker -ProviderName DockerMsftProvider # Find the current version Find-Package -Name Docker -ProviderName DockerMsftProvider # Upgrade Docker EE Install-Package -Name Docker -ProviderName DockerMsftProvider -Update -Force Start-Service Docker ``` ## Reference Documents * [Guide for adding Windows Nodes in Kubernetes](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/) * [Intro to Windows support in Kubernetes](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/) * [Guide for scheduling Windows containers in Kubernetes](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/) * [Kubernetes for Windows Walkthroughs](https://github.com/PatrickLang/KubernetesForWindowsTutorial) # LinuxKit LinuxKit offers a minimal, immutable Linux framework built on containers. To get a taste of what it can do, check out the simple introduction on [LinuxKit's GitHub page](https://github.com/linuxkit/linuxkit). In this discussion, we'll be using LinuxKit to build a Kubernetes image and deploy a simple Kubernetes cluster. ![](/files/-Mj32RzEhCn5QfdEqDHq) This step-by-step guide operates in the `Mac OS X` environment. The components we'll use are: * Kubernetes v1.7.2 * Etcd v3 * Weave * Docker v17.06.0-ce ## Preliminary Needs Before we begin, we need to ensure that: * `Docker` has been installed and activated on the host system. * `Git` has been installed on the host. * The LinuxKit project has been downloaded on the host, we have built Moby and LinuxKit tools. Here are the commands for creating Moby and LinuxKit: ```bash $ git clone https://github.com/linuxkit/linuxkit.git $ cd linuxkit $ make $ ./bin/moby version moby version 0.0 commit: c2b081ed8a9f690820cc0c0568238e641848f58f $ ./bin/linuxkit version linuxkit version 0.0 commit: 0e3ca695d07d1c9870eca71fb7dd9ede31a38380 ``` ## Building a Kubernetes System Image First, we need to create a Linux system that comes pre-packaged with Kubernetes. Luckily there's already an example provided by the authorities. The following steps will guide you through the building process: ```bash $ cd linuxkit/projects/kubernetes/ $ make build-vm-images ... Create outputs: kube-node-kernel kube-node-initrd.img kube-node-cmdline ``` ## Deploying a Kubernetes Cluster Once the image is ready, we can use the following command to start the Master OS and fetch its IP address: ```bash $ ./boot.sh (ns: getty) linuxkit-025000000002:~\# ip addr show dev eth0 2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:50:00:00:00:02 brd ff:ff:ff:ff:ff:ff inet 192.168.65.3/24 brd 192.168.65.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::abf0:9fa4:d0f4:8da2/64 scope link valid_lft forever preferred_lft forever ``` After it has started, open a new console to SSH into the Master and initialize it with kubeadm: ```bash $ cd linuxkit/projects/kubernetes/ $ ./ssh_into_kubelet.sh 192.168.65.3 linuxkit-025000000002:/\# kubeadm-init.sh ... kubeadm join --token 4236d3.29f61af661c49dbf 192.168.65.3:6443 ``` Once kubeadm is finished, you will see a Token. Please remember this Token information. Next, open another console and run the command to initiate the Node: ```bash console1>$ ./boot.sh 1 --token 4236d3.29f61af661c49dbf 192.168.65.3:6443 ``` Note: To initialize nodes, follow the format `./boot.sh [ ...]`. Next, open two additional consoles to join the cluster: ```bash console2> $ ./boot.sh 2 --token 4236d3.29f61af661c49dbf 192.168.65.3:6443 console3> $ ./boot.sh 3 --token 4236d3.29f61af661c49dbf 192.168.65.3:6443 ``` After completing the above, go back to the Master node and run the following command to check the status of the nodes: ```bash $ kubectl get no NAME STATUS AGE VERSION linuxkit-025000000002 Ready 16m v1.7.2 linuxkit-025000000003 Ready 6m v1.7.2 linuxkit-025000000004 Ready 1m v1.7.2 linuxkit-025000000005 Ready 1m v1.7.2 ``` ## Deploying a Simple Nginx Service Kubernetes lets you build applications and services directly using instructions, or design app deployment configurations using YAML and JSON files. Let's spin up a simple Nginx service: ```bash $ kubectl run nginx --image=nginx --replicas=1 --port=80 $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-1423793266-v0hpb 1/1 Running 0 38s 10.42.0.1 linuxkit-025000000004 ``` After that, we will create a Service(svc) to provide external network access to the app: ```bash $ kubectl expose deploy nginx --port=80 --type=NodePort $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.96.0.1 443/TCP 19m nginx 10.108.41.230 80:31773/TCP 5s ``` Since our deployment isn't on physical machines but uses Docker namespace networking, we will need to use `ubuntu-desktop-lxde-vnc` to view the Nginx app: ```bash $ docker run -it --rm -p 6080:80 dorowu/ubuntu-desktop-lxde-vnc ``` After that, connect to HTML VNC via the browser at `http://localhost:6080`. ![](/files/I4CS3NXfMDkVx49MUmSs) Finally, to shut down nodes just execute the following: ```bash $ halt [1503.034689] reboot: Power down ``` If you've followed these steps, congratulations! You've built and deployed your own Kubernetes cluster with LinuxKit! # Setup Addons ## Add-On Components Upon deploying a Kubernetes cluster, it becomes essential to install a series of add-on components (addons). These addons are often crucial for the regular operation of the cluster. The [addon-manager](/en/setup/addon-list/addon-manager), commonly utilized to manage the add-ons in a cluster, operates within the Kubernetes cluster's Master node. It oversees all the extensions in the `$ADDON_PATH` (defaulting to `/etc/kubernetes/addons/`) directory to ensure they are functioning as intended. Some of the common components include: * [addon-manager](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/addon-manager) * [cluster-loadbalancing](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing) * [dashboard](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard) * [device-plugins/nvidia-gpu](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/device-plugins/nvidia-gpu) * [dns-horizontal-autoscaler](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns-horizontal-autoscaler) * [dns](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns) * [fluentd-elasticsearch](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch) * [ip-masq-agent](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/ip-masq-agent) * [istio](https://istio.io) * [kube-proxy](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/kube-proxy) * [metrics-server](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server) * [node-problem-detector](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/node-problem-detector) * [prometheus](https://prometheus.io/) * [storage-class](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-class) For additional extensions, refer to [Installing Addons](https://kubernetes.io/docs/concepts/cluster-administration/addons/) and [Legacy Addons](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons). *** ## Power-Up Your Kubernetes: Essential Add-Ons for Peak Performance Once you've got your Kubernetes cluster up and running, think of it like a smartphone without apps—it's functional, but you're not getting the most out of it. That's where add-ons come in—they're like the apps that power-up your cluster's capabilities. Picture a digital maestro—[addon-manager](/en/setup/addon-list/addon-manager)—nestled within the Master node of your Kubernetes command center. This wizard is keeping an eagle eye on the `$ADDON_PATH` (which, by default, is the `/etc/kubernetes/addons/` directory), ensuring every component is humming along just right. Ready for the tour? Here's the all-star lineup of add-ons that can turn your cluster into a powerhouse: * **Command Central,** [**addon-manager**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/addon-manager): Keeps your add-ons tightly regulated. * **Traffic Cop,** [**cluster-loadbalancing**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing): Directs digital traffic efficiently. * **Mission Control,** [**dashboard**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard): A clear visual on your cluster activities. * **The Muscle,** [**device-plugins/nvidia-gpu**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/device-plugins/nvidia-gpu): Amping up your computing power. * **The Balancer,** [**dns-horizontal-autoscaler**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns-horizontal-autoscaler): Keeps network naming at peak performance. * **The Communicator,** [**dns**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns): For seamless service-name resolutions. * **The Analyzer,** [**fluentd-elasticsearch**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch): Dives deep into data logs. * **Stealth Mode,** [**ip-masq-agent**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/ip-masq-agent): Keeps IP addresses under wraps. * **The Envoy,** [**istio**](https://istio.io): Smart networking for your services. * **The Enforcer,** [**kube-proxy**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/kube-proxy): Manages network rules and connections. * **The Scout,** [**metrics-server**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server): Keeps an eye on resource usage. * **The Watchdog,** [**node-problem-detector**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/node-problem-detector): On the lookout for pesky issues. * **The Statistician,** [**prometheus**](https://prometheus.io/): Monitors and alerts like a pro. * **The Organizer,** [**storage-class**](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-class): Manages your storage needs efficiently. For those hungry for even more power-ups, gear up with additional extensions detailed in [Installing Addons](https://kubernetes.io/docs/concepts/cluster-administration/addons/) and for a nostalgic twist, visit [Legacy Addons](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons). Your Kubernetes cluster is set for the big leagues now! # Addon-manager ## Addon-manager The Addon-manager is a service running on the Kubernetes cluster's Master nodes designed to manage various Add-ons. It maintains all the extensions present in the `$ADDON_PATH` (which defaults to `/etc/kubernetes/addons/`) to ensure they always operate in the desired state. Addon-manager supports two types of labels: * For extensions tagged with `addonmanager.kubernetes.io/mode=Reconcile`, modifications through the API are not allowed. This means that: * Any changes made through the API will automatically revert to the configuration in `/etc/kubernetes/addons/`. * If an extension is deleted via the API, it will be automatically recreated from the configuration in `/etc/kubernetes/addons/`. * Removing configuration from `/etc/kubernetes/addons/` will also delete the corresponding Kubernetes resources. * Essentially, modifications can only be made by adjusting the configuration in `/etc/kubernetes/addons/`. * For extensions with the `addonmanager.kubernetes.io/mode=EnsureExists` label, there's only a check to ensure the existence of the extension without checking for configuration changes. In effect: * The configuration can be modified via the API without it being automatically reverted. * If an extension is deleted via the API, it will be automatically recreated from the configuration in `/etc/kubernetes/addons/`. * However, if the configuration is removed from `/etc/kubernetes/addons/`, the Kubernetes resources will not be deleted. ### Deployment Method Save the following YAML into the `/etc/kubernetes/manifests/kube-addon-manager.yaml` file on all Master nodes: ```yaml apiVersion: v1 kind: Pod metadata: name: kube-addon-manager namespace: kube-system annotations: scheduler.alpha.kubernetes.io/critical-pod: '' seccomp.security.alpha.kubernetes.io/pod: 'docker/default' labels: component: kube-addon-manager spec: hostNetwork: true containers: - name: kube-addon-manager # When updating version also bump it in: # - test/kubemark/resources/manifests/kube-addon-manager.yaml image: k8s.gcr.io/kube-addon-manager:v8.7 command: - /bin/bash - -c - exec /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 resources: requests: cpu: 3m memory: 50Mi volumeMounts: - mountPath: /etc/kubernetes/ name: addons readOnly: true - mountPath: /var/log name: varlog readOnly: false env: - name: KUBECTL_EXTRA_PRUNE_WHITELIST value: {{kubectl_extra_prune_whitelist}} volumes: - hostPath: path: /etc/kubernetes/ name: addons - hostPath: path: /var/log name: varlog ``` ### Source Code The source code for Addon-manager is hosted at . *** ## Addon-manager: A Kubernetes Cluster Maestro Meet the Addon-manager, the essential service that diligently works in the background of Kubernetes Master nodes keeping add-ons in check. It's like a digital conductor for the `$ADDON_PATH` – typically `/etc/kubernetes/addons/` – maintaining a seamless operation of all extensions according to the script written for them. The Addon-manager is adept at handling two kinds of labels that dictate extension behavior: * Extensions marked with `addonmanager.kubernetes.io/mode=Reconcile` play by strict rules: * Try tweaking them through the API, and like a boomerang, they'll revert to their `/etc/kubernetes/addons/` settings. * Delete them, and they magically reappear, thanks to the `/etc/kubernetes/addons/` backup band. * However, pull their files from `/etc/kubernetes/addons/`, and it's curtains down for those Kubernetes resources. * The gist is, backstage configuration edits in `/etc/kubernetes/addons/` are the only way to shuffle their act. * Extensions donning the `addonmanager.kubernetes.io/mode=EnsureExists` label are the free spirits: * API modifications? Go ahead; no strings attached for a rollback. * Vanish through the API, and voilà, they make an encore using the `/etc/kubernetes/addons/` script. * But should their part get axed from `/etc/kubernetes/addons/`, the show goes on without the Kubernetes resources curtain call. ### Setting the Stage To roll out the Addon-manager across the Master nodes' ensemble, simply script the following YAML into the `/etc/kubernetes/manifests/kube-addon-manager.yaml` of each maestro's station: ```yaml ... [YAML content remains unchanged] ... ``` ### Ensemble's Composition For those wanting to peek at the Addon-manager's score, the source code resides at . Consider it an open invitation to see the magic behind the Kubernetes curtain! # DNS DNS within Kubernetes serves as one of its core functionalities, offering naming services through extensions like kube-dns or CoreDNS. ## CoreDNS: The Preferred Choice for Kubernetes Starting with version 1.11, [CoreDNS](https://coredns.io/) is available for naming services and was adopted as the default DNS service from v1.13 onward. CoreDNS distinguishes itself by its increased efficiency and reduced resource consumption, making it the recommended replacement for kube-dns in cluster DNS service provision. The process of upgrading from kube-dns to CoreDNS involves: ```bash $ git clone https://github.com/coredns/deployment $ cd deployment/kubernetes $ ./deploy.sh | kubectl apply -f - $ kubectl delete --namespace=kube-system deployment kube-dns ``` For new deployments, [click here](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns) to view the configuration method for the CoreDNS extension. ## Supported DNS Record Types * Service * A record: Generates `my-svc.my-namespace.svc.cluster.local` with IP resolution in two scenarios: * Normal Service resolves to Cluster IP. * Headless Service resolves to a specified list of Pod IPs. * SRV record: Generates `_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local`. * Pod * A record: `pod-ip-address.my-namespace.pod.cluster.local`. * Specific hostname and subdomain: `hostname.custom-subdomain.default.svc.cluster.local` as illustrated below: ```yaml apiVersion: v1 kind: Pod metadata: name: busybox2 labels: name: busybox spec: hostname: busybox-2 subdomain: default-subdomain containers: - image: busybox command: - sleep - "3600" name: busybox ``` ![](/files/NzXjp7JHASnAExWMRdQ2) ## Configuring Private and Upstream DNS Servers As of Kubernetes 1.6, customization of stub domains and upstream name servers is possible by providing a ConfigMap to kube-dns. For example, the following setup inserts a separate private root DNS server along with two upstream DNS servers. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kube-dns namespace: kube-system data: stubDomains: | {“acme.local”: [“1.2.3.4”]} upstreamNameservers: | [“8.8.8.8”, “8.8.4.4”] ``` Using the specific configuration mentioned above, query requests are initially sent to the DNS cache layer of kube-dns (the Dnsmasq server). The server first checks the request's suffix, sending those with a cluster suffix (e.g., ”.cluster.local”) to kube-dns and names with a stub domain suffix (e.g., ”.acme.local”) to the designated private DNS server \[“1.2.3.4”]. Lastly, requests not matching any of these suffixes will be forwarded to the upstream DNS \[“8.8.8.8”, “8.8.4.4”]. ![](/files/CdYGEvZagpkrQ9ftevDY) ## kube-dns ### Deploying a kube-dns Example DNS services are generally deployed as extensions, for instance by placing the [kube-dns.yaml](https://github.com/feiskyer/kubernetes-handbook/raw/master/manifests/kubedns/kube-dns.yaml) in the `/etc/kubernetes/addons` directory on the Master node. Manual deployment is also an option: ```bash kubectl apply -f https://github.com/feiskyer/kubernetes-handbook/raw/master/manifests/kubedns/kube-dns.yaml ``` This action will initiate a Pod within Kubernetes that contains three containers running the three DNS-related services: ```bash # kube-dns container kube-dns --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2 # dnsmasq container dnsmasq-nanny -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --log-facility=- --server=127.0.0.1#10053 # sidecar container sidecar --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local.,5,A --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local.,5,A ``` Kubernetes v1.10 also supports the Beta version of CoreDNS which has performance advantages over kube-dns. This can be deployed as an extension by placing [coredns.yaml](https://github.com/feiskyer/kubernetes-handbook/blob/master/manifests/kubedns/coredns.yaml) in the `/etc/kubernetes/addons` directory on the Master node, or deployed manually: ```bash kubectl apply -f https://github.com/feiskyer/kubernetes-handbook/raw/master/manifests/kubedns/coredns.yaml ``` ### How kube-dns Works As the diagram below illustrates, kube-dns is comprised of three containers: * kube-dns: The core component of DNS services, mainly consisting of KubeDNS and SkyDNS * KubeDNS is responsible for monitoring changes in Services and Endpoints, and updating the relevant information in SkyDNS. * SkyDNS handles DNS resolution, listening on port 10053 (tcp/udp), as well as port 10055 for metrics. * The kube-dns service also listens on port 8081 for health checks. * dnsmasq-nanny: Responsible for starting dnsmasq and restarting it upon configuration changes. * dnsmasq's upstream is SkyDNS, meaning internal DNS resolution within the cluster is managed by SkyDNS. * sidecar: Responsible for health checks and serving DNS metrics (listening on port 10054). ![](/files/zfazh90os7ff5lG9n2y9) ### A Glimpse at the Source Code The code for kube-dns has been separated from the main Kubernetes repository, now housed at . The code for kube-dns, dnsmasq-nanny, and sidecar all start at `cmd//main.go`, and they call on `pkg/dns`, `pkg/dnsmasq`, and `pkg/sidecar` respectively to execute their tasks. The core DNS resolution, however, directly utilizes code from `github.com/skynetservices/skydns/server` with specific implementation visible at [skynetservices/skydns](https://github.com/skynetservices/skydns/tree/master/server). ## Common Issues **DNS Resolution Issues in Ubuntu 18.04** Ubuntu 18.04 by default enables systemd-resolved, which places `nameserver 127.0.0.53` in the system's /etc/resolv.conf. Since this is a local address, it may cause CoreDNS or kube-dns to fail in resolving external web addresses. The solution involves replacing the systemd-resolved generated resolv.conf file: ```bash sudo rm /etc/resolv.conf sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf ``` Alternatively, manually specify the resolv.conf path for the DNS service: ```bash --resolv-conf=/run/systemd/resolve/resolv.conf ``` ## Reference Materials * [dns-pod-service Introduction](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) * [coredns/coredns](https://github.com/coredns/coredns) # Dashboard Deploying the Kubernetes Dashboard is incredibly straightforward. To get started, simply run: ```bash kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml ``` After a short wait, the dashboard will be ready: ```bash $ kubectl -n kubernetes-dashboard get pod NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-76585494d8-xhhzx 1/1 Running 0 20m kubernetes-dashboard-5996555fd8-snzh9 1/1 Running 0 20m $ kubectl -n kubernetes-dashboard get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.0.58.210 8000/TCP 20m kubernetes-dashboard ClusterIP 10.0.182.172 443/TCP 20m ``` Then, after running `kubectl proxy`, you can access it through the following link: ```bash http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ ``` ## Login Authentication ### Login by importing the API Server's certificate In versions prior to v1.7, the Dashboard did not offer a login feature and was run over http, so you could access it directly through `kubectl port-forard` or `kubectl proxy`. You could also access it directly through the API Server's proxy address – the kubernetes-dashboard address outputted by `kubectl cluster-info`. Since the kubernetes API Server runs over https, you'll need to import the certificate into your system to access it: ```bash # generate p12 cert kubectl config view --flatten -o jsonpath='{.users[?(.name == "username")].user.client-key-data}' | base64 -d > client.key kubectl config view --flatten -o jsonpath='{.users[?(.name == "username")].user.client-certificate-data}' | base64 -d > client.crt openssl pkcs12 -export -in client.crt -inkey client.key -out client.p12 ``` By importing `client.p12` into your system, you could directly access `https:///api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/overview` through your browser. ### Login using the kubeconfig configuration file Starting with version v1.7.0, the Dashboard supports login via kubeconfig configuration files. When you open the Dashboard page, it will automatically redirect to the login interface. Select the Kubeconfig method and choose the local kubeconfig configuration file to proceed. ![](https://user-images.githubusercontent.com/2285385/30416718-8ee657d8-992d-11e7-84c8-9ba5f4c78bb2.png) ### Login using a restricted Token Also starting with version v1.7.0, the Dashboard supports login via Token. Be aware that the Token retrieved from Kubernetes needs to be Base64-decoded before it can be used for login. The following is an example of creating a service account token that can only access the `demo` namespace when RBAC is enabled: ```bash # Create demo namespace kubectl create namespace demo # Create and restrict access to the demo namespace cat </metrics` * Cadvisor metrics: `http://127.0.0.1:8001/api/v1/proxy/nodes//metrics/cadvisor` Thus, in tools like Prometheus, the new Metrics API must be used to obtain this data, as in the following Prometheus configuration that automatically sets up the cadvisor metrics API: ```bash helm install stable/prometheus --set rbac.create=true --name prometheus --namespace monitoring ``` Note: The port monitored by cadvisor will be removed in v1.12, and it is recommended that all external tools use the Kubelet Metrics API instead. ### InfluxDB and Grafana [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/) is an open-source distributed time series, event, and metrics database; Grafana is InfluxDB's Dashboard, providing powerful chart display capabilities. They are often used in combination to display graphically visualized monitoring data. ![](/files/eIBjz66FueGZciEcHecB) ### Heapster Kubelet's built-in cAdvisor only provides single-machine container resource usage statistics, whereas [Heapster](https://github.com/kubernetes/heapster) provides whole-cluster resource monitoring and supports persistent data storage into backends like InfluxDB, Google Cloud Monitoring, or [other backends](https://github.com/kubernetes/heapster). Note: * Heapster is recommended only for Kubernetes v1.7.X or older clusters. * Starting from Kubernetes v1.8, resource usage metrics (such as CPU and memory usage of containers) are obtained through the Metrics API, and HPA also queries necessary data from the metrics-server. * **Heapster has been deprecated in v1.11, and it is recommended to deploy** [**metrics-server**](/en/setup/addon-list/metrics) **instead of Heapster for versions v1.8 and above** Heapster first queries all Node information from the Kubernetes apiserver, then collects node and container resource usage from the kubelet-provided API, while providing Prometheus format data through the `/metrics` API. Heapster-collected data can be pushed to various persistence backend storages, such as InfluxDB, Google Cloud Monitoring, OpenTSDB, etc. ![](/files/xLNto4weEnH0eWoK0vwD) #### Deploying Heapster, InfluxDB, and Grafana After Kubernetes deployment is successful, services such as the dashboard, DNS, and monitoring are also typically deployed by default, such as via `cluster/kube-up.sh`: ```bash $ kubectl cluster-info Kubernetes master is running at https://kubernetes-master Heapster is running at https://kubernetes-master/api/v1/proxy/namespaces/kube-system/services/heapster KubeDNS is running at https://kubernetes-master/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://kubernetes-master/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard Grafana is running at https://kubernetes-master/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana InfluxDB is running at https://kubernetes-master/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb ``` If these services have not been automatically deployed, they can be deployed following the [kubernetes/heapster](https://github.com/kubernetes/heapster/tree/master/deploy/kube-config): ```bash git clone https://github.com/kubernetes/heapster cd heapster kubectl create -f deploy/kube-config/influxdb/ kubectl create -f deploy/kube-config/rbac/heapster-rbac.yaml ``` Note that to access these services, the apiserver certificate must be imported into the browser first for authentication. The visiting process can also be simplified by using the kubectl proxy (no certificate import needed): ```bash # Start proxy kubectl proxy --address='0.0.0.0' --port=8080 --accept-hosts='^*$' & ``` Then, open `http://:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana` to access Grafana. ![](/files/um8WeaY6ZEBxUDgz3dre) ### Prometheus [Prometheus](https://prometheus.io) is another monitoring and time-series database and provides alarm functionality as well. It offers a powerful query language and HTTP interface and also supports data export to Grafana. ![prometheus](/files/9Qbpl8hYbR0GIdpGooV2) Using Prometheus to monitor Kubernetes requires proper data source configuration, a simple example is [prometheus.yml](https://github.com/feiskyer/kubernetes-handbook/tree/39446adab1639adec0fe906a85dfd0ba1f0b45f9/addons/prometheus.txt). It is recommended to use [Prometheus Operator](https://github.com/coreos/prometheus-operator) or [Prometheus Chart](https://github.com/kubernetes/charts/tree/master/stable/prometheus) to deploy and manage Prometheus, such as ```bash helm install stable/prometheus-operator --name prometheus-operator --namespace monitoring ``` Access Prometheus via port forwarding, like `kubectl --namespace monitoring port-forward service/kube-prometheus-prometheus :9090` ![prometheus-web](/files/-LpOEi2ncv8OZnsnzLFo) If the exporter-kubelets feature is not working properly, such as reporting a `server returned HTTP status 401 Unauthorized` error, webhook authentication needs to be configured for the Kubelet: ```bash kubelet --authentication-token-webhook=true --authorization-mode=Webhook ``` If you see K8SControllerManagerDown and K8SSchedulerDown alerts, it means that kube-controller-manager and kube-scheduler are running as Pods in the cluster and the labels of the monitoring services deployed by prometheus do not match theirs. The problem can be solved by modifying the service labels, such as ```bash kubectl -n kube-system set selector service kube-prometheus-exporter-kube-controller-manager component=kube-controller-manager kubectl -n kube-system set selector service kube-prometheus-exporter-kube-scheduler component=kube-scheduler ``` Query the admin password for Grafana ```bash kubectl get secret --namespace monitoring kube-prometheus-grafana -o jsonpath="{.data.user}" | base64 --decode ; echo kubectl get secret --namespace monitoring kube-prometheus-grafana -o jsonpath="{.data.password}" | base64 --decode ; echo ``` Then, access the Grafana interface via port forwarding ```bash kubectl port-forward -n monitoring service/kube-prometheus-grafana :80 ``` Add a Prometheus-type Data Source, fill in the original address `http://prometheus-prometheus-server.monitoring`. > Note: Prometheus Operator does not support service discovery through the `prometheus.io/scrape` annotation and requires you to define [ServiceMonitor](https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md#generic-servicemonitor-example) to fetch service metrics. ### Node Problem Detector Kubernetes nodes may experience various hardware, kernel, or runtime issues that could potentially lead to service anomalies. Node Problem Detector (NPD) is a service designed to monitor these anomalies. NPD runs as a DaemonSet on each Node, updating the NodeCondition (such as KernelDaedlock, DockerHung, BadDisk, etc.) or Node Event (such as OOM Kill, etc.) when anomalies occur. Refer to [kubernetes/node-problem-detector](https://github.com/kubernetes/node-problem-detector#start-daemonset) to deploy NPD, or you can use Helm for deployment: ```bash # add repo helm repo add feisky https://feisky.xyz/kubernetes-charts helm update # install packages helm install feisky/node-problem-detector --namespace kube-system --name npd ``` ### Node Reboot Daemon Nodes in Kubernetes clusters typically enable automatic security updates, which helps to minimize losses due to system vulnerabilities. However, updates involving the kernel generally require a system reboot to take effect. At this point, manual or automatic methods are needed to reboot nodes. [Kured (KUbernetes REboot Daemon)](https://github.com/weaveworks/kured) is such a daemon that * Monitors `/var/run/reboot-required` signal to reboot nodes * Restarts one node at a time using DaemonSet Annotation * Evicts nodes before rebooting and resumes scheduling afterwards * Cancels reboot based on Prometheus alerts (e.g., `--alert-filter-regexp=^(RebootRequired|AnotherBenignAlert|...$`) * Slack notifications Deployment method ```bash kubectl apply -f https://github.com/weaveworks/kured/releases/download/1.0.0/kured-ds.yaml ``` ### Other Container Monitoring Systems In addition to the above monitoring tools, there are many other open source or commercial systems available to assist with monitoring, such as * [Sysdig](http://blog.kubernetes.io/2015/11/monitoring-Kubernetes-with-Sysdig.html) * [Weave scope](https://www.weave.works/docs/scope/latest/features/) * [Datadog](https://www.datadoghq.com/) * [Sematext](https://sematext.com/) #### sysdig sysdig is a container troubleshooting tool that offers both open source and commercial versions. For regular troubleshooting, the open source version suffices. Aside from sysdig, there are two other auxiliary tools * csysdig: Automatically installed with sysdig, provides a command-line interface * [sysdig-inspect](https://github.com/draios/sysdig-inspect): Provides a graphical interface for sysdig-saved trace files (e.g., `sudo sysdig -w filename.scap`) (not real-time) **Install sysdig** ```bash # on Linux curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash # on MacOS brew install sysdig ``` Usage examples ```bash # Refer https://www.sysdig.org/wiki/sysdig-examples/. # View the top network connections for a single container sysdig -pc -c topconns # Show the network data exchanged with the host 192.168.0.1 sysdig -s2000 -A -c echo_fds fd.cip=192.168.0.1 # List all the incoming connections that are not served by apache. sysdig -p"%proc.name %fd.name" "evt.type=accept and proc.name!=httpd" # View the CPU/Network/IO usage of the processes running inside the container. sysdig -pc -c topprocs_cpu container.id=2e854c4525b8 sysdig -pc -c topprocs_net container.id=2e854c4525b8 sysdig -pc -c topfiles_bytes container.id=2e854c4525b8 # See the files where apache spends the most time doing I/O sysdig -c topfiles_time proc.name=httpd # Show all the interactive commands executed inside a given container. sysdig -pc -c spy_users # Show every time a file is opened under /etc. sysdig evt.type=open and fd.name ``` #### Weave Scope Weave Scope is another visual container monitoring and troubleshooting tool. Unlike sysdig, it does not have a powerful command-line tool but does offer a straightforward and user-friendly interactive interface that automatically outlines the entire cluster's topology and can be extended by plugins. From its official website description, its features include * [Interactive topology interface](https://www.weave.works/docs/scope/latest/features/#topology-mapping) * [Graphic mode and table mode](https://www.weave.works/docs/scope/latest/features/#mode) * [Filtering function](https://www.weave.works/docs/scope/latest/features/#flexible-filtering) * [Search function](https://www.weave.works/docs/scope/latest/features/#powerful-search) * [Real-time metrics](https://www.weave.works/docs/scope/latest/features/#real-time-app-and-container-metrics) * [Container troubleshooting](https://www.weave.works/docs/scope/latest/features/#interact-with-and-manage-containers) * [Plugin extensions](https://www.weave.works/docs/scope/latest/features/#custom-plugins) Weave Scope consists of [App and Probe](https://www.weave.works/docs/scope/latest/how-it-works) * Probe is responsible for collecting container and host information and sending it to the App * App processes this information, generates corresponding reports, and displays them in an interactive interface ```bash +--Docker host----------+ +--Docker host----------+ .---------------. | +--Container------+ | | +--Container------+ | | Browser | | | | | | | | | |---------------| | | +-----------+ | | | | +-----------+ | | | |----->| | scope-app |<-----. .----->| scope-app | | | | | | | +-----------+ | | \ / | | +-----------+ | | | | | | ^ | | \/ | | ^ | | '---------------' | | | | | /\ | | | | | | | +-------------+ | | / \ | | +-------------+ | | | | | scope-probe |-----' '-----| scope-probe | | | | | +-------------+ | | | | +-------------+ | | | | | | | | | | | +-----------------+ | | +-----------------+ | +-----------------------+ +-----------------------+ ``` **Install Weave Scope** ```bash kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')&k8s-service-type=LoadBalancer" ``` After installation, the interactive interface can be accessed through the weave-scope-app ```bash kubectl -n weave get service weave-scope-app ``` ![](/files/wvvZ8hcbVqUN2Mz09Zq7) You can also view real-time status and metric data of all containers in the Pod by clicking on the Pod: ![](/files/w654HNU1Fsyy4JsGEQK4) ### Reference Documents * [Kubernetes Heapster](https://github.com/kubernetes/heapster) Now, let's move on to the rephrased version to make it more accessible to a broad audience as a popular science article. ## Keeping an Eye on Kubernetes: A Guide on Tools and Tips The Kubernetes community is like a vibrant ecosystem with a toolbox that helps you peek into the health and state of your containerized applications and clusters. Plus, thanks to Prometheus, you can even get a virtual tap on the shoulder with alerts if anything goes awry. Here's the lowdown on the tools you can strap to your Kubernetes utility belt: * **cAdvisor** is your on-site inspector, built-in with the Kubelet, keeping tabs on resource consumption for containers and nodes, and chatting up the world with its metrics API. * Pair [**InfluxDB**](https://www.influxdata.com/time-series-platform/influxdb/) and [**Grafana**](http://grafana.org/), and you get a dynamic duo providing not just a robust time-series database but also snazzy dashboards to visualize that precious monitoring data. * The [**metrics-server**](/en/setup/addon-list/metrics) is the cluster's main data cruncher, but remember, it's all about the here and now—no dwelling on the past with historical data. * If you’re curious about the state of your Kubernetes resources, [**kube-state-metrics**](https://github.com/kubernetes/kube-state-metrics) is your go-to for up-to-the-moment metrics. * [**Prometheus**](https://prometheus.io) is like the Swiss Army knife in the toolbox—an observant monitoring system and a time-series database, coupled with an alarm bell to alert you. * \*\*\[Node Problem Detector]\( # Logging ## Log Management ELK is the golden trio for container log collection, processing, and searching: * Logstash (or Fluentd) is responsible for log collection * Elasticsearch stores logs and provides search capabilities * Kibana handles log querying and visualization Note: Kubernetes by default uses fluentd (launched as a DaemonSet) to collect logs and then sends them to elasticsearch. **Pro Tip** When deploying a cluster with `cluster/kube-up.sh`, you can set the `KUBE_LOGGING_DESTINATION` environment variable to automatically deploy Elasticsearch and Kibana and use fluentd to collect logs (see configuration at [addons/fluentd-elasticsearch](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch)): ```bash KUBE_LOGGING_DESTINATION=elasticsearch KUBE_ENABLE_NODE_LOGGING=true cluster/kube-up.sh ``` If you're using GCE or GKE, you can also [send logs to Google Cloud Logging](https://kubernetes.io/docs/user-guide/logging/stackdriver/) and integrate with Google Cloud Storage and BigQuery. For other logging solutions, you can customize the docker log driver to send logs to splunk, awslogs, and more. ### Deployment Method Since the Fluentd daemonset is only scheduled to run on Nodes with the label `beta.kubernetes.io/fluentd-ds-ready=true`, you need to label the Nodes accordingly: ```bash kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready=true ``` Then download the manifest and deploy: ```bash $ git clone https://github.com/kubernetes/kubernetes $ cd kubernetes/cluster/addons/fluentd-elasticsearch $ kubectl apply -f . clusterrole "elasticsearch-logging" configured clusterrolebinding "elasticsearch-logging" configured replicationcontroller "elasticsearch-logging-v1" configured service "elasticsearch-logging" configured serviceaccount "elasticsearch-logging" configured clusterrole "fluentd-es" configured clusterrolebinding "fluentd-es" configured daemonset "fluentd-es-v1.24" configured serviceaccount "fluentd-es" configured deployment "kibana-logging" configured service "kibana-logging" configured ``` Note: The Kibana container might take a while during the first startup to optimize and cache bundles (Optimizing and caching bundles for kibana and statusPage. This may take a few minutes). Monitor the initialization via logs: ```bash $ kubectl -n kube-system logs kibana-logging-1237565573-p88lm -f ``` ### Accessing Kibana You can find the Kibana service access URL from the output of `kubectl cluster-info`. Note that you'll need to import the apiserver certificate into your browser for authentication: ```bash $ kubectl cluster-info | grep Kibana Kibana is running at https://10.0.4.3:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy ``` Alternatively, use the kubectl proxy to access without needing to import the certificate: ```bash # Start the proxy kubectl proxy --address='0.0.0.0' --port=8080 --accept-hosts='^*$' & ``` Then open `http://:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana#`. In the Settings -> Indices page, create an index, select Index contains time-based events, use the default `logstash-*` pattern, and click Create. ![](/files/VbnvUmmW11IoKEJsFOnN) ### Filebeat In addition to Fluentd and Logstash, you can use [Filebeat](https://www.elastic.co/products/beats/filebeat) for log collection: ```bash kubectl apply -f https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml ``` Note: The default setup assumes Elasticsearch is accessible via `elasticsearch:9200`. If it's different, modify the details before deployment: ```bash - name: ELASTICSEARCH_HOST value: elasticsearch - name: ELASTICSEARCH_PORT value: "9200" - name: ELASTICSEARCH_USERNAME value: elastic - name: ELASTICSEARCH_PASSWORD value: changeme ``` ### Reference Documents * [Logging Agent For Elasticsearch](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch) * [Logging Using Elasticsearch and Kibana](https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) *** ## The Golden Trio for Log Mastery ELK stands out as the go-to combination for handling the entire lifecycle of log data in container environments: * Logstash (or its alternative, Fluentd) takes on the role of log collection. * Elasticsearch acts as the data storehouse, providing robust search capabilities. * Kibana serves up the user interface, streamlining log queries and visual presentation. Quick heads-up: Kubernetes has a default set-up with fluentd, running as a DaemonSet, to scoop up logs and shuttle them off to Elasticsearch. **Handy Hint** Deploying your cluster with `cluster/kube-up.sh` gets even better when you use the `KUBE_LOGGING_DESTINATION` environment variable. This nifty trick sets up Elasticsearch and Kibana on autopilot, with fluentd gathering the logs for you (dive into the settings at [addons/fluentd-elasticsearch](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch)): ```bash KUBE_LOGGING_DESTINATION=elasticsearch KUBE_ENABLE_NODE_LOGGING=true cluster/kube-up.sh ``` Google's Cloud ecosystem users (think GCE or GKE) can funnel logs straight to Google Cloud Logging and also bring Google Cloud Storage and BigQuery into the mix. Feeling adventurous with logging solutions? Swap out the docker log driver to integrate services like Splunk or awslogs. ### Setting the Scene for Deployment Fluentd daemonsets have a VIP list, and only nodes wearing the `beta.kubernetes.io/fluentd-ds-ready=true` badge get the daemonsets' attention. So start by tagging your nodes: ```bash kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready=true ``` Next up, grab the manifest and deploy to your heart's content: ```bash $ git clone https://github.com/kubernetes/kubernetes $ cd kubernetes/cluster/addons/fluentd-elasticsearch $ kubectl apply -f . ``` (Expect the usual "configured" symphony for roles, bindings, services, and deployments.) Just a note: The first time Kibana boots up, it's going to take its sweet time optimizing and caching (imagine it humming "Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"). You can keep an eye on the warm-up process via logs: ```bash $ kubectl -n kube-system logs kibana-logging-1237565573-p88lm -f ``` ### Peeking into Kibana Kibana's door can be found in the `kubectl cluster-info` output. Just a reminder to sync up with the apiserver certificate in your browser for a smooth handshake: ```bash $ kubectl cluster-info | grep Kibana ``` Or, bypass the whole certificate song and dance by delegating to kubectl proxy: ```bash # Activate the proxy kubectl proxy --address='0.0.0.0' --port=8080 --accept-hosts='^*$' & ``` Go ahead and launch `http://:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana#`. When you land, set up a new index. Go with the flow—choose Index contains time-based events, stick with the `logstash-*` default, and hit Create. ![](/files/VbnvUmmW11IoKEJsFOnN) ### Enter Filebeat For those who march to a different beat, Filebeat from the Elastic family offers another avenue for gathering logs with gusto: ```bash kubectl apply -f https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml ``` Bear in mind, it presumes you can waltz straight into Elasticsearch at `elasticsearch:9200`. If your access route is different, adjust the settings and then deploy: ```bash - name: ELASTICSEARCH_HOST value: elasticsearch ``` (and so on, with the port, username, and password) ### Treasure Trove of References If you're itching to explore deeper, check out these documents: * [Logging Agent For Elasticsearch](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch) * [Logging Using Elasticsearch and Kibana](https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) # Metrics Starting from v1.8, metrics pertaining to resource usage (such as CPU and memory utilization for containers) can now be accessed through the Metrics API. It’s key to note: * The Metrics API is solely for fetching current metrics data and doesn't retain historical data. * The Metrics API URI starts with `/apis/metrics.k8s.io/`, and is maintained on [k8s.io/metrics](https://github.com/kubernetes/metrics). * Deployment of the `metrics-server` is a prerequisite for utilizing this API, as it retrieves data by calling the Kubelet Summary API. ## Kubernetes Monitoring Architecture The [Kubernetes Monitoring Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/monitoring_architecture.md) is composed of two segments: * Core Metrics Pipeline (illustrated in black in the diagram below): These are crucial metrics fundamental to the functioning of Kubernetes, sourced from the Kubelet, cAdvisor, and others, and provided by the metrics-server for the Dashboard, HPA controller, and the like. * Monitoring Pipeline (illustrated in blue in the diagram below): This pipeline is built atop the core metrics, e.g., Prometheus can collect core metrics from the metrics-server, and other non-core metrics from different sources (such as Node Exporter), to then construct a monitoring and alerting system. ![](/files/hZSwX69QR1wI0MjRD8Ry) ## Enabling API Aggregation Prior to deploying the metrics-server, API Aggregation must be enabled in the kube-apiserver by adding the following configuration: ```bash --requestheader-client-ca-file=/etc/kubernetes/certs/proxy-ca.crt --proxy-client-cert-file=/etc/kubernetes/certs/proxy.crt --proxy-client-key-file=/etc/kubernetes/certs/proxy.key --requestheader-allowed-names=aggregator --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User ``` If kube-proxy is not running on the Master, you would also need to configure: ```bash --enable-aggregator-routing=true ``` ## Deploying metrics-server ```bash $ git clone https://github.com/kubernetes-incubator/metrics-server $ cd metrics-server $ kubectl create -f deploy/1.8+/ ``` Subsequently, the metrics-server should be operational: ```bash kubectl -n kube-system get pods -l k8s-app=metrics-server ``` ## Metrics API The [Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/resource-metrics-api.md) can be accessed via `kubectl proxy`: * `http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/nodes` * `http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/nodes/` * `http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/pods` * `http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/namespaces//pods/` These APIs can also be directly accessed through kubectl commands, for example: * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes` * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods` * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/` * `kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces//pods/` ## Troubleshooting If the metrics-server Pod fails to start properly, such as being in a CrashLoopBackOff state with an ever-increasing restartCount, it's likely experiencing communication issues with the kube-apiserver. Checking the logs of the Pod could reveal: ```bash dial tcp 10.96.0.1:443: i/o timeout ``` A possible solution is: ```bash echo "ExecStartPost=/sbin/iptables -P FORWARD ACCEPT" >> /etc/systemd/system/docker.service.d/exec_start.conf systemctl daemon-reload systemctl restart docker ``` ## References * [Core metrics pipeline](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/) * [metrics-server](https://github.com/kubernetes-incubator/metrics-server) # GPU Kubernetes now supports the allocation of GPU resources for containers (currently only NVIDIA GPUs), which is widely used in scenarios like deep learning. ## How to Use ### Kubernetes v1.8 and Later Starting with Kubernetes v1.8, GPUs are supported through the DevicePlugin feature. Prior to use, several configurations are needed: * Enable the following feature gates on kubelet/kube-apiserver/kube-controller-manager: `--feature-gates="DevicePlugins=true"` * Install Nvidia drivers on all Nodes, including NVIDIA Cuda Toolkit and cuDNN * Configure Kubelet to use Docker as the container engine (which is the default setting), as other container engines do not yet support this feature #### NVIDIA Plugin NVIDIA requires nvidia-docker. To install nvidia-docker: ```bash # Install docker-ce sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt-get update sudo apt-get install docker-ce # Add the package repositories curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \ sudo apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/nvidia-docker.list | \ sudo tee /etc/apt/sources.list.d/nvidia-docker.list sudo apt-get update # Install nvidia-docker2 and reload the Docker daemon configuration sudo apt-get install -y nvidia-docker2 sudo pkill -SIGHUP dockerd # Test nvidia-smi with the latest official CUDA image docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi ``` Set Docker default runtime to nvidia: ```bash # cat /etc/docker/daemon.json { "default-runtime": "nvidia", "runtimes": { "nvidia": { "path": "/usr/bin/nvidia-container-runtime", "runtimeArgs": [] } } } ``` Deploy the NVIDIA device plugin: ```bash # For Kubernetes v1.8 kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.8/nvidia-device-plugin.yml # For Kubernetes v1.9 kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml ``` #### GCE/GKE GPU Plugin This plugin does not require nvidia-docker and also supports CRI container runtimes. ```bash # Install NVIDIA drivers on Container-Optimized OS: kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/daemonset.yaml # Install NVIDIA drivers on Ubuntu (experimental): kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/nvidia-driver-installer/ubuntu/daemonset.yaml # Install the device plugin: kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.9/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml ``` #### Example of Requesting `nvidia.com/gpu` Resources ```yaml apiVersion: v1 kind: Pod metadata: name: cuda-vector-add spec: restartPolicy: OnFailure containers: - name: cuda-vector-add # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile image: "k8s.gcr.io/cuda-vector-add:v0.1" resources: limits: nvidia.com/gpu: 1 # requesting 1 GPU ``` ### Kubernetes v1.6 and v1.7 > `alpha.kubernetes.io/nvidia-gpu` has been deprecated in v1.10, please use `nvidia.com/gpu` for newer versions. To use GPUs in Kubernetes v1.6 and v1.7, prerequisite configurations are required: * Install Nvidia drivers on all Nodes, including NVIDIA Cuda Toolkit and cuDNN * Enable the feature gates `--feature-gates="Accelerators=true"` on apiserver and kubelet * Configure Kubelet to use Docker as the container engine (the default setting), as other container engines are not yet supported Use the resource name `alpha.kubernetes.io/nvidia-gpu` to specify the number of GPUs required, for example: ```yaml apiVersion: v1 kind: Pod metadata: name: tensorflow spec: restartPolicy: Never containers: - image: gcr.io/tensorflow/tensorflow:latest-gpu name: gpu-container-1 command: ["python"] env: - name: LD_LIBRARY_PATH value: /usr/lib/nvidia args: - -u - -c - from tensorflow.python.client import device_lib; print(device_lib.list_local_devices()) resources: limits: alpha.kubernetes.io/nvidia-gpu: 1 # requests one GPU volumeMounts: - mountPath: /usr/local/nvidia/bin name: bin - mountPath: /usr/lib/nvidia name: lib - mountPath: /usr/lib/x86_64-linux-gnu/libcuda.so name: libcuda-so - mountPath: /usr/lib/x86_64-linux-gnu/libcuda.so.1 name: libcuda-so-1 - mountPath: /usr/lib/x86_64-linux-gnu/libcuda.so.375.66 name: libcuda-so-375-66 volumes: - name: bin hostPath: path: /usr/lib/nvidia-375/bin - name: lib hostPath: path: /usr/lib/nvidia-375 - name: libcuda-so hostPath: path: /usr/lib/x86_64-linux-gnu/libcuda.so - name: libcuda-so-1 hostPath: path: /usr/lib/x86_64-linux-gnu/libcuda.so.1 - name: libcuda-so-375-66 hostPath: path: /usr/lib/x86_64-linux-gnu/libcuda.so.375.66 ``` ```bash $ kubectl create -f pod.yaml pod "tensorflow" created $ kubectl logs tensorflow ... [name: "/cpu:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 9675741273569321173 , name: "/gpu:0" device_type: "GPU" memory_limit: 11332668621 locality { bus_id: 1 } incarnation: 7807115828340118187 physical_device_desc: "device: 0, name: Tesla K80, pci bus id: 0000:00:04.0" ] ``` Note: * GPU resources must be requested in `resources.limits`, `resources.requests` are not valid * Containers may request either 1 GPU or multiple GPUs, but not fractional parts of a GPU * GPUs cannot be shared among multiple containers * It is assumed by default that all Nodes are equipped with GPUs of the same model ## Multiple GPU Models If the Nodes in your cluster are installed with GPUs of different models, you can use Node Affinity to schedule Pods to Nodes with a specific GPU model. First, label your Nodes with the GPU model during cluster initialization: ```bash # Label your nodes with the accelerator type they have. kubectl label nodes accelerator=nvidia-tesla-k80 kubectl label nodes accelerator=nvidia-tesla-p100 ``` Then, set Node Affinity when creating a Pod: ```yaml apiVersion: v1 kind: Pod metadata: name: cuda-vector-add spec: restartPolicy: OnFailure containers: - name: cuda-vector-add # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile image: "k8s.gcr.io/cuda-vector-add:v0.1" resources: limits: nvidia.com/gpu: 1 nodeSelector: accelerator: nvidia-tesla-p100 # or nvidia-tesla-k80 etc. ``` ## Using CUDA Libraries NVIDIA Cuda Toolkit and cuDNN must be pre-installed on all Nodes. To access `/usr/lib/nvidia-375`, CUDA libraries should be passed to containers as hostPath volumes: ```yaml apiVersion: batch/v1 kind: Job metadata: name: nvidia-smi labels: name: nvidia-smi spec: template: metadata: labels: name: nvidia-smi spec: containers: - name: nvidia-smi image: nvidia/cuda command: ["nvidia-smi"] imagePullPolicy: IfNotPresent resources: limits: alpha.kubernetes.io/nvidia-gpu: 1 volumeMounts: - mountPath: /usr/local/nvidia/bin name: bin - mountPath: /usr/lib/nvidia name: lib volumes: - name: bin hostPath: path: /usr/lib/nvidia-375/bin - name: lib hostPath: path: /usr/lib/nvidia-375 restartPolicy: Never ``` ```bash $ kubectl create -f job.yaml job "nvidia-smi" created $ kubectl get job NAME DESIRED SUCCESSFUL AGE nvidia-smi 1 1 14m $ kubectl get pod -a NAME READY STATUS RESTARTS AGE nvidia-smi-kwd2m 0/1 Completed 0 14m $ kubectl logs nvidia-smi-kwd2m Fri Jun 16 19:49:53 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 375.66 Driver Version: 375.66 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 Off | 0000:00:04.0 Off | 0 | | N/A 74C P0 80W / 149W | 0MiB / 11439MiB | 100% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` ## Appendix: Installing CUDA To install CUDA: ```bash # Check for CUDA and try to install. if ! dpkg-query -W cuda; then # The 16.04 installer works with 16.10. curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb apt-get update apt-get install cuda -y fi ``` To install cuDNN: First, visit the website , register and download cuDNN v5.1, then use the following commands to install it: ```bash tar zxvf cudnn-8.0-linux-x64-v5.1.tgz ln -s /usr/local/cuda-8.0 /usr/local/cuda sudo cp -P cuda/include/cudnn.h /usr/local/cuda/include sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64 sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn* ``` After installation, you can run nvidia-smi to check the status of the GPU devices: ```bash $ nvidia-smi Fri Jun 16 19:33:35 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 375.66 Driver Version: 375.66 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 Off | 0000:00:04.0 Off | 0 | | N/A 74C P0 80W / 149W | 0MiB / 11439MiB | 100% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` ## Reference Documents * [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin) * [Schedule GPUs on Kubernetes](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/) * [GoogleCloudPlatform/container-engine-accelerators](https://github.com/GoogleCloudPlatform/container-engine-accelerators) # Cluster Autoscaler ## Cluster Autoscaler The Cluster AutoScaler is an extension that automatically scales a Kubernetes cluster's Nodes up and down. When cluster capacity is insufficient, it automatically creates new Nodes with Cloud Providers (supporting GCE, GKE, Azure, AKS, AWS, etc.), and automatically deletes Nodes with low resource utilization (below 50%) for an extended period (over 10 minutes) to save on costs. Cluster AutoScaler is maintained independently from the Kubernetes main code base at . ### Deployment Cluster AutoScaler v1.0+ can be deployed using the Docker image `gcr.io/google_containers/cluster-autoscaler:v1.3.0`. Detailed deployment steps can be found at: * GCE: * GKE: * AWS: * Azure: Note that in clusters with RBAC enabled, a [cluster-autoscaler ClusterRole](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/rbac/cluster-autoscaler/cluster-autoscaler-rbac.yaml) should be created. ### How It Works Cluster AutoScaler periodically (default interval is 10s) checks whether there are enough resources to schedule newly created Pods, and when resources are insufficient, it calls the Cloud Provider to create new Nodes. ![](/files/Ev7ERdqw8HQBdqYvgafj) To automatically create and initialize Nodes, Cluster Autoscaler requires that Nodes must belong to a Node Group, such as: * Managed instance groups (MIG) in GCE/GKE * Autoscaling Groups in AWS * Scale Sets and Availability Sets in Azure When there are multiple Node Groups in a cluster, the Node Group selection strategy can be configured using the `--expander=