Kubernetes指南
Linux性能优化实战eBPF 核心技术与实战SDN指南个人博客
EN
EN
  • Overview
  • Introduction
    • Kubernetes Introduction
    • Kubernetes Concepts
    • Kubernetes 101
    • Kubernetes 201
    • Kubernetes Cluster
  • Concepts
    • Concepts
    • Architecture
    • Design Principles
    • Components
      • etcd
      • kube-apiserver
      • kube-scheduler
      • kube-controller-manager
      • kubelet
      • kube-proxy
      • kube-dns
      • Federation
      • kubeadm
      • hyperkube
      • kubectl
    • Objects
      • Autoscaling
      • ConfigMap
      • CronJob
      • CustomResourceDefinition
      • DaemonSet
      • Deployment
      • Ingress
      • Job
      • LocalVolume
      • Namespace
      • NetworkPolicy
      • Node
      • PersistentVolume
      • Pod
      • PodPreset
      • ReplicaSet
      • Resource Quota
      • Secret
      • SecurityContext
      • Service
      • ServiceAccount
      • StatefulSet
      • Volume
  • Setup
    • Setup Guidance
    • kubectl Install
    • Single Machine
    • Feature Gates
    • Best Practice
    • Version Support
    • Setup Cluster
      • kubeadm
      • kops
      • Kubespray
      • Azure
      • Windows
      • LinuxKit
      • kubeasz
    • Setup Addons
      • Addon-manager
      • DNS
      • Dashboard
      • Monitoring
      • Logging
      • Metrics
      • GPU
      • Cluster Autoscaler
      • ip-masq-agent
  • Extension
    • API Extension
      • Aggregation
      • CustomResourceDefinition
    • Access Control
      • Authentication
      • RBAC Authz
      • Admission
    • Scheduler Extension
    • Network Plugin
      • CNI
      • Flannel
      • Calico
      • Weave
      • Cilium
      • OVN
      • Contiv
      • SR-IOV
      • Romana
      • OpenContrail
      • Kuryr
    • Container Runtime
      • CRI-tools
      • Frakti
    • Storage Driver
      • CSI
      • FlexVolume
      • glusterfs
    • Network Policy
    • Ingress Controller
      • Ingress + Letsencrypt
      • minikube Ingress
      • Traefik Ingress
      • Keepalived-VIP
    • Cloud Provider
    • Device Plugin
  • Cloud Native Apps
    • Apps Management
      • Patterns
      • Rolling Update
      • Helm
      • Operator
      • Service Mesh
      • Linkerd
      • Linkerd2
    • Istio
      • Deploy
      • Traffic Management
      • Security
      • Policy
      • Metrics
      • Troubleshooting
      • Community
    • Devops
      • Draft
      • Jenkins X
      • Spinnaker
      • Kompose
      • Skaffold
      • Argo
      • Flux GitOps
  • Practices
    • Overview
    • Resource Management
    • Cluster HA
    • Workload HA
    • Debugging
    • Portmap
    • Portforward
    • User Management
    • GPU
    • HugePage
    • Security
    • Audit
    • Backup
    • Cert Rotation
    • Large Cluster
    • Big Data
      • Spark
      • Tensorflow
    • Serverless
  • Troubleshooting
    • Overview
    • Cluster Troubleshooting
    • Pod Troubleshooting
    • Network Troubleshooting
    • PV Troubleshooting
      • AzureDisk
      • AzureFile
    • Windows Troubleshooting
    • Cloud Platform Troubleshooting
      • Azure
    • Troubleshooting Tools
  • Community
    • Development Guide
    • Unit Test and Integration Test
    • Community Contribution
  • Appendix
    • Ecosystem
    • Learning Resources
    • Domestic Mirrors
    • How to Contribute
    • Reference Documents
由 GitBook 提供支持
在本页
  • Downloads
  • Network Plugins
  • L3 Routing Topology
  • OVS Network Topology
  • Deployment
  • kubeadm
  • Azure
  • Manual Deployment
  • Running Windows Containers
  • Known Issues
  • Secrets and ConfigMaps can only be used as environmental variables
  • Volume Support
  • Image Version Matching
  • Setting CPU and Memory
  • Hyper-V Containers
  • Other Known Issues
  • Appendix: Docker EE Installation Method
  • Reference Documents
  1. Setup
  2. Setup Cluster

Windows

上一页Azure下一页LinuxKit

最后更新于1年前

Beginning with v1.5, Kubernetes started to introduce support for Windows nodes in its alpha version, upgrading this support to beta with the release of v1.9. Some of the key features for Windows containers are include:

  • Pod-level support for Windows containers (isolation=process)

  • Kernel load balancing based on Virtual Filtering Platform (VFP) Hyper-v Switch Extension

  • Windows container management via Container Runtime Interface (CRI)

  • Support for the use of the kubeadm command to add Windows nodes to an existing cluster

  • Recommended use of Windows Server Version 1803+ and Docker Version 17.06+

Note:

  1. Control plane services still run on Linux servers, with only Kubelet, Kube-proxy, Docker, and network plugin services running on Windows nodes.

  2. Windows Server 1803 is recommended since it fixes issues with Windows container symlinks, allowing ServiceAccount and ConfigMap to function normally.

Downloads

You can download the released binary files for Windows servers from . For instance,

wget https://dl.k8s.io/v1.15.0/kubernetes-node-windows-amd64.tar.gz

Alternatively, you can compile from Kubernetes source code:

go get -u k8s.io/kubernetes
cd $GOPATH/src/k8s.io/kubernetes

# Build the kubelet
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet

# Build the kube-proxy
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy

# You will find the output binaries under the folder _output/local/bin/windows/

Network Plugins

The following network plugins are supported in Windows Server (note that the network plugin on Windows nodes must be the same as on Linux nodes):

  1. Flannel v0.10.0+

  2. Calico v3.0.1+

L3 Routing Topology

Example configuration for the wincni network plugin:

{
  "cniVersion": "0.2.0",
  "name": "l2bridge",
  "type": "wincni.exe",
  "master": "Ethernet",
  "ipam": {
    "environment": "azure",
    "subnet": "10.10.187.64/26",
    "routes": [{
      "GW": "10.10.187.66"
    }]
  },
  "dns": {
    "Nameservers": [
      "11.0.0.10"
    ]
  },
  "AdditionalArgs": [{
      "Name": "EndpointPolicy",
      "Value": {
        "Type": "OutBoundNAT",
        "ExceptionList": [
          "11.0.0.0/8",
          "10.10.0.0/16",
          "10.127.132.128/25"
        ]
      }
    },
    {
      "Name": "EndpointPolicy",
      "Value": {
        "Type": "ROUTE",
        "DestinationPrefix": "11.0.0.0/8",
        "NeedEncap": true
      }
    },
    {
      "Name": "EndpointPolicy",
      "Value": {
        "Type": "ROUTE",
        "DestinationPrefix": "10.127.132.213/32",
        "NeedEncap": true
      }
    }
  ]
}

OVS Network Topology

Deployment

kubeadm

If the master node is deployed via kubeadm, Windows nodes can also be deployed through kubeadm:

kubeadm.exe join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

Azure

First, create a Kubernetes cluster configuration file that includes Windows, named windows.json

{
    "apiVersion": "vlabs",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorVersion": "1.11.1",
            "kubernetesConfig": {
                "networkPolicy": "none",
                "enableAggregatedAPIs": true,
                "enableRbac": true
            }
        },
        "masterProfile": {
            "count": 3,
            "dnsPrefix": "kubernetes-windows",
            "vmSize": "Standard_D2_v3"
        },
        "agentPoolProfiles": [
            {
                "name": "windowspool1",
                "count": 3,
                "vmSize": "Standard_D2_v3",
                "availabilityProfile": "AvailabilitySet",
                "osType": "Windows"
            }
        ],
        "windowsProfile": {
            "adminUsername": "<your-username>",
            "adminPassword": "<your-password>"
        },
        "linuxProfile": {
            "adminUsername": "azure",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "<your-ssh-public-key>"
                    }
                ]
            }
        },
        "servicePrincipalProfile": {
            "clientId": "",
            "secret": ""
        }
    }
}

You can then deploy using acs-engine:

# create a new resource group.
az group create --name myResourceGroup  --location "centralus"

# start deploy the kubernetes
acs-engine deploy --resource-group myResourceGroup --subscription-id <subscription-id> --auto-suffix --api-model windows.json --location centralus --dns-prefix <dns-prefix>

# setup kubectl
export KUBECONFIG="$(pwd)/_output/<name-with-suffix>/kubeconfig/kubeconfig.centralus.json"
kubectl get node

Manual Deployment

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name Docker -ProviderName DockerMsftProvider
Restart-Computer -Force

(2) Download kubelet.exe and kube-proxy.exe based on the earlier download section.

(3) Copy the Node spec file (kube config) from the Master node.

(4) Configure the CNI network plugin and base images.

wget https://github.com/Microsoft/SDN/archive/master.zip -o master.zip
Expand-Archive master.zip -DestinationPath master
mkdir C:/k/
mv master/SDN-master/Kubernetes/windows/* C:/k/
rm -recurse -force master,master.zip
docker pull microsoft/windowsservercore:1709
docker tag microsoft/windowsservercore:1709 microsoft/windowsservercore:latest
cd C:/k/
docker build -t kubeletwin/pause .
./start-kubelet.ps1 -ClusterCidr 192.168.0.0/16
./start-kubeproxy.ps1

Running Windows Containers

To schedule a container on a Windows node, use the NodeSelector beta.kubernetes.io/os: windows, for example:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: iis
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: iis
    spec:
      nodeSelector:
        beta.kubernetes.io/os: windows
      containers:
      - name: iis
        image: microsoft/iis
        resources:
          limits:
            memory: "128Mi"
            cpu: 2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: iis
  name: iis
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: iis
  type: NodePort

Running a DaemonSet:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: my-DaemonSet
  labels:
    app: foo
spec:
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
      - name: foo
        image: microsoft/windowsservercore:1709
      nodeSelector:
        beta.kubernetes.io/os: windows

Known Issues

Secrets and ConfigMaps can only be used as environmental variables

This is a known issue with versions 1709 and earlier and can be fixed by upgrading to version 1803.

Volume Support

Local, emptyDir, hostPath, AzureDisk, AzureFile, and flexvolume are currently the only types of volumes supported by Windows containers. It's important to note that the format for the Volume's path needs to be mountPath: "C:\\etc\\foo" or mountPath: "C:/etc/foo".

apiVersion: v1
kind: Pod
metadata:
  name: hostpath-pod
spec:
  containers:
  - name: hostpath-nano
    image: microsoft/nanoserver:1709
    stdin: true
    tty: true
    volumeMounts:
    - name: blah
      mountPath: "C:\\etc\\foo"
      readOnly: true
  nodeSelector:
    beta.kubernetes.io/os: windows
  volumes:
  - name: blah
    hostPath:
      path: "C:\\AzureData"
apiVersion: v1
kind: Pod
metadata:
  name: empty-dir-pod
spec:
  containers:
  - image: microsoft/nanoserver:1709
    name: empty-dir-nano
    stdin: true
    tty: true
    volumeMounts:
    - mountPath: /cache
      name: cache-volume
    - mountPath: C:/scratch
      name: scratch-volume
  volumes:
  - name: cache-volume
    emptyDir: {}
  - name: scratch-volume
    emptyDir: {}
  nodeSelector:
    beta.kubernetes.io/os: windows

Image Version Matching

In Windows Server version 1709, you must use images with the 1709 tag, such as

  • microsoft/aspnet:4.7.1-windowsservercore-1709

  • microsoft/windowsservercore:1709

  • microsoft/iis:windowsservercore-1709

Likewise, for Windows Server version 1803, you must use images with the 1803 tag. For Windows Server 2016, you need to use images with the ltsc2016 tag, such as microsoft/windowsservercore:ltsc2016.

Setting CPU and Memory

Starting from v1.10, Kubernetes supports setting CPU and memory for Windows containers:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iis
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: iis
    spec:
      containers:
      - name: iis
        image: microsoft/iis
        resources:
          limits:
            memory: "128Mi"
            cpu: 2
        ports:
        - containerPort: 80

Hyper-V Containers

Starting from v1.10, containers with Hyper-V isolation are supported (Alpha). Before use, the kubelet needs to be configured to enable the HyperVContainer feature switch. Then you can specify a container for Hyper-V isolation using Annotation experimental.windows.kubernetes.io/isolation-type=hyperv:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iis
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: iis
      annotations:
        experimental.windows.kubernetes.io/isolation-type: hyperv
    spec:
      containers:
      - name: iis
        image: microsoft/iis
        ports:
        - containerPort: 80

Other Known Issues

  • Only Windows Server 1709 or later versions support running multiple containers in a Pod (only Process isolation is supported)

  • StatefulSet is not currently supported

  • Automatic expansion of Windows Server Container Pods (Horizontal Pod Autoscaling) is not yet supported

  • The OS version of the Windows container needs to match the Host OS version; otherwise, the container will not be able to start

  • When using L3 or Host GW networks, you cannot access Kubernetes Services directly from the Windows Node (there is no such issue when using OVS/OVN)

  • The Weave network plugin is not currently supported

  • Calico network plugin only supports Policy-Only mode

Appendix: Docker EE Installation Method

To install Docker EE stable version:

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider
Restart-Computer -Force

To install Docker EE preview version:

Install-Module DockerProvider
Install-Package -Name Docker -ProviderName DockerProvider -RequiredVersion preview

To upgrade Docker EE version:

# Check the installed version
Get-Package -Name Docker -ProviderName DockerMsftProvider

# Find the current version
Find-Package -Name Docker -ProviderName DockerMsftProvider

# Upgrade Docker EE
Install-Package -Name Docker -ProviderName DockerMsftProvider -Update -Force
Start-Service Docker

Reference Documents

L3 routing network plugins like , where routing is configured in TOR switches, routers, or cloud services

For more network topology modes, please refer to .

On Azure, it is recommended to use for automatic deployment of Master and Windows nodes.

(1) Install Docker on Windows Server by following these instructions:

(5) Use to start kubelet.exe, and use to start kube-proxy.exe

(6) If you are using the Host-Gateway network plugin, you will also need to use to add static routes.

For a detailed step-by-step guide, you can refer to .

On Window Server running on VMWare Fusion kubelet.exe may fail to start (this has been fixed in )

For .NET containers that need to use : as an environment variable, you can replace : in the environment variable with __ (see for reference)

https://github.com/kubernetes/kubernetes/releases
wincni
Azure VNET CNI Plugin
Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay
win-bridge
win-overlay
Windows container network drivers
Install Docker
start-kubelet.ps1
start-kubeproxy.ps1
AddRoutes.ps1
this
#57124
here
Guide for adding Windows Nodes in Kubernetes
Intro to Windows support in Kubernetes
Guide for scheduling Windows containers in Kubernetes
Kubernetes for Windows Walkthroughs
acs-engine