Cilium

Cilium is an open-source high-performance networking solution for containers based on eBPF and XDP. The source code is available on https://github.com/cilium/cilium. Its main features include:

• Security-wise, it supports L3/L4/L7 security policies which can be categorized according to their methodology into:

  • Security identity-based security policies

  • CIDR-based security policies

  • Label-based security policies

• On the networking front, it supports a flat layer 3 network, such as:

  • Overlay networks, including VXLAN and Geneve, among others.

  • Linux routing networks, which encompass the native Linux routing and advanced network routing by cloud providers, etc.

• Provides BPF-based load balancing

• Offers convenient monitoring and troubleshooting capabilities

eBPF and XDP

eBPF (extended Berkeley Packet Filter) evolved from BPF and provides a packet filtering mechanism inside the kernel. The basic idea of BPF is to give the user two SOCKET options: SO_ATTACH_FILTER and SO_ATTACH_BPF, allowing the addition of custom filters to sockets, where only the packets that meet the specified filter conditions are sent up to user space. SO_ATTACH_FILTER inserts cBPF code, while SO_ATTACH_BPF deals with eBPF code. eBPF is an enhancement over cBPF, and network utility tools like tcpdump still use the cBPF version; these are automatically converted to eBPF by the kernel when loaded. Linux kernel version 3.15 introduced eBPF, which expanded the capabilities of BPF and enriched the instruction set. It provides a virtual machine within the kernel where user-space can pass filtering rules in the form of virtual machine instructions, which the kernel then uses to filter network packets.

XDP (eXpress Data Path) delivers a high-performance, programmable network data path for the Linux kernel. Since it's handling network packets before they enter the network stack, it tremendously boosts the performance of Linux networking. XDP seems similar to DPDK, but it has several advantages over DPDK, such as:

• No dependency on third-party libraries and licenses

• Supports both poll-mode and interrupt-mode networking

• No need to allocate large memory pages

• No dedicated CPU cores required

• No need for a new security model

Of course, the performance boost with XDP comes at a cost; it sacrifices generality and fairness: (1) it does not provide queuing disciplines (qdisc), and in the event of a slower TX device, packets are dropped, so XDP should not be used when RX is faster than TX; (2) XDP programs are specialized and lack the generality of the network protocol stack.

Deployment

System requirements:

• Linux Kernel >= 4.8 (4.9.17 LTS recommended)

• KV storage (etcd >= 3.1.0 or consul >= 0.6.4)

Kubernetes Cluster

# mount BPF filesystem on all nodes
$ mount bpffs /sys/fs/bpf -t bpf

$ wget https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/1.10/cilium.yaml
$ vim cilium.yaml
[adjust the etcd address]

$ kubectl create -f ./cilium.yaml

minikube

minikube start --network-plugin=cni --bootstrapper=localkube --memory=4096 --extra-config=apiserver.Authorization.Mode=RBAC
kubectl create clusterrolebinding kube-system-default-binding-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/addons/etcd/standalone-etcd.yaml
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.10/cilium.yaml

Istio

# cluster clusterrolebindings
kubectl create clusterrolebinding kube-system-default-binding-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
# etcd
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/addons/etcd/standalone-etcd.yaml

# cilium
curl -s https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.10/cilium.yaml | \
  sed -e 's/sidecar-http-proxy: "false"/sidecar-http-proxy: "true"/' | \
  kubectl create -f -

# Istio
curl -L https://git.io/getLatestIstio | sh -
ISTIO_VERSION=$(curl -L -s https://api.github.com/repos/istio/istio/releases/latest | jq -r .tag_name)
cd istio-${ISTIO_VERSION}
cp bin/istioctl /usr/local/bin

# Patch with cilium pilot
sed -e 's,docker\.io/istio/pilot:,docker.io/cilium/istio_pilot:,' \
      < install/kubernetes/istio.yaml | \
      kubectl create -f -

# Configure Istio’s sidecar injection to use Cilium’s Docker images for the sidecar proxies
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes-istio/istio-sidecar-injector-configmap-release.yaml

Security Policies

TCP policy:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "L3-L4 policy to restrict deathstar access to empire ships only"
metadata:
  name: "rule1"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP

CIDR policy:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "cidr-rule"
spec:
  endpointSelector:
    matchLabels:
      app: myService
  egress:
  - toCIDR:
    - 20.1.1.1/32
  - toCIDRSet:
    - cidr: 10.0.0.0/8
      except:
      - 10.96.0.0/12

L7 HTTP policy:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "L7 policy to restrict access to specific HTTP call"
metadata:
  name: "rule1"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/v1/request-landing"

Monitoring

microscope aggregates monitoring data from all nodes (obtained via cilium monitor). Usage is as follows:

$ kubectl apply -f
https://github.com/cilium/microscope/blob/master/docs/microscope.yaml
$ kubectl exec -n kube-system microscope -- microscope -h

References

• Cilium documentation