CustomResourceDefinition
CustomResourceDefinition(CRD)是 v1.7 新增的无需改变代码就可以扩展 Kubernetes API 的机制,用来管理自定义对象。它实际上是 ThirdPartyResources(TPR)的升级版本,而 TPR 已经在 v1.8 中弃用。

API 版本对照表

Kubernetes 版本
CRD API 版本
v1.8+
apiextensions.k8s.io/v1beta1

CRD 示例

下面的例子会创建一个 /apis/stable.example.com/v1/namespaces/<namespace>/crontabs/… 的自定义 API:
1
apiVersion: apiextensions.k8s.io/v1beta1
2
kind: CustomResourceDefinition
3
metadata:
4
# name must match the spec fields below, and be in the form: <plural>.<group>
5
name: crontabs.stable.example.com
6
spec:
7
# group name to use for REST API: /apis/<group>/<version>
8
group: stable.example.com
9
# versions to use for REST API: /apis/<group>/<version>
10
versions:
11
- name: v1beta1
12
# Each version can be enabled/disabled by Served flag.
13
served: true
14
# One and only one version must be marked as the storage version.
15
storage: true
16
- name: v1
17
served: true
18
storage: false
19
# either Namespaced or Cluster
20
scope: Namespaced
21
names:
22
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
23
plural: crontabs
24
# singular name to be used as an alias on the CLI and for display
25
singular: crontab
26
# kind is normally the CamelCased singular type. Your resource manifests use this.
27
kind: CronTab
28
# shortNames allow shorter string to match your resource on the CLI
29
shortNames:
30
- ct
Copied!
API 创建好后,就可以创建具体的 CronTab 对象了
1
$ cat my-cronjob.yaml
2
apiVersion: "stable.example.com/v1"
3
kind: CronTab
4
metadata:
5
name: my-new-cron-object
6
spec:
7
cronSpec: "* * * * /5"
8
image: my-awesome-cron-image
9
10
$ kubectl create -f my-crontab.yaml
11
crontab "my-new-cron-object" created
12
13
$ kubectl get crontab
14
NAME KIND
15
my-new-cron-object CronTab.v1.stable.example.com
16
$ kubectl get crontab my-new-cron-object -o yaml
17
apiVersion: stable.example.com/v1
18
kind: CronTab
19
metadata:
20
creationTimestamp: 2017-07-03T19:00:56Z
21
name: my-new-cron-object
22
namespace: default
23
resourceVersion: "20630"
24
selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
25
uid: 5c82083e-5fbd-11e7-a204-42010a8c0002
26
spec:
27
cronSpec: '* * * * /5'
28
image: my-awesome-cron-image
Copied!

Finalizer

Finalizer 用于实现控制器的异步预删除钩子,可以通过 metadata.finalizers 来指定 Finalizer。
1
apiVersion: "stable.example.com/v1"
2
kind: CronTab
3
metadata:
4
finalizers:
5
- finalizer.stable.example.com
Copied!
Finalizer 指定后,客户端删除对象的操作只会设置 metadata.deletionTimestamp 而不是直接删除。这会触发正在监听 CRD 的控制器,控制器执行一些删除前的清理操作,从列表中删除自己的 finalizer,然后再重新发起一个删除操作。此时,被删除的对象才会真正删除。

Validation

v1.8 开始新增了实验性的基于 OpenAPI v3 schema 的验证(Validation)机制,可以用来提前验证用户提交的资源是否符合规范。使用该功能需要配置 kube-apiserver 的 --feature-gates=CustomResourceValidation=true
比如下面的 CRD 要求
    spec.cronSpec 必须是匹配正则表达式的字符串
    spec.replicas 必须是从 1 到 10 的整数
1
apiVersion: apiextensions.k8s.io/v1beta1
2
kind: CustomResourceDefinition
3
metadata:
4
name: crontabs.stable.example.com
5
spec:
6
group: stable.example.com
7
version: v1
8
scope: Namespaced
9
names:
10
plural: crontabs
11
singular: crontab
12
kind: CronTab
13
shortNames:
14
- ct
15
validation:
16
# openAPIV3Schema is the schema for validating custom objects.
17
openAPIV3Schema:
18
properties:
19
spec:
20
properties:
21
cronSpec:
22
type: string
23
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}#x27;
24
replicas:
25
type: integer
26
minimum: 1
27
maximum: 10
Copied!
这样,在创建下面的 CronTab 时
1
apiVersion: "stable.example.com/v1"
2
kind: CronTab
3
metadata:
4
name: my-new-cron-object
5
spec:
6
cronSpec: "* * * *"
7
image: my-awesome-cron-image
8
replicas: 15
Copied!
会报验证失败的错误:
1
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"","clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
2
validation failure list:
3
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}#x27;
4
spec.replicas in body should be less than or equal to 10
Copied!

Subresources

v1.10 开始 CRD 还支持 /status/scale 等两个子资源(Beta),并且从 v1.11 开始默认开启。
v1.10 版本使用前需要在 kube-apiserver 开启 --feature-gates=CustomResourceSubresources=true
1
# resourcedefinition.yaml
2
apiVersion: apiextensions.k8s.io/v1beta1
3
kind: CustomResourceDefinition
4
metadata:
5
name: crontabs.stable.example.com
6
spec:
7
group: stable.example.com
8
version: v1
9
scope: Namespaced
10
names:
11
plural: crontabs
12
singular: crontab
13
kind: CronTab
14
shortNames:
15
- ct
16
# subresources describes the subresources for custom resources.
17
subresources:
18
# status enables the status subresource.
19
status: {}
20
# scale enables the scale subresource.
21
scale:
22
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
23
specReplicasPath: .spec.replicas
24
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
25
statusReplicasPath: .status.replicas
26
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
27
labelSelectorPath: .status.labelSelector
Copied!
1
$ kubectl create -f resourcedefinition.yaml
2
$ kubectl create -f- <<EOF
3
apiVersion: "stable.example.com/v1"
4
kind: CronTab
5
metadata:
6
name: my-new-cron-object
7
spec:
8
cronSpec: "* * * * */5"
9
image: my-awesome-cron-image
10
replicas: 3
11
EOF
12
13
$ kubectl scale --replicas=5 crontabs/my-new-cron-object
14
crontabs "my-new-cron-object" scaled
15
16
$ kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
17
5
Copied!

Categories

Categories 用来将 CRD 对象分组,这样就可以使用 kubectl get <category-name> 来查询属于该组的所有对象。
1
# resourcedefinition.yaml
2
apiVersion: apiextensions.k8s.io/v1beta1
3
kind: CustomResourceDefinition
4
metadata:
5
name: crontabs.stable.example.com
6
spec:
7
group: stable.example.com
8
version: v1
9
scope: Namespaced
10
names:
11
plural: crontabs
12
singular: crontab
13
kind: CronTab
14
shortNames:
15
- ct
16
# categories is a list of grouped resources the custom resource belongs to.
17
categories:
18
- all
Copied!
1
# my-crontab.yaml
2
apiVersion: "stable.example.com/v1"
3
kind: CronTab
4
metadata:
5
name: my-new-cron-object
6
spec:
7
cronSpec: "* * * * */5"
8
image: my-awesome-cron-image
Copied!
1
$ kubectl create -f resourcedefinition.yaml
2
$ kubectl create -f my-crontab.yaml
3
$ kubectl get all
4
NAME AGE
5
crontabs/my-new-cron-object 3s
Copied!

CRD 控制器

在使用 CRD 扩展 Kubernetes API 时,通常还需要实现一个新建资源的控制器,监听新资源的变化情况,并作进一步的处理。
https://github.com/kubernetes/sample-controller 提供了一个 CRD 控制器的示例,包括
    如何注册资源 Foo
    如何创建、删除和查询 Foo 对象
    如何监听 Foo 资源对象的变化情况

Kubebuilder

从上面的实例中可以看到从头构建一个 CRD 控制器并不容易,需要对 Kubernetes 的 API 有深入了解,并且RBAC 集成、镜像构建、持续集成和部署等都需要很大工作量。
kubebuilder 正是为解决这个问题而生,为 CRD 控制器提供了一个简单易用的框架,并可直接生成镜像构建、持续集成、持续部署等所需的资源文件。

安装

1
# Install kubebuilder
2
VERSION=1.0.1
3
wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v${VERSION}/kubebuilder_${VERSION}_linux_amd64.tar.gz
4
tar zxvf kubebuilder_${VERSION}_linux_amd64.tar.gz
5
sudo mv kubebuilder_${VERSION}_linux_amd64 /usr/local/kubebuilder
6
export PATH=$PATH:/usr/local/kubebuilder/bin
7
8
# Install dep kustomize
9
go get -u github.com/golang/dep/cmd/dep
10
go get github.com/kubernetes-sigs/kustomize
Copied!

使用方法

初始化项目

1
mkdir -p $GOPATH/src/demo
2
cd $GOPATH/src/demo
3
kubebuilder init --domain k8s.io --license apache2 --owner "The Kubernetes Authors"
Copied!

创建 API

1
kubebuilder create api --group ships --version v1beta1 --kind Sloop
Copied!
然后按照实际需要修改 pkg/apis/ship/v1beta1/sloop_types.gopkg/controller/sloop/sloop_controller.go 增加业务逻辑。

本地运行测试

1
make install
2
make run
Copied!
如果碰到错误 ValidationError(CustomResourceDefinition.status): missing required field "storedVersions" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionStatus],可以手动修改 config/crds/ships_v1beta1_sloop.yaml:
```yaml status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: []
然后运行 kubectl apply -f config/crds 创建 CRD。
然后就可以用 ships.k8s.io/v1beta1 来创建 Kind 为 Sloop 的资源了,比如
1
kubectl apply -f config/samples/ships_v1beta1_sloop.yaml
Copied!

构建镜像并部署控制器

1
# 替换 IMG 为你自己的
2
export IMG=feisky/demo-crd:v1
3
make docker-build
4
make docker-push
5
make deploy
Copied!
kustomize 已经不再支持通配符,因而上述 make deploy 可能会碰到 Load from path ../rbac/*.yaml failed 错误,解决方法是手动修改 config/default/kustomization.yaml:
resources:
    ../rbac/rbac_role.yaml
    ../rbac/rbac_role_binding.yaml
    ../manager/manager.yaml
然后执行 kustomize build config/default | kubectl apply -f - 部署,默认部署到 demo-system namespace 中。

文档和测试

1
# run unit tests
2
make test
3
4
# generate docs
5
kubebuilder docs
Copied!

参考文档

最近更新 1mo ago