CustomResourceDefinition(CRD)是 v1.7 新增的无需改变代码就可以扩展 Kubernetes API 的机制,用来管理自定义对象。它实际上是 ThirdPartyResources(TPR)的升级版本,而 TPR 已经在 v1.8 中弃用。
Kubernetes 版本 | CRD API 版本 |
v1.8+ | apiextensions.k8s.io/v1beta1 |
下面的例子会创建一个 /apis/stable.example.com/v1/namespaces/<namespace>/crontabs/…
的自定义 API:
apiVersion: apiextensions.k8s.io/v1beta1kind: CustomResourceDefinitionmetadata:# name must match the spec fields below, and be in the form: <plural>.<group>name: crontabs.stable.example.comspec:# group name to use for REST API: /apis/<group>/<version>group: stable.example.com# versions to use for REST API: /apis/<group>/<version>versions:- name: v1beta1# Each version can be enabled/disabled by Served flag.served: true# One and only one version must be marked as the storage version.storage: true- name: v1served: truestorage: false# either Namespaced or Clusterscope: Namespacednames:# plural name to be used in the URL: /apis/<group>/<version>/<plural>plural: crontabs# singular name to be used as an alias on the CLI and for displaysingular: crontab# kind is normally the CamelCased singular type. Your resource manifests use this.kind: CronTab# shortNames allow shorter string to match your resource on the CLIshortNames:- ct
API 创建好后,就可以创建具体的 CronTab 对象了
$ cat my-cronjob.yamlapiVersion: "stable.example.com/v1"kind: CronTabmetadata:name: my-new-cron-objectspec:cronSpec: "* * * * /5"image: my-awesome-cron-image$ kubectl create -f my-crontab.yamlcrontab "my-new-cron-object" created$ kubectl get crontabNAME KINDmy-new-cron-object CronTab.v1.stable.example.com$ kubectl get crontab my-new-cron-object -o yamlapiVersion: stable.example.com/v1kind: CronTabmetadata:creationTimestamp: 2017-07-03T19:00:56Zname: my-new-cron-objectnamespace: defaultresourceVersion: "20630"selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-objectuid: 5c82083e-5fbd-11e7-a204-42010a8c0002spec:cronSpec: '* * * * /5'image: my-awesome-cron-image
Finalizer 用于实现控制器的异步预删除钩子,可以通过 metadata.finalizers
来指定 Finalizer。
apiVersion: "stable.example.com/v1"kind: CronTabmetadata:finalizers:- finalizer.stable.example.com
Finalizer 指定后,客户端删除对象的操作只会设置 metadata.deletionTimestamp
而不是直接删除。这会触发正在监听 CRD 的控制器,控制器执行一些删除前的清理操作,从列表中删除自己的 finalizer,然后再重新发起一个删除操作。此时,被删除的对象才会真正删除。
v1.8 开始新增了实验性的基于 OpenAPI v3 schema 的验证(Validation)机制,可以用来提前验证用户提交的资源是否符合规范。使用该功能需要配置 kube-apiserver 的 --feature-gates=CustomResourceValidation=true
。
比如下面的 CRD 要求
spec.cronSpec
必须是匹配正则表达式的字符串
spec.replicas
必须是从 1 到 10 的整数
apiVersion: apiextensions.k8s.io/v1beta1kind: CustomResourceDefinitionmetadata:name: crontabs.stable.example.comspec:group: stable.example.comversion: v1scope: Namespacednames:plural: crontabssingular: crontabkind: CronTabshortNames:- ctvalidation:# openAPIV3Schema is the schema for validating custom objects.openAPIV3Schema:properties:spec:properties:cronSpec:type: stringpattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'replicas:type: integerminimum: 1maximum: 10
这样,在创建下面的 CronTab 时
apiVersion: "stable.example.com/v1"kind: CronTabmetadata:name: my-new-cron-objectspec:cronSpec: "* * * *"image: my-awesome-cron-imagereplicas: 15
会报验证失败的错误:
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"","clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:validation failure list:spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'spec.replicas in body should be less than or equal to 10
v1.10 开始 CRD 还支持 /status
和 /scale
等两个子资源(Beta),并且从 v1.11 开始默认开启。
v1.10 版本使用前需要在
kube-apiserver
开启--feature-gates=CustomResourceSubresources=true
。
# resourcedefinition.yamlapiVersion: apiextensions.k8s.io/v1beta1kind: CustomResourceDefinitionmetadata:name: crontabs.stable.example.comspec:group: stable.example.comversion: v1scope: Namespacednames:plural: crontabssingular: crontabkind: CronTabshortNames:- ct# subresources describes the subresources for custom resources.subresources:# status enables the status subresource.status: {}# scale enables the scale subresource.scale:# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.specReplicasPath: .spec.replicas# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.statusReplicasPath: .status.replicas# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.labelSelectorPath: .status.labelSelector
$ kubectl create -f resourcedefinition.yaml$ kubectl create -f- <<EOFapiVersion: "stable.example.com/v1"kind: CronTabmetadata:name: my-new-cron-objectspec:cronSpec: "* * * * */5"image: my-awesome-cron-imagereplicas: 3EOF$ kubectl scale --replicas=5 crontabs/my-new-cron-objectcrontabs "my-new-cron-object" scaled$ kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'5
Categories 用来将 CRD 对象分组,这样就可以使用 kubectl get <category-name>
来查询属于该组的所有对象。
# resourcedefinition.yamlapiVersion: apiextensions.k8s.io/v1beta1kind: CustomResourceDefinitionmetadata:name: crontabs.stable.example.comspec:group: stable.example.comversion: v1scope: Namespacednames:plural: crontabssingular: crontabkind: CronTabshortNames:- ct# categories is a list of grouped resources the custom resource belongs to.categories:- all
# my-crontab.yamlapiVersion: "stable.example.com/v1"kind: CronTabmetadata:name: my-new-cron-objectspec:cronSpec: "* * * * */5"image: my-awesome-cron-image
$ kubectl create -f resourcedefinition.yaml$ kubectl create -f my-crontab.yaml$ kubectl get allNAME AGEcrontabs/my-new-cron-object 3s
在使用 CRD 扩展 Kubernetes API 时,通常还需要实现一个新建资源的控制器,监听新资源的变化情况,并作进一步的处理。
https://github.com/kubernetes/sample-controller 提供了一个 CRD 控制器的示例,包括
如何注册资源 Foo
如何创建、删除和查询 Foo
对象
如何监听 Foo
资源对象的变化情况
从上面的实例中可以看到从头构建一个 CRD 控制器并不容易,需要对 Kubernetes 的 API 有深入了解,并且RBAC 集成、镜像构建、持续集成和部署等都需要很大工作量。
kubebuilder 正是为解决这个问题而生,为 CRD 控制器提供了一个简单易用的框架,并可直接生成镜像构建、持续集成、持续部署等所需的资源文件。
# Install kubebuilderVERSION=1.0.1wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v${VERSION}/kubebuilder_${VERSION}_linux_amd64.tar.gztar zxvf kubebuilder_${VERSION}_linux_amd64.tar.gzsudo mv kubebuilder_${VERSION}_linux_amd64 /usr/local/kubebuilderexport PATH=$PATH:/usr/local/kubebuilder/bin# Install dep kustomizego get -u github.com/golang/dep/cmd/depgo get github.com/kubernetes-sigs/kustomize
mkdir -p $GOPATH/src/democd $GOPATH/src/demokubebuilder init --domain k8s.io --license apache2 --owner "The Kubernetes Authors"
kubebuilder create api --group ships --version v1beta1 --kind Sloop
然后按照实际需要修改 pkg/apis/ship/v1beta1/sloop_types.go
和 pkg/controller/sloop/sloop_controller.go
增加业务逻辑。
make installmake run
如果碰到错误
ValidationError(CustomResourceDefinition.status): missing required field "storedVersions" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionStatus]
,可以手动修改config/crds/ships_v1beta1_sloop.yaml
:```yaml status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: []
然后运行
kubectl apply -f config/crds
创建 CRD。
然后就可以用 ships.k8s.io/v1beta1
来创建 Kind 为 Sloop
的资源了,比如
kubectl apply -f config/samples/ships_v1beta1_sloop.yaml
# 替换 IMG 为你自己的export IMG=feisky/demo-crd:v1make docker-buildmake docker-pushmake deploy
kustomize 已经不再支持通配符,因而上述
make deploy
可能会碰到Load from path ../rbac/*.yaml failed
错误,解决方法是手动修改config/default/kustomization.yaml
:resources:
../rbac/rbac_role.yaml
../rbac/rbac_role_binding.yaml
../manager/manager.yaml
然后执行
kustomize build config/default | kubectl apply -f -
部署,默认部署到demo-system
namespace 中。
# run unit testsmake test# generate docskubebuilder docs