CustomResourceDefinition
CustomResourceDefinition (CRD) is an ingenious mechanism introduced in v1.7 that allows you to extend the Kubernetes API without tinkering with code to manage custom objects. Practically speaking, it's an upgraded version of ThirdPartyResources (TPR), which was deprecated in v1.8.
API Version Comparison Table
v1.8+
apiextensions.k8s.io/v1beta1
CRD Example
The example below crafts a custom API: /apis/stable.example.com/v1/namespaces/<namespace>/crontabs/…
.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# versions to use for REST API: /apis/<group>/<version>
versions:
- name: v1beta1
# Each version can be enabled/disabled by the Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
- name: v1
served: true
storage: false
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow a shorter string to match your resource on the CLI
shortNames:
- ct
Once the API is set up, you can proceed to create specific CronTab objects.
$ cat my-cronjob.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * /5"
image: my-awesome-cron-image
$ kubectl create -f my-crontab.yaml
crontab "my-new-cron-object" created
$ kubectl get crontab
NAME KIND
my-new-cron-object CronTab.v1.stable.example.com
$ kubectl get crontab my-new-cron-object -o yaml
apiVersion: stable.example.com/v1
kind: CronTab
metadata:
creationTimestamp: 2017-07-03T19:00:56Z
name: my-new-cron-object
namespace: default
resourceVersion: "20630"
selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
uid: 5c82083e-5fbd-11e7-a204-42010a8c0002
spec:
cronSpec: '* * * * /5'
image: my-awesome-cron-image
Finalizer
Finalizers are used to implement asynchronous pre-deletion hooks for controllers, which can be specified via metadata.finalizers
.
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
finalizers:
- finalizer.stable.example.com
Once the finalizer is indicated, the operation to delete an object by a client will merely set metadata.deletionTimestamp
instead of performing a direct deletion. This triggers controllers that are listening to the CRD to perform cleanup operations before deletion, remove its own finalizer from the list, and then initiate a new deletion operation. Only then is the object to be deleted truly eliminated.
Validation
Starting from v1.8, an experimental validation mechanism based on OpenAPI v3 schema was added to validate the conformity of resources submitted by users in advance. To use this function, you need to configure the kube-apiserver's --feature-gates=CustomResourceValidation=true
.
For instance, the CRD below necessitates:
spec.cronSpec
to be a string that matches a regular expressionspec.replicas
to be an integer between 1 and 10
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
version: v1
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
validation:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
properties:
spec:
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
replicas:
type: integer
minimum: 1
maximum: 10
For example, when creating the following CronTab:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * *"
image: my-awesome-cron-image
replicas: 15
You'll encounter a validation failure error:
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"","clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10
Subresources
From v1.10 onwards, CRD also supports two subresources /status
and /scale
in the beta version, and they are enabled by default from v1.11.
To use in v1.10, you need to enable
--feature-gates=CustomResourceSubresources=true
on thekube-apiserver
.
# resourcedefinition.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
version: v1
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
# subresources describes the subresources for custom resources.
subresources:
# status enables the status subresource.
status: {}
# scale enables the scale subresource.
scale:
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
specReplicasPath: .spec.replicas
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
statusReplicasPath: .status.replicas
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
labelSelectorPath: .status.labelSelector
$ kubectl create -f resourcedefinition.yaml
$ kubectl create -f- <<EOF
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
replicas: 3
EOF
$ kubectl scale --replicas=5 crontabs/my-new-cron-object
crontabs "my-new-cron-object" scaled
$ kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
5
Categories
Categories are used to group CRD objects, making it possible to query all objects belonging to that group with kubectl get <category-name>
.
# resourcedefinition.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
version: v1
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
# categories is a list of grouped resources the custom resource belongs to.
categories:
- all
# my-crontab.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
$ kubectl create -f resourcedefinition.yaml
$ kubectl create -f my-crontab.yaml
$ kubectl get all
NAME AGE
crontabs/my-new-cron-object 3s
CRD Controllers
When extending the Kubernetes API using CRD, it's generally also necessary to implement a controller for the new resources to monitor their changes and make further processing.
https://github.com/kubernetes/sample-controller provides an example of a CRD controller, including
How to register
Foo
resourcesHow to create, delete, and query
Foo
objectsHow to monitor changes of
Foo
resources
Kubebuilder
As demonstrated above, building a CRD controller from scratch is quite challenging considering the level of understanding required for Kubernetes's API. Integrating RBAC, constructing images, and continuous integration and deployment all demand a large amount of work.
kubebuilder exists to solve this issue, providing an easy-to-use framework for creating CRD controllers and directly generating the resource files needed for image building, continuous integration, and deployment.
Installation
# Install kubebuilder
VERSION=1.0.1
wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v${VERSION}/kubebuilder_${VERSION}_linux_amd64.tar.gz
tar zxvf kubebuilder_${VERSION}_linux_amd64.tar.gz
sudo mv kubebuilder_${VERSION}_linux_amd64 /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin
# Install dep kustomize
go get -u github.com/golang/dep/cmd/dep
go get github.com/kubernetes-sigs/kustomize
How to Use
Initialize the Project
mkdir -p $GOPATH/src/demo
cd $GOPATH/src/demo
kubebuilder init --domain k8s.io --license apache2 --owner "The Kubernetes Authors"
Create API
kubebuilder create api --group ships --version v1beta1 --kind Sloop
Then, depending on your actual needs, modify pkg/apis/ship/v1beta1/sloop_types.go
and pkg/controller/sloop/sloop_controller.go
to add business logic.
Run Local Test
make install
make run
If you run into the error
ValidationError(CustomResourceDefinition.status): missing required field "storedVersions" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionStatus]
, manually modifyconfig/crds/ships_v1beta1_sloop.yaml
:```yaml status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: []
Then run
kubectl apply -f config/crds
to create the CRD.
You can then create resources with Kind as Sloop
using ships.k8s.io/v1beta1
, such as
kubectl apply -f config/samples/ships_v1beta1_sloop.yaml
Build Image and Deploy Controller
# Replace IMG with your own
export IMG=feisky/demo-crd:v1
make docker-build
make docker-push
make deploy
kustomize no longer supports wildcards, so the above
make deploy
may encounter aLoad from path ../rbac/*.yaml failed
error. The solution is to manually modifyconfig/default/kustomization.yaml
:resources:
../rbac/rbac_role.yaml
../rbac/rbac_role_binding.yaml
../manager/manager.yaml
Then execute
kustomize build config/default | kubectl apply -f -
to deploy. By default, it's deployed to thedemo-system
namespace.
Documentation and Testing
# run unit tests
make test
# generate docs
kubebuilder docs
References
最后更新于