Helm Based Operator Quickstart
This guide walks through an example of building a simple nginx-operator powered by Helm using tools and libraries provided by the Operator SDK.
Create a new project
Use the CLI to create a new Helm-based nginx-operator project:
operator-sdk new nginx-operator --api-version=example.com/v1alpha1 --kind=Nginx --type=helm cd nginx-operator
This creates the nginx-operator project specifically for watching the
Nginx resource with APIVersion
example.com/v1alpha1 and Kind
For Helm-based projects,
operator-sdk new also generates the RBAC rules
deploy/role.yaml based on the resources that would be deployed by the
chart’s default manifest. Be sure to double check that the rules generated
deploy/role.yaml meet the operator’s permission requirements.
To learn more about the project directory structure, see the project layout doc.
Use an existing chart
Instead of creating your project with a boilerplate Helm chart, you can also use
--helm-chart-version to use an existing chart, either from your local filesystem or a remote chart repository.
--helm-chart is specified,
--kind become optional. If left unset, the SDK will default
charts.helm.k8s.io/v1alpha1 and will deduce
--kind from the specified chart.
--helm-chart is a local chart archive or directory, it will be validated and unpacked or copied into the project.
Otherwise, the SDK will attempt to fetch the specified helm chart from a remote repository.
If a custom repository URL is not specified by
--helm-chart-repo, the following chart reference formats are supported:
<repoName>/<chartName>: Fetch the helm chart named
chartNamefrom the helm chart repository named
repoName, as specified in the $HELM_HOME/repositories/repositories.yaml file.
<url>: Fetch the helm chart archive at the specified URL.
If a custom repository URL is specified by
--helm-chart-repo, the only supported format for
<chartName>: Fetch the helm chart named
chartNamein the helm chart repository specified by the
--helm-chart-version is not set, the SDK will fetch the latest available version of the helm chart. Otherwise, it will fetch the specified version. The option
--helm-chart-version is not used when
--helm-chart itself refers to a specific version, for example when it is a local path or a URL.
Note: For more details and examples run
operator-sdk new --help.
Read the operator scope documentation on how to run your operator as namespace-scoped vs cluster-scoped.
Customize the operator logic
For this example the nginx-operator will execute the following
reconciliation logic for each
Nginx Custom Resource (CR):
- Create a nginx Deployment if it doesn’t exist
- Create a nginx Service if it doesn’t exist
- Create a nginx Ingress if it is enabled and doesn’t exist
- Ensure that the Deployment, Service, and optional Ingress match the desired configuration (e.g. replica count, image, service type, etc) as specified by the
Watch the Nginx CR
By default, the nginx-operator watches
Nginx resource events as shown
watches.yaml and executes Helm releases using the specified chart:
--- - version: v1alpha1 group: example.com kind: Nginx chart: helm-charts/nginx
Reviewing the Nginx Helm Chart
When a Helm operator project is created, the SDK creates an example Helm chart that contains a set of templates for a simple Nginx release.
For this example, we have templates for deployment, service, and ingress resources, along with a NOTES.txt template, which Helm chart developers use to convey helpful information about a release.
If you aren’t already familiar with Helm Charts, take a moment to review the Helm Chart developer documentation.
Understanding the Nginx CR spec
Helm uses a concept called values to provide customizations
to a Helm chart’s defaults, which are defined in the Helm chart’s
Overriding these defaults is as simple as setting the desired values in the CR spec. Let’s use the number of replicas as an example.
helm-charts/nginx/values.yaml, we see that the chart has a
replicaCount and it is set to
1 by default. If we want to have
2 nginx instances in our deployment, we would need to make sure our CR spec
deploy/crds/example.com_v1alpha1_nginx_cr.yaml to look like the following:
apiVersion: example.com/v1alpha1 kind: Nginx metadata: name: example-nginx spec: replicaCount: 2
Similarly, we see that the default service port is set to
80, but we would
like to use
8080, so we’ll again update
by adding the service port override:
apiVersion: example.com/v1alpha1 kind: Nginx metadata: name: example-nginx spec: replicaCount: 2 service: port: 8080
As you may have noticed, the Helm operator simply applies the entire spec as if
it was the contents of a values file, just like
helm install -f ./overrides.yaml
Build and run the operator
Before running the operator, Kubernetes needs to know about the new custom resource definition the operator will be watching.
Deploy the CRD:
kubectl create -f deploy/crds/example.com_nginxes_crd.yaml
Once this is done, there are two ways to run the operator:
- As a pod inside a Kubernetes cluster
- As a go program outside the cluster using
1. Run as a pod inside a Kubernetes cluster
Running as a pod inside a Kubernetes cluster is preferred for production use.
Build the nginx-operator image and push it to a registry:
operator-sdk build quay.io/example/nginx-operator:v0.0.1 docker push quay.io/example/nginx-operator:v0.0.1
Kubernetes deployment manifests are generated in
deployment image in this file needs to be modified from the placeholder
REPLACE_IMAGE to the previous built image. To do this run:
sed -i 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml
If you are performing these steps on OSX, use the following
sed command instead:
sed -i "" 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml
Deploy the nginx-operator:
kubectl create -f deploy/service_account.yaml kubectl create -f deploy/role.yaml kubectl create -f deploy/role_binding.yaml kubectl create -f deploy/operator.yaml
Verify that the nginx-operator is up and running:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-operator 1 1 1 1 1m
2. Run outside the cluster
This method is preferred during the development cycle to speed up deployment and testing.
Run the operator locally with the default Kubernetes config file present at
$ operator-sdk run local INFO Go Version: go1.10.3 INFO Go OS/Arch: linux/amd64 INFO operator-sdk Version: v0.1.1+git
Run the operator locally with a provided Kubernetes config file:
$ operator-sdk run local --kubeconfig=<path_to_config> INFO Go Version: go1.10.3 INFO Go OS/Arch: linux/amd64 INFO operator-sdk Version: v0.2.0+git
3. Deploy your Operator with the Operator Lifecycle Manager (OLM)
OLM will manage creation of most if not all resources required to run your operator,
using a bit of setup from other
operator-sdk commands. Check out the OLM integration
user guide for more information.
Deploy the Nginx custom resource
Apply the nginx CR that we modified earlier:
kubectl apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml
Ensure that the nginx-operator creates the deployment for the CR:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 2 2 2 2 1m
Check the pods to confirm 2 replicas were created:
$ kubectl get pods NAME READY STATUS RESTARTS AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-fjcr9 1/1 Running 0 1m example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-ljbzl 1/1 Running 0 1m
Check that the service port is set to
$ kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 ClusterIP 10.96.26.3 <none> 8080/TCP 1m
Update the replicaCount and remove the port
spec.replicaCount field from 2 to 3, remove the
field, and apply the change:
$ cat deploy/crds/example.com_v1alpha1_nginx_cr.yaml apiVersion: "example.com/v1alpha1" kind: "Nginx" metadata: name: "example-nginx" spec: replicaCount: 3 $ kubectl apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml
Confirm that the operator changes the deployment size:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 3 3 3 3 1m
Check that the service port is set to the default (
$ kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 ClusterIP 10.96.26.3 <none> 80/TCP 1m
Clean up the resources:
kubectl delete -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml kubectl delete -f deploy/operator.yaml kubectl delete -f deploy/role_binding.yaml kubectl delete -f deploy/role.yaml kubectl delete -f deploy/service_account.yaml kubectl delete -f deploy/crds/example.com_nginxes_crd.yaml
NOTE Additional CR/CRD’s can be added to the project by running, for example, the command :
operator-sdk add api --api-version=cache.example.com/v1alpha1 --kind=AppService
For more information, refer cli doc.