Lab: Kubernetes Extensions
Access the client terminal
Login to ibmcloud
Connect to your cluster
Create a Custom Resource
Operators
Ready Made Operators
Create a Custom Resource and Operator using the Operator SDK
Install sdk-operator
Create the Operator
Cleanup
Application CRD
Access the web-terminal
Go to the URL for your web-terminal, given to you by the instructor team.
Login
export CLUSTER_NAME=<your cluster name>
ibmcloud login
oc config current-context
ibmcloud login
API endpoint: https://cloud.ibm.com
Email> a.newell1@remkoh.dev
Password>
Authenticating...
OK
Select an account:
1. A Newell's Account (ed69e4e9e2d74660b6cc45d0e47cd8b7)
2. Advowork (e2b54d0c3bbe4180b1ee63a0e2a7aba4) <-> 1840867
Enter a number> 2
Targeted account Advowork (e2b54d0c3bbe4180b1ee63a0e2a7aba4) <-> 1840867
Select a region (or press enter to skip):
1. au-syd
2. in-che
3. jp-osa
4. jp-tok
5. kr-seo
6. eu-de
7. eu-gb
8. us-south
9. us-south-test
10. us-east
Enter a number> 10
Targeted region us-east
Connect to your cluster by downloading the cluster config,
ibmcloud ks cluster config --cluster $CLUSTER_NAME
Check you can access the cluster,
oc config current-context
oc get nodes
Create a Custom Resource (CR)
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
Custom Resource Definitions (CRD) were added in Kubernetes v1.7 in June 2017. A CRD defines Custom Resources (CR). A CR is an extension of the Kubernetes API that allows you to store your own API Objects and lets the API Server handle the lifecycle of a CR. On their own, CRs simply let you store and retrieve structured data.
For instance, our Guestbook application consists of an object Guestbook
with attributes GuestbookTitle
and GuestbookSubtitle
, and a Guestbook handles objectes of type GuestbookMessage
with attributes Message
, Sender
.
You have to ask yourself if it makes sense if your objects are added as a Custom Resource to Kubernetes or not. If your API is a Declarative API you can consider adding a CR.
Your API has a small number of small objects (resources).
The objects define configuration of applications or infrastructure.
The objects are updated relatively infrequently.
Users often need to read and write the objects.
main operations on the objects are CRUD (create, read, update and delete).
Transactions between objects are not required.
It doesn't immediately make sense to store messages by Guestbook users in Kubernetes, but it might make sense to store meta-data about a Guestbook deployment, for instance the title and subtitle of a Guestbook deployment, assigned resources or replicas.
Another benefit of adding a Custom Resource is to view your types in the Kubernetes Dashboard.
If you want to deploy a Guestbook instance as a Kubernetes API object and let the Kubernetes API Server handle the lifecycle events of the Guestbook deployment, you can create a Custom Resource Definition (CRD) for the Guestbook object as follows. That way you can deploy multiple Guestbooks with different titles and let each be managed by Kubernetes.
cat <<EOF >>guestbook-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: guestbooks.apps.ibm.com
spec:
group: apps.ibm.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
guestbookTitle:
type: string
guestbookSubtitle:
type: string
scope: Namespaced
names:
plural: guestbooks
singular: guestbook
kind: Guestbook
shortNames:
- gb
EOF
You can see that the
apiVersion
is part of theapiextensions.k8s.io/v1
API Group in Kubernetes, which is the API that enables extensions, and thekind
is set toCustomResourceDefinition
.The
served
flag can disable and enable a version.Only 1 version can be flagged as the storage version.
The
spec.names.kind
is used by your resource manifests and should be CamelCased.
Create the Custom Resource for the Guestbook witht he command,
oc create -f guestbook-crd.yaml
When run in the terminal,
$ kubectl create -f guestbook-crd.yaml
customresourcedefinition.apiextensions.k8s.io/guestbooks.apps.ibm.com created
You have now added a CR to the Kubernetes API, but you have not yet created a deployment of type Guestbook yet.
Create a resource specification of type Guestbook named my-guestbook
,
cat <<EOF >>my-guestbook.yaml
apiVersion: "apps.ibm.com/v1"
kind: Guestbook
metadata:
name: my-guestbook
spec:
guestbookTitle: "The Chemical Wedding of Remko"
guestbookSubtitle: "First Day of Many"
EOF
And to create the my-guestbook
resource, run the command
oc create -f my-guestbook.yaml
When run in the terminal,
$ oc create -f my-guestbook.yaml
guestbook.apps.ibm.com/my-guestbook created
If you list all Kubernetes resources, only the default Kubernetes service is listed. To list your Custom Resources, add the extended type to your command.
$ oc get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.21.0.1 <none> 443/TCP 5d14h
$ oc get guestbook
NAME AGE
my-guestbook 8m32s
To read the details for the my-guestbook
of type Guestbook
, describe the instance,
$ oc describe guestbook my-guestbook
Name: my-guestbook
Namespace: default
Labels: <none>
Annotations: <none>
API Version: apps.ibm.com/v1
Kind: Guestbook
Metadata:
Creation Timestamp: 2020-06-30T20:31:36Z
Generation: 1
Resource Version: 1081471
Self Link: /apis/apps.ibm.com/v1/namespaces/default/guestbooks/my-guestbook
UID: dcbdcafc-999d-4051-9244-0315093357e7
Spec:
Guestbook Subtitle: First Day of Many
Guestbook Title: The Chemical Wedding of Remko
Events: <none>
Or retrieve the resource information by specifying the type,
$ oc get Guestbook -o yaml
apiVersion: v1
items:
- apiVersion: apps.ibm.com/v1
kind: Guestbook
metadata:
creationTimestamp: "2020-07-02T04:41:57Z"
generation: 1
name: my-guestbook
namespace: default
resourceVersion: "1903244"
selfLink: /apis/apps.ibm.com/v1/namespaces/default/guestbooks/my-guestbook
uid: 3f774899-3070-4e00-b74c-a6a14654faeb
spec:
guestbookSubtitle: First Day of Many
guestbookTitle: The Chemical Wedding of Remko
kind: List
metadata:
resourceVersion: ""
selfLink: ""
In the Kubernetes Dashboard web console, you can browse to Custom Resource Definitions and find the Guestbook CRD.

You have now created a new type or Custom Resource (CR) and created an instance of your new type. But just having a new type and a new instance of the type, does not add as much control over the instances yet, we can basically only create and delete a static type with some descriptive meta-data. With a custom controller or Operator
you can over-write the methods that are triggered at certain lifecycle events.
Cleanup
oc delete guestbook my-guestbook
oc delete customresourcedefinition guestbooks.apps.ibm.com
Operators
https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
Operators are clients of the Kubernetes API that act as controllers for a Custom Resource.
To write applications that use the Kubernetes REST API, you can use one of the following supported client libraries:
In addition, there are many community-maintained client libraries.
Ready made operators
At the Kubernetes OperatorHub.io, you find ready to use operators written by the community.

For a Red Hat curated list of Kubernetes Operators you can go to the OPenShift console > Operators > OperatorHub.

For a list of Red Hat certified partner software go to the Red Hat Marketplace at http://ibm.biz/clientrhm

Create a Custom Resource and Operator using the Operator SDK
To write your own operator you can use existing tools:
KUDO (Kubernetes Universal Declarative Operator),
Metacontroller using custom WebHooks,
the Operator Framework.
The Operator SDK provides the following workflow to develop a new Operator:
The following workflow is for a new Go operator:
Create a new operator project using the SDK Command Line Interface(CLI)
Define new resource APIs by adding Custom Resource Definitions(CRD)
Define Controllers to watch and reconcile resources
Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
Use the SDK CLI to build and generate the operator deployment manifests
Install sdk-operator
The following section uses the sdk-operator
cli, which depends on Go
to be installed. Check if both tools are installed in your web-terminal,
go version
operator-sdk version
If you see a command not found
error, install both now.
For detailed installation instructions go here.
To install the Operator SDK in Ubuntu, you need to install the Go tools and the Operator SDK.
curl -LO https://golang.org/dl/go1.14.4.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.14.4.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
go version
Install the Operator SDK,
curl -LO https://github.com/operator-framework/operator-sdk/releases/download/v0.18.2/operator-sdk-v0.18.2-x86_64-linux-gnu
chmod +x operator-sdk-v0.18.2-x86_64-linux-gnu
sudo mkdir -p /usr/local/bin/
sudo cp operator-sdk-v0.18.2-x86_64-linux-gnu /usr/local/bin/operator-sdk
rm operator-sdk-v0.18.2-x86_64-linux-gnu
operator-sdk version
Check again,
go version
operator-sdk version
Create the Operator
1. Create a New Project
Create a new Operator project,
export DOCKER_USERNAME=<your-docker-username>
and
export OPERATOR_NAME=guestbook-operator
export OPERATOR_PROJECT=guestbook-project
export OPERATOR_GROUP=guestbook.ibm.com
export OPERATOR_VERSION=v1
export CRD_KIND=Guestbook
Create a new Operator project,
operator-sdk new $OPERATOR_PROJECT --type go --repo github.com/$DOCKER_USERNAME/$OPERATOR_NAME
cd $OPERATOR_PROJECT
The scaffolding of a new project will create an operator, an api and a controller.

2. Create a new API
Add a new API definition for a new Custom Resource under pkg/apis
and generate the Custom Resource Definition (CRD) and Custom Resource (CR) files under deploy/crds
.
operator-sdk add api --api-version=$OPERATOR_GROUP/$OPERATOR_VERSION --kind=$CRD_KIND
The command will create a new API, a Custom Resource (CR), a Custom Resource Definition (CRD).

One file is created in pkg/apis
called addtoscheme_guestbook_v1.go
that registers the new schema. One new file is created in pkg/apis/guestbook
called group.go
that defines the package. Four new files are created in pkg/apis/guestbook/v1
:
doc.go,
guestbook_types.go,
register.go,
zz_generated.deepcopy.go.
The guestbook_types.go
file,
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// GuestbookSpec defines the desired state of Guestbook
type GuestbookSpec struct {
}
// GuestbookStatus defines the observed state of Guestbook
type GuestbookStatus struct {
}
type Guestbook struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec GuestbookSpec `json:"spec,omitempty"`
Status GuestbookStatus `json:"status,omitempty"`
}
// GuestbookList contains a list of Guestbook
type GuestbookList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Guestbook `json:"items"`
}
func init() {
SchemeBuilder.Register(&Guestbook{}, &GuestbookList{})
}
The Custom Resource (CR) in file deploy/crds/guestbook.remkoh.dev_v1_guestbook_cr
,
apiVersion: guestbook.remkoh.dev/v1
kind: Guestbook
metadata:
name: example-guestbook
spec:
# Add fields here
size: 3
The Custom Resource Definition (CRD) in file deploy/crds/guestbook.remkoh.dev_guestbooks_crd.yaml
,
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: guestbooks.guestbook.remkoh.dev
spec:
group: guestbook.remkoh.dev
names:
kind: Guestbook
listKind: GuestbookList
plural: guestbooks
singular: guestbook
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: Guestbook is the Schema for the guestbooks API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: GuestbookSpec defines the desired state of Guestbook
type: object
status:
description: GuestbookStatus defines the observed state of Guestbook
type: object
type: object
served: true
storage: true
subresources:
status: {}
3. Create a new Controller
Add a new controller under pkg/controller/<kind>
.
operator-sdk add controller --api-version=$OPERATOR_GROUP/$OPERATOR_VERSION --kind=$CRD_KIND
This command creates two files in pkg/controller
:
add_guestbook.go
, which registers the new controller, andguestbook/guestbook_controller.go
, which is the actual custom controller logic.
The file guestbook/guestbook_controller.go
defines the Reconcile
function,
// Reconcile reads state of the cluster for a Guestbook object and makes changes based on the state read and what is in the Guestbook.Spec
// TODO(user): User must modify this Reconcile function to implement their own Controller logic. This example creates a Pod as an example
func (r *ReconcileGuestbook) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
// Fetch the Guestbook instance
instance := &guestbookv1.Guestbook{}
...
// Define a new Pod object
pod := newPodForCR(instance)
...
}
4. Compile and Build the Code
The operator-sdk build command compiles the code and builds the executables. fter you built the image, push it to your image registry, e.g. Docker hub.
operator-sdk build docker.io/$DOCKER_USERNAME/$OPERATOR_NAME
Login in to the container registry,
docker login docker.io -u $DOCKER_USERNAME
And push the build image to your image repository,
docker push docker.io/$DOCKER_USERNAME/$OPERATOR_NAME
5. Deploy the Operator
First replace the image attribute in the operator resource with the built image,
Inspect the deploy/operator.yaml file,
cat deploy/operator.yaml
And replace the REPLACE_IMAGE
value,
$ sed -i "s|REPLACE_IMAGE|docker.io/$DOCKER_USERNAME/$OPERATOR_NAME|g" deploy/operator.yaml
Make sure you are connected to the OpenShift cluster (see above how to connect), and deploy the operator with the following template code.
oc create sa $OPERATOR_PROJECT
oc create -f deploy/role.yaml
oc create -f deploy/role_binding.yaml
oc create -f deploy/crds/${OPERATOR_GROUP}_${CRD_KIND,,}s_crd.yaml
oc create -f deploy/operator.yaml
oc create -f deploy/crds/${OPERATOR_GROUP}_${OPERATOR_VERSION}_${CRD_KIND,,}_cr.yaml
Verify the deployment,
oc get deployment $OPERATOR_PROJECT
oc get pod -l app=example-${CRD_KIND,,}
oc describe ${CRD_KIND,,}s.${OPERATOR_GROUP} example-${CRD_KIND,,}
For our example Guestbook project the above templates should resolve as follows,
$ oc create sa guestbook-project
$ oc create -f deploy/role.yaml
$ oc create -f deploy/role_binding.yaml
$ oc create -f deploy/crds/guestbook.remkoh.dev_guestbooks_crd.yaml
$ oc create -f deploy/operator.yaml
$ oc create -f deploy/crds/guestbook.remkoh.dev_v1_guestbook_cr.yaml
$ oc get deployment guestbook-project
$ oc get pod -l app=example-guestbook
$ oc describe guestbooks.guestbook.remkoh.dev example-guestbook
Cleanup
oc delete sa $OPERATOR_PROJECT
oc delete role $OPERATOR_PROJECT
oc delete rolebinding $OPERATOR_PROJECT
oc delete customresourcedefinition ${CRD_KIND,,}s.${OPERATOR_GROUP}
oc delete deployment $OPERATOR_PROJECT
Application CRD
The Application CRD (Custom Resource Definition) and Controller provide the following:
Describe an applications metadata,
A point to connect the infrastructure, such as Deployments, to as a root object.
Application level health checks.
This could be used by:
Application operators,
Tools, such as Helm, and
Dashboards.
apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
name: "guestbook"
labels:
app.kubernetes.io/name: "guestbook"
spec:
selector:
matchLabels:
app.kubernetes.io/name: "guestbook"
componentKinds:
- group: v1
kind: Deployment
- group: v1
kind: Service
descriptor:
type: "guestbook"
keywords:
- "gb"
- "guestbook"
links:
- description: Github
url: "https://github.com/IBM/guestbook"
version: "0.1.0"
description: "The Guestbook application is an example app to demonstrate key Kubernetes functionality."
maintainers:
- name: IBM Developer
email: developer@ibm.com
owners:
- name: IBM Developer
email: developer@ibm.com
Last updated
Was this helpful?