Deep dive admission controllers

What are admission controllers

• Gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether. For example, when a namespace is deleted and subsequently enters the Terminating state, the NamespaceLifecycle admission controller is what prevents any new objects from being created in this namespace.
• Among the more than 30 admission controllers shipped with Kubernetes, two take a special role because of their nearly limitless flexibility - ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks. This approach decouples the admission controller logic from the Kubernetes API server, thus allowing users to implement custom logic to be executed whenever resources are created, updated, or deleted in a Kubernetes cluster.

Ref: k8s-blog

How to write basic validation webhook from beginning

• https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks
• https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation.html
• https://medium.com/ovni/writing-a-very-basic-kubernetes-mutating-admission-webhook-398dbbcb63ec
• https://github.com/banzaicloud/admission-webhook-example

Experiments

Missions

• Write a validating admission webhook
  • If Pod has the label webhook-validate:true. Do not allow the creation of Pod in default namespace
• Write a mutating admission webhook
  • Add some custom annotation for pods who have the label webhook-mutate:true

Prerequisites

• A running K8S cluster (we will use kind in this experiment)

    brew update
    brew upgrade kind kubectl
    kind create cluster --config kind.yaml --image kindest/node:v1.20.2
  

    kind: Cluster
    apiVersion: kind.sigs.k8s.io/v1alpha3
    kubeadmConfigPatches:
    - |
        apiVersion: kubeadm.k8s.io/v1beta2
        kind: ClusterConfiguration
        metadata:
        name: config
        apiServer:
          extraArgs:
            "enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
    nodes:
    - role: control-plane
    - role: worker
    - role: worker
    - role: worker
  

• Ensure that the admissionregistration.k8s.io/v1 or admissionregistration.k8s.io/v1beta1 API is enabled.

    kubectl api-versions | grep admissionregistration
    admissionregistration.k8s.io/v1
    admissionregistration.k8s.io/v1beta1
  

Write a webhook server

The code is modified based on link. It has been forked in to github.

• Checkout the code
• Build the docker image

docker build --no-cache -f Dockerfile -t sample-webhook-server:v1 --rm=true .

• Load into kind

kind load docker-image sample-webhook-server:v1

Generate certs and keys

Note: You have to change the "/CN=sample-webhook-server.webhook.svc" in genkeys.sh to be "/CN=<service-name>.<namespace>.svc"

cd $GOPATH/src/github.com/danniel1205/sample-webhook-server/
mkdir -p keys
./hacks/genkeys.sh keys

tree keys
keys
├── ca.crt
├── ca.key
├── ca.srl
├── webhook-server-tls.crt
└── webhook-server-tls.key

Create a namespace

kubectl create namespace webhook

Create secrets from the keys generated

kubectl create secret tls webhook-tls --key=./keys/webhook-server-tls.key --cert=./keys/webhook-server-tls.crt -n webhook

Create the webhook service

kubectl apply -f $GOPATH/src/github.com/danniel1205/sample-webhook-server/deploy/01-deployment.yaml

kubectl get svc,deployment,pod -n webhook
NAME                            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/sample-webhook-server   ClusterIP   10.99.16.47   <none>        443/TCP   2m33s

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sample-webhook-server   1/1     1            1           2m33s

NAME                                         READY   STATUS    RESTARTS   AGE
pod/sample-webhook-server-6449948fcb-d9pq9   1/1     Running   0          25s

Create ValidatingWebhookConfiguration

Note: Update the CABundle in 02-validating-webhook-config.yaml to be base64 encoded of keys/ca.crt

kubectl apply -f $GOPATH/src/github.com/danniel1205/sample-webhook-server/deploy/02-validating-webhook-config.yaml

Try to create the test pod

kubectl apply -f $GOPATH/src/github.com/danniel1205/sample-webhook-server/deploy/test-validating-pod.yaml

Error from server: error when creating "deploy/test-pod.yaml": admission webhook "sample-webhook-server.example.com" denied the request: the namespace must be specified to create pod

Create MutatingWebhookConfiguration

Note: Update the CABundle in 03-mutating-webhook-config.yaml to be base64 encoded of keys/ca.crt

kubectl apply -f $GOPATH/src/github.com/danniel1205/sample-webhook-server/deploy/03-mutating-webhook-config.yaml

Q&A

• Why validating admission is after mutating admission ? The reason is whatever request object a validating webhook sees needs to be the final version that would be persisted to etcd
• If we have two webhooks are applied to the same resource, in which order the request will be validated ? The request will be validated against the webhooks specified in the ValidatingWebhookConfiguration.webhooks one by one. Which means if the first succeeded, the request will be sent to the second one.