Skip to main content
Back to Library
mastering ckad certified kubernetes application developer

StatefulSet and DaemonSet Solutions

9 min read Chapter 72 of 87
Summary

Step-by-step solutions for Exercises 2 and 3: deploying...

Step-by-step solutions for Exercises 2 and 3: deploying a 3-replica StatefulSet with headless Service, verifying stable Pod naming, per-Pod PVCs, DNS resolution, and data persistence across Pod restarts; creating a DaemonSet with nodeSelector, verifying node-scoped scheduling, and observing dynamic Pod creation and removal as labels change.

StatefulSet and DaemonSet Solutions

Solution: Exercise 2 — StatefulSet with Headless Service

This exercise tests the core StatefulSet workflow: headless Service creation, StatefulSet deployment with volumeClaimTemplates, stable Pod naming verification, per-Pod PVC verification, data persistence across Pod deletion, and DNS resolution.

Step 1: Create the Namespace

kubectl create namespace stateful-exercise
namespace/stateful-exercise created

Step 2: Create the Headless Service

# web-headless-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: web-headless
  namespace: stateful-exercise
  labels:
    app: web
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - port: 80
      name: web
kubectl apply -f web-headless-svc.yaml
service/web-headless created

Verify the headless Service:

kubectl get svc web-headless -n stateful-exercise
NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
web-headless   ClusterIP   None         <none>        80/TCP    5s

The CLUSTER-IP column shows None — this confirms the Service is headless. A headless Service does not allocate a virtual IP; instead, DNS queries for the Service name return the individual Pod IP addresses.

Step 3: Create the StatefulSet

# web-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: stateful-exercise
spec:
  serviceName: web-headless
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: html
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: html
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Mi
kubectl apply -f web-statefulset.yaml
statefulset.apps/web created

Key configuration details:

  • serviceName: web-headless — links the StatefulSet to the headless Service. This must match the Service’s metadata.name exactly. Without this, Pod DNS records will not be created.
  • volumeClaimTemplates[].metadata.name: html — matches the volumeMounts[].name in the container spec. The resulting PVCs are named html-web-0, html-web-1, html-web-2.
  • accessModes: ReadWriteOnce — each PVC is mounted by a single Pod. This is the correct mode for per-Pod storage.

Common error: If serviceName doesn’t match an existing headless Service, the StatefulSet is created but Pod DNS records won’t work. Kubernetes does not validate that the referenced Service exists at creation time.

Step 4: Verify Pod Names

Wait for all Pods to be Running:

kubectl get pods -n stateful-exercise -w
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          30s
web-1   1/1     Running   0          20s
web-2   1/1     Running   0          10s

Pods are named web-0, web-1, web-2 — ordinal names derived from the StatefulSet name. Notice the creation order: web-0 was created first, became Ready, then web-1 was created, and finally web-2. This is the default OrderedReady pod management policy.

Step 5: Verify Per-Pod PVCs

kubectl get pvc -n stateful-exercise
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   AGE
html-web-0   Bound    pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890   100Mi      RWO            45s
html-web-1   Bound    pvc-b2c3d4e5-f6a7-8901-bcde-f12345678901   100Mi      RWO            35s
html-web-2   Bound    pvc-c3d4e5f6-a7b8-9012-cdef-012345678912   100Mi      RWO            25s

Each Pod has its own PVC, named <volumeClaimTemplate-name>-<pod-name>. All PVCs are Bound, meaning storage has been provisioned.

Common error: If PVCs are stuck in Pending, you likely don’t have a default StorageClass configured. In a Kind cluster, the standard StorageClass is provisioned by default. Check with kubectl get storageclass. If no default exists, add storageClassName to the volumeClaimTemplate spec pointing to an available StorageClass.

Step 6: Write Data and Verify Persistence

Write a unique file to each Pod’s volume:

kubectl exec web-0 -n stateful-exercise -- sh -c 'echo "pod-0-data" > /usr/share/nginx/html/index.html'
kubectl exec web-1 -n stateful-exercise -- sh -c 'echo "pod-1-data" > /usr/share/nginx/html/index.html'
kubectl exec web-2 -n stateful-exercise -- sh -c 'echo "pod-2-data" > /usr/share/nginx/html/index.html'

Verify the data is in place:

kubectl exec web-0 -n stateful-exercise -- cat /usr/share/nginx/html/index.html
pod-0-data

Now delete web-0 and verify the replacement Pod retains the data:

kubectl delete pod web-0 -n stateful-exercise
pod "web-0" deleted

Wait for the replacement Pod:

kubectl get pods -n stateful-exercise -w
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          5s     # New Pod, same name
web-1   1/1     Running   0          3m
web-2   1/1     Running   0          3m

The replacement Pod is named web-0 — the same name as the original. Check the data:

kubectl exec web-0 -n stateful-exercise -- cat /usr/share/nginx/html/index.html
pod-0-data

The data persisted. The replacement Pod web-0 mounted the same PVC html-web-0, which still contains the file written by the original Pod. This is the core StatefulSet guarantee: stable identity is coupled with stable storage.

Step 7: Verify Stable DNS Resolution

Launch a temporary Pod to test DNS:

kubectl run dns-test --rm -it --image=busybox:1.36 -n stateful-exercise -- nslookup web-0.web-headless
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.web-headless
Address 1: 10.244.0.15 web-0.web-headless.stateful-exercise.svc.cluster.local

The DNS entry web-0.web-headless resolves to the Pod’s IP address. The fully qualified domain name is web-0.web-headless.stateful-exercise.svc.cluster.local.

Test connectivity to a specific Pod by its DNS name:

kubectl run curl-test --rm -it --image=curlimages/curl -n stateful-exercise -- curl -s web-1.web-headless
pod-1-data

The curl request reached web-1 specifically (not a random Pod), and returned the unique data written to that Pod’s volume. This demonstrates both stable DNS and stable storage working together.

Common error: If DNS resolution fails with NXDOMAIN, verify that serviceName in the StatefulSet matches the headless Service name. Also ensure the test Pod is in the same namespace as the StatefulSet (cross-namespace resolution requires the full FQDN).

Cleanup

kubectl delete statefulset web -n stateful-exercise
kubectl delete pvc html-web-0 html-web-1 html-web-2 -n stateful-exercise
kubectl delete svc web-headless -n stateful-exercise
kubectl delete namespace stateful-exercise

Deleting the StatefulSet does not delete the PVCs. You must delete them explicitly. This is by design — accidental StatefulSet deletion should not destroy persistent data.

Solution: Exercise 3 — DaemonSet with Node Selection

This exercise tests DaemonSet creation with nodeSelector, dynamic Pod scheduling when labels change, and Pod removal when labels are removed.

Step 1: Create the Namespace

kubectl create namespace daemon-exercise
namespace/daemon-exercise created

Step 2: Identify and Label a Node

List your nodes:

kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   10d   v1.30.0
kind-worker          Ready    <none>          10d   v1.30.0
kind-worker2         Ready    <none>          10d   v1.30.0

Label one worker node:

kubectl label node kind-worker disk=ssd
node/kind-worker labeled

Verify the label:

kubectl get nodes --show-labels | grep disk=ssd
kind-worker   Ready    <none>   10d   v1.30.0   ...,disk=ssd,...

Step 3: Create the DaemonSet

# node-monitor-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
  namespace: daemon-exercise
  labels:
    app: node-monitor
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      nodeSelector:
        disk: ssd
      containers:
        - name: monitor
          image: busybox:1.36
          command: ["sh", "-c", "while true; do echo $(hostname) $(date); sleep 60; done"]
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 100m
              memory: 128Mi
kubectl apply -f node-monitor-ds.yaml
daemonset.apps/node-monitor created

Step 4: Verify the DaemonSet Runs Only on Labeled Nodes

kubectl get daemonset node-monitor -n daemon-exercise
NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-monitor   1         1         1       1             1           disk=ssd        10s

DESIRED is 1 because only one node has the disk=ssd label. The NODE SELECTOR column confirms the targeting.

Verify which node the Pod is running on:

kubectl get pods -n daemon-exercise -o wide
NAME                 READY   STATUS    RESTARTS   AGE   IP            NODE
node-monitor-xk7m2   1/1     Running   0          15s   10.244.1.12   kind-worker

The Pod is running on kind-worker — the node we labeled with disk=ssd. No Pod is running on kind-worker2 or kind-control-plane.

Check the Pod logs to verify it’s functioning:

kubectl logs -n daemon-exercise -l app=node-monitor
node-monitor-xk7m2 Sun Feb 15 10:45:00 UTC 2026

Step 5: Label a Second Node

kubectl label node kind-worker2 disk=ssd
node/kind-worker2 labeled

Wait a few seconds, then check the DaemonSet:

kubectl get daemonset node-monitor -n daemon-exercise
NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-monitor   2         2         2       2             2           disk=ssd        2m

DESIRED increased from 1 to 2. A new Pod was automatically scheduled on kind-worker2.

Verify both Pods:

kubectl get pods -n daemon-exercise -o wide
NAME                 READY   STATUS    RESTARTS   AGE    IP            NODE
node-monitor-xk7m2   1/1     Running   0          2m     10.244.1.12   kind-worker
node-monitor-ab3k8   1/1     Running   0          10s    10.244.2.8    kind-worker2

Two Pods, one per labeled node. The DaemonSet controller detected the new matching node and scheduled a Pod automatically.

Step 6: Remove a Label and Verify Pod Removal

Remove the disk=ssd label from kind-worker:

kubectl label node kind-worker disk-
node/kind-worker unlabeled

The trailing - removes the label. Wait a few seconds:

kubectl get daemonset node-monitor -n daemon-exercise
NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-monitor   1         1         1       1             1           disk=ssd        3m

DESIRED dropped back to 1. The Pod on kind-worker was terminated.

Verify:

kubectl get pods -n daemon-exercise -o wide
NAME                 READY   STATUS    RESTARTS   AGE    IP            NODE
node-monitor-ab3k8   1/1     Running   0          75s    10.244.2.8    kind-worker2

Only the Pod on kind-worker2 remains. The DaemonSet controller continuously reconciles the desired state (one Pod per matching node) with the actual state. When kind-worker lost its disk=ssd label, it no longer matched the nodeSelector, and the Pod was removed.

Cleanup

kubectl label node kind-worker2 disk-
kubectl delete namespace daemon-exercise

Troubleshooting Reference

SymptomCauseFix
StatefulSet Pods stuck in PendingNo default StorageClass or insufficient storageCheck kubectl get storageclass and ensure a default exists
StatefulSet Pod web-1 not startingweb-0 not Ready yet (OrderedReady policy)Wait for web-0 to be Running and Ready
DNS lookup returns NXDOMAINserviceName mismatch or wrong namespaceVerify serviceName matches the headless Service name
PVC still exists after deleting StatefulSetBy design — PVCs are never auto-deletedDelete PVCs manually: kubectl delete pvc <name>
DaemonSet shows DESIRED=0No nodes match the nodeSelectorCheck node labels with kubectl get nodes --show-labels
DaemonSet Pod not on control planeControl plane has NoSchedule taintAdd toleration for node-role.kubernetes.io/control-plane
DaemonSet Pod in CrashLoopBackOffContainer command errorCheck kubectl logs <pod> — verify the command syntax
DaemonSet Pod in PendingInsufficient resources on matching nodesLower resource requests in the DaemonSet spec