Prove It Works

Run the smoke test

Where you are

The cluster is built. The control plane is up, both worker nodes are ready, and pods can reach each other across nodes. Now you confirm that each core capability works from end to end: secret data is encrypted at rest, you can create a deployment, forward a port, read logs, run a command inside a container, and expose an application through a service. If these pass, you have a real working cluster.

Encrypt secret data at rest

First you verify that Kubernetes encrypts secret data before it is written to etcd, the database that holds cluster state.

Create a generic secret:

kubectl create secret generic kubernetes-the-hard-way \
  --from-literal="mykey=mydata"

Print a hexdump of the kubernetes-the-hard-way secret stored in etcd:

ssh root@server \
    'etcdctl get /registry/secrets/default/kubernetes-the-hard-way | hexdump -C'

00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
00000040  3a 76 31 3a 6b 65 79 31  3a 4f 1b 80 d8 89 72 f4  |:v1:key1:O....r.|
00000050  60 8a 2c a0 76 1a e1 dc  98 d6 00 7a a4 2f f3 92  |`.,.v......z./..|
00000060  87 63 c9 22 f4 58 c8 27  b9 ff 2c 2e 1a b6 55 be  |.c.".X.'..,...U.|
00000070  d5 5c 4d 69 82 2f b7 e4  b3 b0 12 e1 58 c4 9c 77  |.\Mi./......X..w|
00000080  78 0c 1a 90 c9 c1 23 6c  73 8e 6e fd 8e 9c 3d 84  |x.....#ls.n...=.|
00000090  7d bf 69 81 ce c9 aa 38  be 3b dd 66 aa a3 33 27  |}.i....8.;.f..3'|
000000a0  df be 6d ac 1c 6d 8a 82  df b3 19 da 0f 93 94 1e  |..m..m..........|
000000b0  e0 7d 46 8d b5 14 d0 c5  97 e2 94 76 26 a8 cb 33  |.}F........v&..3|
000000c0  57 2a d0 27 a6 5a e1 76  a7 3f f0 b7 0a 7b ff 53  |W*.'.Z.v.?...{.S|
000000d0  cf c9 1a 18 5b 45 f8 b1  06 3b a9 45 02 76 23 61  |....[E...;.E.v#a|
000000e0  5e dc 86 cf 8e a4 d3 c9  5c 6a 6f e6 33 7b 5b 8f  |^.......\jo.3{[.|
000000f0  fb 8a 14 74 58 f9 49 2f  97 98 cc 5c d4 4a 10 1a  |...tX.I/...\.J..|
00000100  64 0a 79 21 68 a0 9e 7a  03 b7 19 e6 20 e4 1b ce  |d.y!h..z.... ...|
00000110  91 64 ce 90 d9 4f 86 ca  fb 45 2f d6 56 93 68 e1  |.d...O...E/.V.h.|
00000120  0b aa 8c a0 20 a6 97 fa  a1 de 07 6d 5b 4c 02 96  |.... ......m[L..|
00000130  31 70 20 83 16 f9 0a 22  5c 63 ad f1 ea 41 a7 1e  |1p ...."\c...A..|
00000140  29 1a d4 a4 e9 d7 0c 04  74 66 04 6d 73 d8 2e 3f  |).......tf.ms..?|
00000150  f0 b9 2f 77 bd 07 d7 7c  42 0a                    |../w...|B.|
0000015a

The etcd key should be prefixed with k8s:enc:aescbc:v1:key1, which indicates the aescbc provider was used to encrypt the data with the key1 encryption key. The secret value never appears in plain text in the dump.

Create a deployment

Next you create and manage a Deployment, which is the object Kubernetes uses to run and keep an application running.

Create a deployment for the nginx web server:

kubectl create deployment nginx \
  --image=nginx:latest

List the pod created by the nginx deployment:

kubectl get pods -l app=nginx

NAME                     READY   STATUS    RESTARTS   AGE
nginx-56fcf95486-c8dnx   1/1     Running   0          8s

Forward a port

Now you reach the running application from your own machine using port forwarding, which sends traffic from a local port straight to a port on a pod.

Retrieve the full name of the nginx pod:

POD_NAME=$(kubectl get pods -l app=nginx \
  -o jsonpath="{.items[0].metadata.name}")

Forward port 8080 on your local machine to port 80 of the nginx pod:

kubectl port-forward $POD_NAME 8080:80

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

In a new terminal make an HTTP request using the forwarding address:

curl --head http://127.0.0.1:8080

HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Sun, 06 Apr 2025 17:17:12 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
Connection: keep-alive
ETag: "67a34638-267"
Accept-Ranges: bytes

Switch back to the previous terminal and stop the port forwarding to the nginx pod:

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
^C

Read logs

Now you read the container logs, which is how you see what an application has printed while running.

Print the nginx pod logs:

kubectl logs $POD_NAME

...
127.0.0.1 - - [06/Apr/2025:17:17:12 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.88.1" "-"

The log shows the request you just made through the forwarded port.

Run a command inside a container

Now you run a command inside a running container, which is how you inspect or debug an application in place.

Print the nginx version by executing the nginx -v command in the nginx container:

kubectl exec -ti $POD_NAME -- nginx -v

nginx version: nginx/1.27.4

Expose the application with a service

Finally you expose the application using a Service, which gives a set of pods a stable way to receive traffic.

Expose the nginx deployment using a NodePort service:

kubectl expose deployment nginx \
  --port 80 --type NodePort

The LoadBalancer service type can not be used because your cluster is not configured with cloud provider integration. Setting up cloud provider integration is out of scope for this tutorial.

PHI lens. A NodePort opens the same port on every node and lets traffic from outside the cluster reach the application. In a healthcare cluster that is exactly how protected health information gets exposed by accident, so confirm here that you understand what you just opened. Before any real workload you would lock this down, for example with a firewall rule or an ingress controller, instead of leaving a port open on every node.

Retrieve the node port assigned to the nginx service:

NODE_PORT=$(kubectl get svc nginx \
  --output=jsonpath='{range .spec.ports[0]}{.nodePort}')

Retrieve the hostname of the node running the nginx pod:

NODE_NAME=$(kubectl get pods \
  -l app=nginx \
  -o jsonpath="{.items[0].spec.nodeName}")

Make an HTTP request using the IP address and the nginx node port:

curl -I http://${NODE_NAME}:${NODE_PORT}

Server: nginx/1.27.4
Date: Sun, 06 Apr 2025 17:18:36 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
Connection: keep-alive
ETag: "67a34638-267"
Accept-Ranges: bytes

The request reaches nginx through the node port. Every core capability now works.

Tear down

When you are done with the cluster, you remove the machines you created.

Earlier versions of this guide created cloud resources for compute and networking. This version does not. All configuration lives on the jumpbox, the server, and the nodes, so there is nothing in a cloud account to unwind.

Cleanup is just deleting the virtual machines you created for this exercise. Stop them if you want to come back later, or delete them to free the resources for good. Once they are gone, the cluster is gone.

Key terms

Deployment: A Kubernetes object that runs an application and keeps it running. You tell it which container image to use and how many copies you want, and it creates the pods and replaces any that fail.

Port forward: A way to connect a port on your local machine directly to a port on a pod. Traffic you send to the local port arrives at the pod, which lets you reach an application without exposing it to anyone else.

NodePort service: A Service that opens the same port on every node in the cluster. Any request that reaches a node on that port is routed to the application, including requests from outside the cluster.

Reflect

You just confirmed the cluster runs a stock nginx image with default settings. If you needed this cluster to run a specific healthcare application instead, which of these tests would you change first, and what would you check that you did not check here?

Next: Optional extension: run a PHI scrubber on your cluster