Kubernetes CKS sample exam question 4 with answer - API Server crash

Question:

I
The idea here is to misconfigure the Apiserver in different ways, then check possible log locations for errors.
You should be very comfortable with situations where the Apiserver is not coming back up.
Configure the Apiserver manifest with a new argument --this-is-very-wrong.
Check if the Pod comes back up and what logs this causes.
Fix the Apiserver again.
Log locations to check:

/var/log/pods
/var/log/containers
crictl ps + crictl logs
docker ps + docker logs (in case when Docker is used)
kubelet logs: /var/log/syslog or journalctl

II
Change the existing Apiserver manifest argument to: --etcd-servers=this-is-very-wrong
Check what the logs say, without using anything in /var.
Fix the Apiserver again.
Log locations to check:

crictl ps + crictl logs
docker ps + docker logs (in case when Docker is used)
kubelet logs: journalctl

III
Change the Apiserver manifest and add invalid YAML, something like this:

apiVersionTHIS IS VERY ::::: WRONG v1
kind: Pod
metadata:

Check what the logs say, and fix again.
Fix the Apiserver again.
Log locations to check:

/var/log/pods
/var/log/containers
crictl ps + crictl logs
docker ps + docker logs (in case when Docker is used)
kubelet logs: /var/log/syslog or journalctl

Answer:

I
Make a backup first:

cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.orig

Make the change in /etc/kubernetes/manifests/kube-apiserver.yaml:

...
spec:
  containers:
  - command:
    - kube-apiserver
    - --this-is-very-wrong
...

Wait until container restarts:

watch crictl ps -a

API server won't come up - check the logs:

less +G /var/log/pods/kube-system_kube-apiserver-controlplane_0103fb254ea1bf2f261b9d12ef40fe36/kube-apiserver/4.log

Output should be like this:

2023-09-06T19:27:23.84700297Z stderr F Error: unknown flag: --this-is-very-wrong

Undo the changes:

cp ~/kube-apiserver.yaml.orig /etc/kubernetes/manifests/kube-apiserver.yaml

Check if API Server comes back:

watch crictl ps
k -n kube-system get pod

II

Make a backup first:

cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.orig

Make the change in /etc/kubernetes/manifests/kube-apiserver.yaml:

...
spec:
  containers:
  - command:
    - kube-apiserver
    - --etcd-servers=this-is-very-wrong
...

Check with crictl running and exited containers:

crictl ps -a

We can see:

CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID              POD
...
14942f12da1ab       6f6e73fa8162b       43 seconds ago       Exited              kube-apiserver            1                   a0ffd2d921d5f       kube-apiserver-controlplane
...

Check the logs of this container:

cricrtl logs 14942f12da1ab

The ourput is something like this:

W0906 19:35:50.067139       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "this-is-very-wrong",
  "ServerName": "this-is-very-wrong",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp: address this-is-very-wrong: missing port in address"

Undo the changes:

cp ~/kube-apiserver.yaml.orig /etc/kubernetes/manifests/kube-apiserver.yaml

Check if API Server comes back:

watch crictl ps
k -n kube-system get pod

III

Make a backup first:

cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.orig

Make the change in /etc/kubernetes/manifests/kube-apiserver.yaml:

apiVersionTHIS IS VERY ::::: WRONG v1
kind: Pod
metadata:
  annotations:
...

Wait until container restarts - it is not even in the list:

watch crictl ps -a

Try syslog:

tail -f /var/log/syslog | grep apiserver

Output:

Sep  6 19:43:44 controlplane kubelet[24852]: E0906 19:43:44.353397   24852 file.go:187] "Could not process manifest file" err="/etc/kubernetes/manifests/kube-apiserver.yaml: couldn't parse as pod(Object 'apiVersion' is missing in '{\"apiVersionTHIS IS VERY ::::\":\"WRONG v1\"

Or journalctl:

journalctl | grep apiserver

Output:

Sep 06 19:44:44 controlplane kubelet[24852]: E0906 19:44:44.353269   24852 file.go:187] "Could not process manifest file" err="/etc/kubernetes/manifests/kube-apiserver.yaml: couldn't parse as pod(Object 'apiVersion' is missing in '{\"apiVersionTHIS IS VERY ::::\":\"WRONG v1\"

Undo the changes:

cp ~/kube-apiserver.yaml.orig /etc/kubernetes/manifests/kube-apiserver.yaml

Check if API Server comes back:

watch crictl ps
k -n kube-system get pod