Pod Affinity

To prevent the control plane nodes from being loaded by user deployments, you can configure a Deployment or StatefulSet so that it will not place pods on control plane nodes.

The following is placed under spec.template.spec section:

 affinity:
   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
       - matchExpressions:
         - key: node-role.kubernetes.io/control-plane
           operator: DoesNotExist

Do this for your app and database pods, and then apply the YAML file to stop and restart the pod(s).

Determining pod location

To see where a pod is running, use:

kubectl get pods {-A | -n NAMESPACE} -o wide

To see what pods are running on a node, use:

kubectl get pods -A -o wide --field-selector spec.nodeName=NODE-NAME

Temperature (Raspberry PI)

To find out the temperature, and see if there is/was throttling, you can run these commands on a Raspberry PI 4B:

vcgencmd measure_temp
vcgencmd get_throttled

The temperature is in Celsius, and the throttling is a bit map. Some definitions I found, are:

Value    Meaning relevant here
0x0      No current/historical throttling flags
0xe0000  Historical thermal/throttling-related flags, but no current under-voltage bit
0xe0008  Same historical flags plus current under-voltage

Kubespray Runs

I used Kubespray to deploy my Kubernetes cluster. Afterward, I realized that I needed to customize some settings for kube-vip (provides a virtual IP for the API Server, which is shared by multiple control plane hosts).

I had some “extra” settings that I applied in two files. In inventory/mycluster/group_vars/k8s_cluster/addons.yml:

kube_vip_leaseduration: 30
kube_vip_renewdeadline: 20
kube_vip_retryperiod: 4

And in inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

leader-elect-lease-duration: “60s”
leader-elect-renew-deadline: “40s”
leader-elect-retry-period: “10s”

kube_kubeadm_scheduler_extra_args:
leader-elect-lease-duration: “60s”
leader-elect-renew-deadline: “40s”
leader-elect-retry-period: “10s”

The question I had, was how to ensure that these would be applied correctly, if I ever rebuilt or upgraded the cluster. Asking ClaudeAI, these are the steps I was guided into using.

First, to make sure that we have all the needed info on the cluster. This is done by using the “facts” target.

ansible-playbook playbooks/facts.yml -b \
  -i ../picluster/inventory/mycluster/hosts.yaml \
  -u ${USER} -b -v --private-key=~/.ssh/id_ed25519

Second, I needed to know what playbook step(s) of the Kubespray cluster install would need to run, to update with these parameters. The following command can be used to get the tags for the various splaybook teps of a run:

 
ansible-playbook -i ../picluster/inventory/mycluster/hosts.yaml \
  cluster.yml --list-tags
...
 play #9 (k8s_cluster): Install Kubernetes nodes TASK TAGS: [..., kube-vip, ...]

Next, I have three control plane nodes, so I had to repeat each of the following tasks for each node. It starts with trying a dry run of the change, in my case, on node “lock”:

 ansible-playbook -i ../picluster/inventory/mycluster/hosts.yaml cluster.yml \
--limit=lock --tags="kube-vip" --check --diff \
-u ${USER} -b -v --private-key=~/.ssh/id_ed25519

Once I identified that the diffs looked correct, the command was re-run, without the “–check –diff”. To verify the change, I checked on the node, that the settings were there, and made sure that the kube-vip pod was running:

cat /etc/kubernetes/manifests/kube-vip.yml | grep -A1 "cp_enable\|vip_leaseduration\|vip_renewdeadline\|vip_retryperiod"

kubectl -n kube-system get pod kube-vip-lock

The next playbook to run was for play #10:

play #10 (kube_control_plane): Install the control plane   TASK TAGS: […, control-plane, …, upgrade]

Again, it is the same process of running with –check and –diff and then without, if it works. Here is the command:

 ansible-playbook -i ../picluster/inventory/mycluster/hosts.yaml cluster.yml \
--limit=lock --tags="control-plane" --check --diff \
-u ${USER} -b -v --private-key=~/.ssh/id_ed25519

After running w/o the check/diff, I verified on the node (lock) for the leader election settings, and check that the pod is running and log looks good:

grep leader-elect /etc/kubernetes/manifests/kube-controller-manager.yaml
grep leader-elect /etc/kubernetes/manifests/kube-scheduler.yaml

kubectl -n kube-system get pod kube-controller-manager-lock kube-scheduler-lock
kubectl -n kube-system logs kube-controller-manager-lock --tail=10

Once confirmed that it was successful, I repeated the above steps with each of the other control plane nodes. When running the ansible commands, watch the log for errors, and ensure that the step worked correctly. You may get unrelated errors that can be ignored (ask AI to check). In my case, the kube-vip play had a Calico error, and the control-plane play had a image error, but they were deemed unrelated:

TASK [network_plugin/calico : Assert that current calico version is enough for upgrade] ***************************************************

fatal: [lock -> rabbit]: FAILED! => {“msg”: “The conditional check ‘calico_version_on_server.stdout.removeprefix(‘v’) is version(calico_min_version_required, ‘>=’)’ failed. The error was: Input version value cannot be empty”}

TASK [kubernetes/control-plane : Install | Copy kubectl binary from download dir] *********************************************************

fatal: [lock]: FAILED! => {“changed”: false, “msg”: “Source /tmp/releases/kubectl-1.34.2-arm64 not found”}