Nonsense and other useful things

O, all the things that can go wrong when replacing a failing NVMe card with a new one in the server. Based on that I wrote this poem, all by myself of course, not using AI (*not*). The poem below was written by the gpt-oss (20b) model running on my RTX 3090 which still worked after reenabling Intel VT-d. Somehow, I feel a bit like ‘Brave Sir Robin’ from Monty Python.

Continue reading →

During development it often occurs that with local testing, docker is used for building and running container images. Then, when working with a local kubernetes cluster such as for instance k3d, you also want to test out kubernetes deployment. This requires the local k8s cluster to be able to access the images. Unfortunately, the docker deamon provides its own API and does not provide a registry API. So this setup requires you to run a separate registry (in a docker container) and push images after building them to the local registry after which k3d can access them.

The workflow typically looks like this:

• build docker image and test locally without kubernetes
• push docker image to local registry (e.g. at localhost:5000)
• run tests on local kubernetes cluster

The first step can be automated easily using docker compose and by tagging the images that to be of the form localhost:5000/<imagename>. The second step is then docker compose push. However, in practice it is really easy to forget to do the second step and this can cost some time for troubleshooting. Wouldn’t it be nice if we could make it work without having to push manually every time after building?
Continue reading →

In April 2024 I successfully passed the CKS certification exam, but compared to the Certified Kubernetes Application Developer and Certified Kubernetes Administrator this was the toughest one yet. Not because the exam is particularly hard. The questions were all in closed form, I guess so that automatic grading is possible, but the main difficulties are that:

• there are many new topics such as runtime security with Falco, Seccomp, Apparmor, and gvisor, as well as security scanning tools such as trivy and kube-bench.
• the time pressure during the exam is high. I experienced that first hand by not being able to finish 1 of the questions. To go fast you need to prepare things very well. In particular, I setup my own single node kubernetes cluster using Vagrant so I could test out many topics using a standard kubeadm setup.
• the auditing during the exam was the worst experience yet. The proctor took more than 30 minutes to release the exam so I could start. Additionally, the proctor interrupted me twice, I think, for no good reason that caused a huge interruption of my flow.

Why I took the exam

Continue reading →

After the debacle with Rocky linux and Red Hat, I decided to move to a more future proof setup with my kubernetes cluster fully running on Debian. I chose Debian 12 at the time but unfortunately Nvidia did not yet support Debian 12 in their CUDA repository. Using the CUDA repository is the easiest way to install the Nvidia drivers and much better than running a script. See package manager installation on this Nvidia page. Therefore, I decided to use Debian 11 instead which was supported. That on its own gave some headaches in the setup and required some hacks to get it working. Also, looking at future migrations (considering e.g. replacing calico by cilium), it is a lot better to run the same OS on all kubenetes nodes.

Continue reading →

Upgrade for centos/rocky linux to new major versions is not supported. If you look for it there are procedures, but the changes between major versions can be so siginificant that this is not a good idea. Therefore, an approach I often use is to upgrade by installing the newer version next to the current version. Of course, I have also paritioned my old system to separate my home directory and other important directory into separate partitions so that the upgrade does not require copying user files.

This has some risks, since you might lose connectivity to the old linux version if the upgrade fails. Continue reading →

When I setup my kubernetes cluster using kubeadm some years ago, I decided to use a simple non-HA setup of kubernetes, because (1) it simplifies the setup and (2) the cluster will be running on a single server anyway. In fact I am using kubernetes mainly for deployment flexibility and not as much for high availability. Now I am running a mix of centos 8 stream and centos 7 nodes which will all be end of life in June 2024. With kubernetes, doing such an upgrade should be easy:

• add a new controller node on a newer OS and join it to the cluster
• remove the old controller
• replace worker nodes one by one

However, the first two steps will give problems since in a non-HA setup, the IP address of the API server is used by all components that connect to the API server. This will make it impossible to switch over to the new controller node in a transparent way.

Continue reading →