Kubernetes debugging and pvc backups

  Yesterday I was doing miscellaneous on my storage server, aka my NAS. On it I run two pods in Kubernetes that utilize pvcs, aka persistent volume claims. In my setup these are managed via OpenEBS. Somehow some wires got crossed, and both pods had written to each others pvcs. This was obvious when looking into the pvcs, because some of the files where named for software a and some for software b. Both pvcs had both sets of files. My best guess is they actually got swapped by OpenEBS.

  What I ended up doing was making a backup of both the .img files in /var/openebs. I then mounted them with commands mount file.img /mnt/a -o loop. This then allowed me to access the contents of the image files. I then deleted the old pvcs, and made new ones. I then started a special pod that attached to the new pvc, and used kubectl cp to get the data back in. In the end I then restarted the original pods with the new pvcs by the same names.

  This then highlighted hey, you might want to backup the contents of pvcs. I found multiple solutions to this problem like velero, stash, and more. The problem I found with these is they where extreme overkill. They want to backup to some cloud storage service like S3. I just want to copy the data out of the pvcs. They also require their own helm charts so they can run as a pod. I am pretty sure with enough research and time I can bend them to my will, but I was curious if I could come up with a simpler solution.

  In the process of researching how to do that I re-ran across ephemeral containers and debug containers. I have heard of these before, and understand the basic concept. But I hadn’t really worked with them. I just looked it up and it seems that they have only been usable for a little over a year, since Kubernetes 1.25 came out. I also know they had continued to gain features since then.

  The use of debug containers is such a game changer with Kubernetes. Historicially you have had only a few choices. These choices all have their downsides.

  1. Put all the tools you wanted to ever use in your container and make it very fat
  2. Put all the tools you wanted to ever use in a different container in your pod, but have it always running
  3. Put all the tools you wanted to ever use in a different container in your pod, not have to run it all the time, but have to restart your pod to add the container

  The big advantage of debug containers is they can container whatever you want from any one docker image including custom ones you make for yourself. So if you want an Ubuntu docker image with every tool known to man, you can make it and use it. But even with debug containers there is a huge limitation that seems to make its usefulness very limit. That limitation is that without jumping through some hoops your debug container’s filesystem and your pod’s main container’s filesystem can’t see each other.

  In comes /proc/1/root to the rescue. The path /proc/1/root in the debug container filesystem is a link to your pod’s main container’s filesystem. My first thought was to run something like mkdir /mnt/root ; mount /proc/1/root /mnt/root -o bind to bind mount it to /mnt/root. This runs into permission problems. I wasn’t able to get this to work just yet, but I get the impression from everything that I read that using the kubectl debug’s command line argument --profile, this will probably be possible at some point in the future. Instead the solution that works today is ln -s /proc/1/root /mnt/root. This will then allow you to use commands like ls /mnt/root or cd /mnt/root. So at this point you have all the power you built into your debug container but with complete filesystem access to your pod’s main container’s filesystem.

  Yet another trick I learned along the way is you can use debug containers to get the rough equilivent of SSH access to your Kubernetes workers/nodes/instances. Given that you can pick your container, and hence it’s contents, this too is extremely powerful. It is also very convenient. Here is a documentation for node shell session.

  Coming back to the topic of backups. I will preface this with these caveats.

  1. This is definitely not the best way to go about this. That is doubling true if it is a production environment. See the other solutions I mentioned above, or research the many others.
  2. This way doesn’t guarantee the consistency of the files you backed up. If the process writes to the files as you are backing them up, parts of the backup may be unsable. Even worse if the files are database files, where you need max consistency across all of them.

  I had originally thought about using a volume mount. It would have used a directory on the host. The problem with this idea is I didn’t see a way of actually doing this ephemerally. I could do it by always mounting a directory from the host into the pod, but that isn’t what I want. I wouldn’t be surprised if this feature gets added in the future.

  I built my own docker image based on ubuntu. In it I installed openssh-client. Yes, there are probably at least 20 other choices you could find on DockerHub. I made my own, because my original plan was to include more packages. I ended up not needing them for now. Yet I expect I will end up growing this image over time, and making it my goto debug container docker image.

  I started trying to use rsync to copy files from /config in the pvcs to the host. I quickly realized given I wanted the user on the host to be chrooted, rsync was just going to make it harder. So I ended up using scp in sftp mode. This is for two reasons. One, it is the latest best practice, and two the host has a new enough version of OpenSSH that it was the easist choice. Here is the Dockerfile for my docker image, and my upload.sh script. I leave how to deal with OpenSSH authentication as an excercise for you.

Dockerfile:

FROM ubuntu

RUN apt-get update ; apt-get -y install openssh-client

COPY upload.sh /usr/local/sbin

RUN ln -s /proc/1/root /mnt/root

upload.sh:

#!/bin/bash

CONTAINER=$1
SSH_USERNAME=$2
SSH_HOST=$3
CONTAINER_DIRECTORY=$4

USER_HOST="${SSH_USERNAME}@${SSH_HOST}"
TIMESTAMP=`date -I`
REMOTE_TIMESTAMP_DIR="/backups/$CONTAINER/$TIMESTAMP"

cd /mnt/root
ssh-keyscan -H ${SSH_HOST} 2>/dev/null >> ~/.ssh/known_hosts 
mkdir /tmp/${TIMESTAMP}
# Hack to workaround scp's inablity to make nested remote directories
# If the remote side wasn't a chroot limited to sftp, you could use ssh
scp -qrs /tmp/${TIMESTAMP} ${USER_HOST}:${REMOTE_TIMESTAMP_DIR} 
scp -qrs ${CONTAINER_DIRECTORY} kube-backups@${SSH_HOST}:${REMOTE_TIMESTAMP_DIR}${CONTAINER_DIRECTORY}

Things you will need to setup on a remote server:

  1. A server with OpenSSH installed
  2. A user on the server using something like adduser kube-backups.
  3. Make that user’s shell /sbin/nologin
  4. Leave the default home directory for the user
  5. Somewhere on the remote server you need a directory for the chroot, like /var/kube-backups
  6. Run chown root:kube-backups ; chmod 750 /var/kube-backups to set the permissions properly
  7. Run mkdir /var/kube-backups/backups
  8. Run chown kube-backups:kube-backups /var/kube-backups/backups

Append to /etc/ssh/sshd_config:

Subsystem kube-backups internal-sftp 
   Match Group kube-backups
   ChrootDirectory /var/kube-backups
   ForceCommand internal-sftp
   X11Forwarding no
   AllowTcpForwarding no

Command to restart sshd:

systemctl restart sshd.service

  OpenSSH’s chroot feature is picky about the user you are logging in as must not own the root of the chroot. Yet you do need a directory in the chroot for the user to write into, otherise the setup is useless.

  Now for the magic command:

kubectl debug -it sabnzbd-0 --image=docker-registry.cygnusx-1.org/tools:0.1.4 --target=sabnzbd -- upload.sh sabnzbd kube-backups storage /config

Output:

Targeting container "sabnzbd". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-vtw2s.
If you don't see a command prompt, try pressing enter.
Session ended, the ephemeral container will not be restarted but may be reattached using 'kubectl attach sabnzbd-0 -c debugger-vtw2s -i -t' if it is still running

Usage:

kubectl debug -it (name of pod to backup) --image=docker-(path to your docker image) --target=(name of the container in your pod to backup) -- upload.sh (directoy on the remote server to write the backups into) (remote username) (remote hostname) (container directory to backup)

  Once you have successfully backed up something from a pvc you should be able to see the data on the remote server.

ls /var/kube-backups/backups
sabnzbd
ls /var/kube-backups/backups/sabnzbd
2023-10-08
ls /var/kube-backups/backups/sabnzbd/2023-10-08
config