Octoprint as a container, aka how to setup software via helm charts

  As I mentioned in Wifi trouble on an old NUC I recently had issues with my Raspberry Pi 4 that I used to run Octoprint. I was setting up the Raspberry Pi 4 and my Ender 3 V2 3D printer again after many months. The end goal is to print some fixtures for my new desktop CNC machine.

  In the process of re-setting up the Raspberry Pi 4 running Octopi. I had decided to upgrade the underlying distro Raspbian from buster to bullseye to bookworm. These are codenames for different Debian releases that Raspbian is based on. I was able to do that just fine via modifying the /etc/apt/sources.list file for each upgrade, and running apt-get update ; apt-get dist-upgrade. It was a slow process that took hours, but it went well.

  At the end of the I rebooted the Raspberry Pi 4, and found my ALFA AWLS036ACH wasn’t working. I then tried to re-compile the 8812au-20210629 driver, but ran into compiling errors. What I ended up finding is that the kernel for bookworm is compiled with LLVM. This then required running commands like LLVM=1 make oldconfig ; LLVM=1 make prepare to get the kernel configuration in a happy state for 8812au-20210629 to compile with 8812au-20210629. Compiling 8812au-20210629 was sped up by using make -j4 to take advantage of all four cores of the Raspberry Pi 4.

  Right as I got this finished my Raspberry Pi 4 died. I am not exactly sure what happened, but I am pretty sure that at least the SD card smoked. Maybe the Raspberry Pi 4 itself too. I say this, because the SD card was extremely hot to the touch when I went to examine both. Also both smelled like smoke. This is when I decided I was done with my Raspberry Pi 4 even if I could salavage it. This also happened at around 1am that night, and so then I went to bed.

  In the morning my new plan was to use one of three Intel NUC6i7KYB NUCs I had laying around as a replacement host for Octoprint. I picked one of them, and it already had Fedora Linux 36 installed on it. So I then upgraded it to Fedora Linux 37 and then Fedora Linux 38 using dnf system-upgrade download --releasever=37 ; dnf system-upgrade reboot and dnf system-upgrade download --releasever=38 ; dnf system-upgrade reboot.

  The next step was to get Octoprint setup. I had already done some research and found the octoprint docker image. But I am not a fan of running software in Docker. It isn’t that I am not a fan of containers. It is just that I greatly prefer to run things in Kubernetes using Helm charts. I have been running SABnzbd, Sonarr, and vaultwarden in Kubernetes for years. Here is a list of software that helps with those.

Kubernetes helpers

  • cert-manager, LetsEncrypt certificate management
  • external-dns, software to glue Kubernetes services, load balancers, and ingress controllers to DNS records in your favorite DNS provider like Route53
  • ingress-nginx, an easy to use nginx based ingress controller, and not to be confused with kubernetes-ingress formerly known as nginx-ingress
  • metallb, easy to use Kubernetes load balancer provider, and works with ingress-nginx
  • openebs, Kubernetes storage driver, aka CSI driver, that lets you create EBS volume like persistent volumes

  I have previous used pure upstream Kubernetes based on Kelsey Hightower’s Kubernetes the Hard Way. Which is how I still run the above services today. But since I set that up k3s has sigificantly matured, and has become the clear front-runner for Kubernetes outside of a cloud. I have also used k0s before, but found some design flaws.

  I then became to setup everything with the commands below. The commands should work on more Linux distributions with the exception of the dnf command. Though I did provide an alternative method. That the process should be so universal is part of the magic of Kubernetes.

All the pods running in k3s/Kubernetes:

kubectl get pods -A
NAMESPACE       NAME                                            READY   STATUS    RESTARTS         AGE
kube-system     svclb-ingress-nginx-controller-f9b322c2-xsqtv   2/2     Running   12 (4h15m ago)   2d17h
cert-manager    cert-manager-5468bbb5fd-p7wkr                   1/1     Running   8 (4h15m ago)    2d18h
kube-system     coredns-77ccd57875-dg9jd                        1/1     Running   7 (4h15m ago)    2d20h
default         octoprint-55d456799d-pjqms                      1/1     Running   5 (4h15m ago)    2d5h
cert-manager    cert-manager-cainjector-6f455799dd-fk27b        1/1     Running   24 (4h15m ago)   2d18h
ingress-nginx   ingress-nginx-controller-5f99db95d8-989f5       1/1     Running   6 (4h15m ago)    2d17h
kube-system     local-path-provisioner-957fdf8bc-p75pz          1/1     Running   10 (4h15m ago)   2d20h
cert-manager    cert-manager-webhook-54bd8d56d6-zxwzk           1/1     Running   8 (4h15m ago)    2d18h
kube-system     metrics-server-648b5df564-2ss84                 1/1     Running   11 (4h15m ago)   2d20h
metallb         metallb-speaker-kv8zj                           4/4     Running   58 (4h15m ago)   2d17h
metallb         metallb-controller-784fd54657-twxz7             1/1     Running   8 (4h15m ago)    2d17h

Octoprint web ui with an SSL certificate:
Octoprint

DNS lookup showing the A record for octoprint.cygnusx-1.org using the first ip address from the MetalLB range:

nslookup octoprint.cygnusx-1.org
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	octoprint.cygnusx-1.org
Address: 192.168.54.33

  In the end if you have setup a DNS record in your domain, and maybe setup port forwarding through your router to the system running Octoprint, then you should be able to go to https://octoprint.yourdomain.com/. Note k3s has it’s own local-path-provisioner built-in as an alternative to openebs.

  Note this could be used setup one or many different Helm charts for all kinds of software. Have a look at whats available on ArtifactHub or GitHub. It also isn’t hard to write your own Helm chart for any Docker image you can find on DockerHub.

Commands:
Run via sudo or as root for setting up k3s, ingress-nginx, cert-manager, metallb, and k8s-at-home to setup Octoprint:

curl -sfL https://get.k3s.io | sh - # Not a fan of this style at all, but it is k3s's standard install method. There are alternative methods.
dnf install helm # Installed helm from Fedora. Alternatively can be downloaded from the GitHub links above.
cd
mkdir $HOME/.kube
ln -s /etc/rancher/k3s/k3s.yaml .kube/config # Setting up ~/.kube/config for using kubectl with k3s
helm uninstall traefik -n kube-system
helm uninstall traefik-crd -n kube-system # Uninstalling traefik, k3s's default ingress-controller
kubectl delete pods --field-selector=status.phase=Succeeded -n kube-system # Cleaning up Completed pods
helm repo add k8s-at-home https://k8s-at-home.com/charts/
helm repo update
helm upgrade --install octoprint k8s-at-home/octoprint -f octoprint-values.yaml
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx -n ingress-nginx ingress-nginx/ingress-nginx --create-namespace
helm repo update
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace --values ingress-nginx.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.12.4 --set installCRDs=true # 1.12.4 is the latest at this time.
helm repo add metallb https://metallb.github.io/metallb
helm update
helm upgrade --install  metallb metallb/metallb -n metallb --create-namespace
kubectl create -f electron-route53-secret-access-key.yaml
kubectl create -f clusterissuer.yaml
kubectl create -f certificate.yaml
kubectl get certificate # You want to see the certificate READY status as TRUE. 
kubectl logs `kubectl get pod  -A | grep cert-manager | grep -Ev 'cainjector|webhook' | awk '{ print $2" -n "$1 }'` -f # If you need to see what is going on with the cert-manager and your certificate
kubectl get certificate -o yaml # Another useful command
kubectl create -f metallb-resources.yaml
kubectl create -f ingress.yaml

Files:
octoprint-values.yaml:

enableServiceLinks: false # Octoprint will break without this

env:
  CAMERA_DEV: /dev/video0 # My Logitech 4k BRIO webcam
  #ENABLE_MJPG_STREAMER: false # Can be useful to uncomment when first setting up Octoprint if you don't have a webcam or don't have one plugged in
  TZ: America/Phoenix # Change to your timezone

image:
  tag: 1.9.2 # Current latest Octoprint version

persistence:
  data:
    accessMode: ReadWriteOnce # Don't use ReadWriteMany
    enabled: true
    size: 1G # Creating a 1gb persistent volume for Octoprint to persisent in. Without this you will get the setup wizard everytime you restart the pod.

ingress-nginx.yaml:

controller:
  extraArgs:
    default-ssl-certificate: default/electron-tls # This sets a default SSL certificate that comes from cert-manager. Change this to match your certificate's secretName name, not the metadata name.

electron-route53-secret-access-key.yaml:

apiVersion: v1
data:
  secret-access-key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQ== # echo -n 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' | base64
                                                                              # ^ base64 encoding
                                                                              # This is an example secret-access-key from AWS's documentation.
                                                                              # echo -n 'd0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQ==' | base64 -d
                                                                              # ^ base64 decoding
kind: Secret
metadata:
  name: electron-route53-secret-access-key # electron is the system's hostname
  namespace: cert-manager
type: Opaque

clusterissuer.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  annotations:
  name: letsencrypt-electron # electron is the system's hostname
spec:
  acme:
    email: letsencrypt@cygnusx-1.org # Set this to your email address
    privateKeySecretRef:
      name: electron-issuer-account-key # electron is the system's hostname
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        route53:
          accessKeyID: AKIAxxxxxxxxxxxxxxxx # Replace this
          region: us-west-2 # Set this based on what region your route53 zone lives in
          secretAccessKeySecretRef:
            key: secret-access-key
            name: electron-route53-secret-access-key # electron is the system's hostname
      selector:
        dnsZones:
        - cygnusx-1.org # Set this to your domain

certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: electron # electron is the system's hostname
spec:
  # Secret names are always required.
  secretName: electron-tls # Set this consistently everywhere

  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - Cygnus X-1 # Change this, and feel free to make some name up
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 4096
  usages:
    - server auth
  dnsNames:
    - octoprint.cygnusx-1.org
  issuerRef:
    name: letsencrypt-electron # Make this match the issuer's metadata name
    kind: ClusterIssuer
    group: cert-manager.io

metallb-resources.yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  creationTimestamp: null
  name: default
  namespace: metallb 
spec:
  addresses:
  - 192.168.54.33-192.168.54.35 # Pick a small range outside your DHCP range
status: {}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  creationTimestamp: null
  name: l2advertisement1
  namespace: metallb 
spec:
  ipAddressPools:
  - default
status: {}
---

ingress.yaml:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
#    external-dns.alpha.kubernetes.io/hostname: octoprint.cygnusx-1.org # I am not using it, but this is how you could use external-dns with your domain.
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  name: octoprint
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: octoprint.cygnusx-1.org # Change this to be your domain
    http:
      paths:
      - backend:
          service:
            name: octoprint
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - octoprint.cygnusx-1.org # Change this to be your domain
    secretName: electron-tls # Make this consistent everywhere

Additional Links:
Ingress controllers / API gateways:

Certificate management:

Load balancers:

DNS:

CSI drivers:

Monitoring:

Packaging:

Images:

Continious deployment:

Terraform:

Auto-scalers:

General: