Hello Everyone,
Another posting season is upon us. Why do I state this now, Vanguard renewal? Canadian tax season on its way? No, I just woke up today to a massive thunderstorm with torrential rain and came to the conclusion that the 20 cm of snow and ice that lay on the ground literally a week ago must have either been a dream or “the times they are a changing!”.
So on that note lets talk Kubernetes. Stating anything is the best never really fits especially when it comes to IT. Some things fit better in some situations and others well in others.
A Myriad of Kubernetes options
When it comes to Kubernetes there is a myriad of options out there. If you are heading down the virtualization route with Kubevirt then a distribution like Openshfit https://www.redhat.com/en/technologies/cloud-computing/openshift or SUSE Virtualization (Harvester) https://www.suse.com/products/rancher/virtualization/ are most likely your best bet, at least for on premises. If you need hardcore security and immutability Talos, which even calls itself “The Kubernetes Operating System” https://www.talos.dev/ is a fantastic choice.
When it comes to labs, demos, and environments where you need lean and mean, K3s https://k3s.io/ in my opinion is top of the charts.
Legend has it that back in the day, and when it comes to Kubernetes we are talking a few years ago, not decades, a fellow by the name of Darren Shepherd at Rancher Labs (which was later acquired by SUSE) did not like how long it was taking him to fire up Kubernetes clusters while working on one of his projects. He decided to go into to the code and remove all the things that he considered superfluous and the end result was not only a drastically cut down distribution, but also a much simpler deployment process. The great thing is that almost all of the standard kubectl commands work exactly the same with K3s as with K8s. Also you deploy the distribution from a single binary. So what is the real deal with K3s?
Pros & Cons
- Lightweight and resource-efficient — single binary under 100MB, uses significantly less RAM and CPU than a full K8s install, making it ideal for edge devices, IoT, and small servers.
- Simple installation — can be set up with a single command (
curl -sfL https://get.k3s.io | sh -), compared to the multi-step process of bootstrapping a standard K8s cluster. - Batteries included — ships with Flannel, CoreDNS, Traefik, a local-path storage provisioner, and a service load balancer out of the box, so you get a functional cluster immediately.
- Fully conformant Kubernetes — passes CNCF conformance tests, so you get full API compatibility without sacrificing portability.
- Lower operational complexity — bundles all control plane components into a single process and uses SQLite by default, eliminating the need to manage a separate etcd cluster.
- ARM support — runs natively on ARM64 and ARMv7, making it a strong choice for Raspberry Pi clusters and other ARM-based hardware.
- Fast startup time — the cluster comes up in seconds rather than minutes, useful for development, CI/CD, and ephemeral environments.
- Automatic certificate management — handles TLS certificate generation and rotation internally without extra tooling.
- HA support — supports high-availability configurations with embedded etcd or an external datastore (MySQL, PostgreSQL) when you need production-grade resilience.
- Small attack surface — fewer components and dependencies compared to full K8s means fewer things to patch and secure.
- Great for dev/test parity — lets developers run a real Kubernetes environment locally that closely mirrors production without heavy resource requirements.
Now this would not be a fair article if I did not mention some cons:
- Limited default storage options — ships with only a local-path provisioner, so you'll need to set up something like Longhorn, Rook-Ceph, or an NFS provisioner for production-grade persistent storage.
- Single-process architecture can be a drawback at scale — bundling everything into one binary simplifies things, but it means you can't independently scale or tune individual control plane components the way you can with full K8s.
- Smaller community and ecosystem — while growing, the K3s community is smaller than mainline Kubernetes, so you may find fewer tutorials, forum answers, and third-party integrations tailored specifically to K3s.
- Not ideal for very large clusters — K3s is optimized for small to medium workloads. If you're running hundreds of nodes or thousands of pods, a full K8s distribution is generally a better fit.
- SQLite limitations — the default SQLite datastore doesn't support multi-server HA. You need to switch to embedded etcd or an external database for high availability, which adds back some complexity.
- Replaced components can cause surprises — if you're used to a standard K8s setup, the swapped-in defaults (Traefik instead of Nginx, Flannel instead of Calico, etc.) can lead to unexpected behavior or require extra work to replace with your preferred tools.
- Fewer built-in extension points — features like custom schedulers, advanced admission controllers, or certain cloud controller integrations may require more manual configuration than with managed K8s offerings.
- Version lag or mismatch — K3s releases trail upstream Kubernetes slightly, so the latest K8s features or patches may not be immediately available.
- Less enterprise support — while Rancher/SUSE backs K3s, the ecosystem of enterprise support, managed services, and vendor tooling is much deeper for standard K8s distributions like EKS, AKS, or GKE.
- Debugging can be harder — since everything runs as a single process, isolating issues in a specific control plane component (API server, scheduler, controller manager) is less straightforward than with a traditional multi-process setup.
Who uses it?
You can find K3s extensively used in these environments:
Edge and IoT
Telecom and 5G
CI/CD and development environments
Retail
Agriculture and industrial
Government and defense
Managed platforms
Ok Geoff enough talk how do I start!
I am not selling real estate in a Florida swamp zone I promise and yes this is Kubernetes, which you can deploy in this fashion if you so like https://github.com/kelseyhightower/kubernetes-the-hard-way or instead try k3s like this:
curl -sfL https://get.k3s.io | sh -Yes that is it! That is all that you need to do in order to be up and running with a simple single node cluster.
Of course there are many other options including adding extra control nodes for HA and worker nodes, but I will leave that for a future post.
Oh and I almost forgot, if you look at previous articles in this Kubernetes Korner group you will see an avalanche of posts that were based on K3s.
Nevertheless I thought it would be fun to do a step by step walk through of all the different K3s options out there so stay tuned. Of course the Kasten series will also be continued so the Korner will have an active season this year!
