Skip to main content

Kubernetes Korner

  • 452 Topics
  • 1,545 Comments

Events

michaelcade
michaelcadeProduct Strategy
Welcome to Kubernetes Korner

I wanted to start by welcoming everyone to our new Kubernetes Korner where we can discuss all things cloud-native, Kubernetes and DevOps. More importantly its where we can come to share our experiences in learning this still relatively new world and ask questions of our fellow community members. We are also hoping to gather feedback around the Kasten K10 platform and Open-Source projects so that we can better understand the product strategy and feedback on where to go next and how to improve the overall experience. We also want this to be a community space to ask your Kasten questions, I will be active in here as well as some of the Kasten product managers and then I expect we have a few community members that are also hands on with Kasten K10 daily that can offer their advice and solutions. I also wanted to kick off the Korner with a question to you all, each, and every one of us will have a different background and learning journey when it comes to Kubernetes and DevOps. I want to kn

99016
V
C
ChrisWongNew Here
Kasten Backup repository sizing

Hi I am looking for some guidance for the backup data export sizing with Kasten. Let’s say, my case has 10TB production data, daily backup with 30 restore points, 5% delta change daily. I assume the compression and deduplication will save up 50%. Does it mean that I need to have at least 10TBx30x50% =150TB for my backup repository ? I do see in a lab that the second backup is much smaller than the first backup. Is there any incremental backup / global deduplication in the backup data export behind the scene ? Chris

6274
K
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Container Security Playground

Hi Everyone, Kubernetes is built on containers and to try something new I decided to setup a security playground leveraging two different container environments. IMPORTANT! Only run these tests and tools on your own test environments. If you run them anywhere else you must get permission for testing purposes. Security software will detect attacks and can get you in trouble so learn at home in the privacy of your own testing environment! Today will will setup a test vulnerable application called juice-shop from Owasp in a podman container. We will then setup in Proxmox a LXC Kali Linux container which going forward we will use to test the application juice-shop for vulnerabilities. Kali Linux is a fantastic security distribution that can be used to test and probe all sort of IT components. Today we will have some fun with the juice-shop but going forward you just know that we will attack a Kubernetes cluster! First we will go get the Kali Linux LXC containter: https://www.kali.org/get-k

100
Geoff Burke
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Podman backups?

Backup, Backup, Backup. That is our mantra. So what about Podman? I use to scripts to backup my Podman containers that I will admit I wrote with the help of AI 😀 since my coding skills are not enviable. podman-backup.sh #!/bin/bash # Podman Container Backup Script# This script creates application-aware backups of Podman containers# Run this script before performing VM backups # Set backup directoryBACKUP_DIR="${HOME}/podman_backups/$(date +%Y%m%d_%H%M%S)"LOG_FILE="${BACKUP_DIR}/backup.log"CONTAINER_INFO_DIR="${BACKUP_DIR}/container_info"IMAGES_DIR="${BACKUP_DIR}/images"VOLUMES_DIR="${BACKUP_DIR}/volumes"CONFIG_DIR="${BACKUP_DIR}/config" # Create backup directory structuremkdir -p "${BACKUP_DIR}" "${CONTAINER_INFO_DIR}" "${IMAGES_DIR}" "${VOLUMES_DIR}" "${CONFIG_DIR}" # Start loggingexec > >(tee -a "${LOG_FILE}") 2>&1echo "Starting Podman container backup at $(date)"echo "Backup location: ${BACKUP_DIR}" # Check if running with sudo/rootif [ "$EUID" -ne 0 ]; then echo "⚠️ WARNING: Not r

461
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Rancher Desktop 1.18.2 Is out and of course with AI

Hi Everyone, Going forward I am going to focus more on Kubernetes/Podman/Docker with AI. Not because I am jumping on the AI hype wave, or because I am completely an Anti-AI-er 😂, but instead because there are things happening in this space and as my guest Marino Wijay said on the last Kubernetes Korner show: AI and Cloud Native are definitely going to have a common path going forward. It did not take long for Rancher Desktop to jump on the bandwagon. As of Version 1.17 and following a similar effort by Podman Desktop https://podman-desktop.io/docs/ai-lab Rancher Desktop has now launched its own AI extension https://docs.rancherdesktop.io/tutorials/working-with-llms/. The extension is using OpenWebUI https://www.openwebui.com/, ollama and by default leveraging Tinyllama LLM, it will also discover any other LLMs you may have downloaded so you can choose one of them instead if you choose to. Interesting that they have websearch and Rag baked in using SearXNG which I had never heard of ht

321
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Be Kind with Podman and Cloud Provider Kind!

KIND, otherwise known as Kubernetes in Docker is a well known testing environment that many people use to get practice on Kubernetes. However, with changes to Docker licensing and advancements with Podman I wanted to see if I could run KIND on Podman. It turns out this is not only possible but some of the massive networking headaches when trying to expose applications to your network or even local host have also been effectively solved with Cloud Provider Kind https://github.com/kubernetes-sigs/cloud-provider-kind Today I will install a Kind cluster and Kasten on my Arch Linux laptop running only podman (not Podman Desktop). I should then be able to leverage ingress in Kasten to access the dashboard. To install Podman you can find the instructions here: https://podman.io/docs/installation There is a Kind Arch Linux package https://aur.archlinux.org/packages/kind and for your distribution check the Kind documetation: https://kind.sigs.k8s.io/docs/user/quick-start/ First I will install a

732
Geoff Burke
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Quo Vadis Kubekorner?

Disclaimer: This post is 100 percent AI free, real Organic Irish/German braincell fed! Quo Vadis? or Where are you going? Is my question. What are the plans for this year’s Kubernetes Korner? First of all I am going to try to pull in some of the incredible Kubernetes talent in the Toronto area for the kubekorner show, stay tuned. Second of all there are some exciting Kubernetes AI topics in the works which I plan to start posting about. Kasten continues to develop and move forward so naturally will continue to post on that topic. One question would be whether or not we should expand and do more topics related to containerization in general, like Podman, Docker, LXC? On the non Kubernetes front you could possibly, if the stars align, soon be in for a big surprise. How about getting the Rickatron ​@Rick Vanover and Cadezilla ​@michaelcade together somewhere on a show? You never know but I have heard rumors! If you have any ideas about what content you would like to see here please shoot

334
Geoff Burke
michaelcade
michaelcadeProduct Strategy
Kollect: A Modern Take on RVTools for Cloud Environments

A couple of months ago I posted about Kollect and introducing this as a tool to provide an inventory of your Kubernetes clusters and cloud based resources in Microsoft Azure and AWS. During the festive period I decided to tackle one of the remaining areas I wanted to include as an option to inventory, this was to collect a Veeam Backup & Replication inventory using the tool via the Veeam API but hiding some of the complexities that this brings. Veeam Backup & Replication REST API Reference I will start off by sharing the Veeam RESTAPI reference here. Swagger Not really relevant for you to know to use the tool but will give you a good idea on how I was able to understand the APIs available for us to show the data in Kollect. I think I first wrote about Swagger in around 2016 so it is a tried and tested way to expose, test and describe your APIs. Here I used it to understand the API endpoints available and how to use them in the tool “Swagger allows you to describe the structure of your

1194
Geoff Burke
michaelcade
michaelcadeProduct Strategy
Kollect - A Cloud & Kubernetes Inventory tool

I have been building, I use the term building loosely. I noticed there was a gap in the world of cloud and cloud native when it comes to gaining an inventory of all resources and objects in a specific platform or environment. The project is called Kollect, because we are going to collect the inventory using existing configuration on the system, there is no way to input credentials to access natively in the binary. The K is because this project started out with a focus on Kubernetes… and all Kubernetes app have to start with a K. That’s it! Who remembers RVtools? RVtools is useful tool known and loved across the VMware community which can quickly provide an inventory of your vSphere environment. I took this tool as the inspiration for my hacking something together. Getting Started You can get started with looking ath the project here. You can either follow the readme instructions and this will involve building from source and this would require golang to be installed on the system you w

3155
michaelcade
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
End of the year bragging :)

Hi Folks, I love communities and Veeam is the best (well wait.. oops I forgot I know run one so I will say tied for best just in case the boss is reading :) ) What a great end of year present to be again included in this group as well at open-appsec. Data Protection and Security now live side by side, and with Kasten you know Kubernetes is in there too!! https://www.openappsec.io/ambassadors-list

191
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
December too lat for a new Harvest? Not at all. Let's install Harvester on retired Desktop

For my post last week I had installed Harvester from Suse nested in a Proxmox VM. I must admit I that I was so taken with it that I decided to get it going on bare metal. Mind you the bare metal that lives in my basement generally are poor old desktops that worked night and day probably on excel spreadsheets or on atrociously boring financial PowerPoint presentations. Nevertheless perhaps they too dreamed one day of using their computational power for a real good cause! 😀 However I digress. My bare metal server is actually a beefed up Optiplex dell which was purchased on some amazon green Thursday sale or something like that. Super Bare Metal First things first, I boot off of my Ventoy Flash and choose Harvester Ventoy Next press enter again to start the installation: Installation I will ignore the hardware checks, after all this is a lab: Proceed I will create a new Harvester cluster. (My apologies for the photo quality, first time I am living off the camera and not screenshots). New

430
Geoff Burke
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Kasten 7.5 keeping SUSE VMs Alive!

So folks it finally happened. I backed up and restored a Virtual Machine with Kasten! The Kasten 7.5 announcement stated that they had added Suse Virtualization (Harvester) to the support list. Since I had always wanted to try Harvester this seemed like the perfect time to give this a try. Setting up Harvester nested in a Proxmox VM turned out to be simpler than simple. Normally with new Kubernetes endeavors I do my yoga deep breathing stuff before starting as you often run in to rabbit holes of new problems and deep tech that you have to do some quick deep dive learning on. This was not the case with Harvester where everything seemed to follow Veeam’s old Motto: “It Just Works” . Here are the instructions to setup Harverster https://docs.harvesterhci.io/v1.4/install/index If you are nesting in Proxmox make sure to select a cpu type of Host. The VM console will look like this when up and ready to go: The Harvester Dashboard is slick and creating a Virtual Machine was relatively simple

390
Geoff Burke
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Kasten 7.5 Before we go to far.. Where is Grafana?

Hi folks, As promised a quick blog post about Kasten 7.5. Keep in mind that ​@mattbator will be doing a webinar on 7.5, make sure to sign up! https://www.veeam.com/webinar/new-capabilities-for-kubernetes-data-protection I wanted to go over today Grafana, or better put the fact that grafana is no longer included in the default Kasten installation. However, they have been kind enough to provide us with setup instructions that you can find here: https://www.veeam.com/kb4635 I will admit I was a bit of a dare devil and just upgraded and am install grafana after the fact. They do warn you to install but … First we need to add the grafana helm repo chart: helm repo add grafana https://grafana.github.io/helm-charts helm repo update Next I will create namespace, you could include this in the helm install command but old habits.. kubectl create namespace monitoring Next comes a very important part. You need to create a values file with the specific values that you want for this setup and of cou

261
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
7.5 out of the hive!

Here it is. Watch this space :)

222
mattbator
A
ali.kComes here often
cannot restore namespace on EKS

Dears, i am facing issue with restoring a namespace showing the below error, can anyone advise, noting that all the pods were restored in running state but not in ready state phases: - attempt: 3 endTime: 2024-10-22T15:02:43Z errors: - cause: '{"cause":{"cause":{"cause":{"cause":{"message":"Specified 4 replicas and only have 0 available replicas: could not get Deployment{Namespace: default-restore, Name: dbs-ac-accessgroup-integration-service-accessgroupintegsvc}: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline"},"fields":[{"name":"deployment","value":"dbs-ac-accessgroup-integration-service-accessgroupintegsvc"}],"file":"kasten.io/k10/kio/kube/workload/workload.go:38","function":"kasten.io/k10/kio/kube/workload.WaitForWorkloadReady","linenumber":38,"message":"Deployment not in ready state"},"fields":[{"name":"namespace","value":"default-restore"},{"name":"name","value":"dbs-ac-accessgroup-integration-service-accessgroupintegsvc"}],"file":"kaste

193
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
The Kube Vanguard Smallstep, Cert-Manager, Kasten Multi-Cluster and Veeam 100 Berlin

The Kube Vanguard is back!

223
dips
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
KASTEN A TO Z Multi-Cluster Setup

There are many cases where we have numerous clusters to manage and importantly backup. Today we will setup a multi-cluster data protection setup for our Kubernetes clusters. This process has been made significantly easier in recent editions of Kasten. SETUP We will start off on the Multi-Cluster page of one of our Kubernetes clusters. For demo purposes I am leveraging two separate K3s clusters. Multi-Cluster On our first cluster we will press Promote to Primary which will turn this cluster into the Primary cluster of the group: Name the Cluster Confirm After a few moments we will be presented with this screen displaying our Primary Cluster in the list: Multi-Cluster Page On the left we now have a new Menu where we can create global policies, profiles and other components that can span all of our clusters. ADD ANOTHER CLUSER To add another cluster we first need to get a join token. The last menu item on the left is “Join Tokens”. When we click on this we can retrieve the join token: Joi

400
Geoff Burke
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
CNCF Canada East Meetup

Hi Everyone, I have to admit one of the reasons that I love going to these various meetups is because of the fantastic locations/buildings where they are held. Yesterday’s CNCF Kubernetes Canada East Meetup was held in a great building right down at the Water Front 207 Queens Quay West. I would have taken more photos but I notice the security guard in the office giving me strange looks while I walked around looking for better angles, obviously not the expected behavior of a Kubernetes nerd! These meetups are also great as we get to see familiar faces. Both leaders of the Toronto AI and Cybersecurity meetup were there as were other AI and Kubernetes stars from the huge local booming IT community. The Meetup Pizza came first As always everyone was given a rundown on all the latest Kubernetes news. Specifically Kubernetes version 1.31 Eli was discussed. He are some photos of the major topics: The removal of Kustomize from kubectl came as a surprise to me personally. We then had 3 great ta

80
Geoff Burke
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
KASTEN A To Z ... Why not OOTBI? SPECIAL DAY

Hi Everyone, The Kasten A to Z series continues, our goals is to have a place where someone can go from start to finish in their Kubernetes backup journey. Why would you want to start this journey in the first place? The use of containerized workloads is on the rise and all predictions point to Kubernetes gaining more and more traction in the years to come. Where there are workloads there need to be backups, and where there are backups there need to be skilled data protection professionals! I think this would work well as lyrics to a song: K10 A to Z why not Oobti? Anyways, I think I missed the boat on my rock and roll career, just by a tad. Follow along and sing the data resilience song ! Zero Trust is a Must or else your workloads will go bust! So why is this a special day? Well because for all on premises Kubernetes workloads this is one way to keep your data safe. SNAPSHOTS ARE NOT BACKUPS Remember what your parents said about VMware back in the day? Snapshots are not backups! Fast

361
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Kubehuddle Toronto Workshop

Hey Folks, I am attending a very interesting workshop for KubeHuddle attendees here in Toronto. A lot of security stuff on a Saturday! https://ca.kubehuddle.com/kubehuddle-workshop-day

151
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Kasten A to Z Day 7 Exporting Snapshots to a Location Profile

I am back after learning how to Fly Hi Everyone, We are back to the Kasten A to Z series. There was a delay due to a short vacation and my hitting a pothole on my bike and learning how to fly. Well my flight lasted about as long lived as my first ever pod on a Kubernetes cluster and ended with a massive crash minus the loop. My write hand can now type again so off I go. Snapshots are not Backups ! There is a famous saying “Snapshots are not backups” and this applies to Kasten as well. While we can do local snapshots on the cluster these have only limited value. If you lose the cluster you lose the snapshots. Location Profiles and Exports Kasten allows us to export local snapshot to Location profiles. These can be on local S3 or NFS or Cloud instance. If you are leveraging the Vsphere-csi-driver then you can even export the data portion of your snapshots to a Veeam Repository. Keep in mind that in the latter case you would still need to leverage one of the other types of location profil

481
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
K8sGPT

I just completed the Kodekloud course on K8sGPT and immediately went to playing with it in one of my local clusters. K8sGPT can analyze your lusters and tell you what it things is wrong. You can also filter it results in numerous ways. I will be using Ollama in the background so that I can use for free and local. A great blog that helped me out can be found here: https://addozhang.medium.com/k8sgpt-ollama-a-free-kubernetes-automated-diagnostic-solution-d453b63f112f You can even write your own analyzers as the github page states (https://github.com/k8sgpt-ai/k8sgpt) “K8sGPT uses analyzers to triage and diagnose issues in your cluster. It has a set of analyzers that are built in, but you will be able to write your own analyzers.” Another great feature is that K8sGPT allows you to Anonymize your output removing any cluster sensitive details. First we need to install Ollama. I am going to run the binary on the control plane node: curl -sSL https://ollama.com/install.sh | sh ollama -v versi

211
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
2 super cool things which distracted me away from my VMCA studying :)

Ok Folks, You know how it is, you have something on the calendar and need to focus solely on that and right during those moments on cue the little devil in your ear whispers, “hey check out these 2 utilities that you have not seen yet”. Well, I managed to fight off temptation long enough to pass my VMCA exam but you can bet that right after cycling back home instead of resting my exhausted brain, it was time to turn on the CLI!! Devilish distraction number 1 “BAT” So you use cat to checkout files? Did you known that a cat can add a bat and then you end up with that: batcat pod.yaml batcat You can find more info here: https://www.cyberciti.biz/open-source/bat-linux-command-a-cat-clone-with-written-in-rust/ Devilish distraction number 2 “KindScaler” I have to admit that I could not wait with this one and jumped away from the Veeam User Guide for a good 30 minutes to indulge. The thing is that if you use Kind (Kubernetes in Docker) for testing it is a pain if you setup a cluster and then

371
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
Kasten A to Z Day 6 Upgrading Kasten

UPGRADING KASTEN I was going to talk about exporting local snapshots via location profiles today.However, when I logged in to my system I saw an alert that a new version of Kasten was released so thought it better to go over upgrading Kasten. This will be a short post because upgrading Kasten is very simple. We have an Alert When a newer version of Kasten becomes available you will notice an alert in the upper right hand corner of the WebUI. Alert! If you click on the circle it will give you more details about its nature. In this case we a being told about a new version of Kasten. Alert Information If you click on the "See Kasten Upgrade Docs" button it very conveniently takes you to the instruction page: Kasten Documentation In fact you can simply go to "Settings - System Information" in the WebUI and the Cluster information will display that an upgrade is available: System Information Further down there is an "Upgrade Status" Section which also contains the Helm commands needed to pe

161
Chris.Childerhose
Geoff Burke
Geoff BurkeVeeam Legend, Veeam Vanguard
The Kube Vanguard is Diving into AI... Oh no... Ollama!

Hi folks, I was out paddle boarding in the sun all morning and must have been hit hard by the rays as when I got back I dived into AI. Well not quite I bought a course a few weeks back on Udemy have been turtling through it. So I decided to have some fun today on the Kube Vanguard.

231
Chris.Childerhose

About this group

Welcome to Kubernetes Korner!

This is the starting point for joining and contributing to the Kubernetes community - sharing your learning journey, discussing your data management requirements within Kubernetes, having solution discussions with SMEs and providing feedback to product teams.

 

Leaders:

Geoff Burke

Community Manager at Object First. 20 years experience in IT. 13 years working with Data Protection. Worked with top Canadian Service Providers leveraging Veeam Cloud Connect. Recently has been become intensely focused on Kubernetes. Certifications include VMCA2022 VMCE2023, KCNA, CKAD, CKA

Follow him via https://www.linkedin.com/in/geoff-burke-0a174684/

 

Members

NikolaPejkova
Chris.Arceneaux
falkob
mkevenaar
Rick Vanover
KorP
AndreAtkinson
victorwu
Chris.Childerhose
wesmrt
eprieto
Viperian
michaelcade
DChiavari
Alessio Di Benedetto
haslund
chas
Mildur
MicoolPaul
regnor
tsmith
vNote42
marco_s
Eddie Kwok
Timothy Dewin
Raffaele Valensise
J
Aly Idriss
BertrandFR
Luiz Eduardo Serrano
+ 261
Terms & Conditions