Hi Folks,
I wasted a lot of time in my life doing dumb things. Luckily a remain young in spirit and continue to do dumb things even now. Like troubleshooting an ignition process on Flatcar when the issue was staring me in the face. When dealing with failures and errors be it in backup or anything else try the “metaphor” method in order to help you solve the problem.
For me it was that services were not starting when booting the immutable OS. Permission denied came up in the journalctl output but it made no sense since that part of the filesystem was chmod with the right access. Then I checked the location and the files were there with the correct names but… there size was zero. I checked the ignition file creation and all looked fine. I struggled until it hit me when on my walk and passing by a location that I used to frequent quite often.. the beer store.
If you are in line up to buy beer and you have brought your empty bottles to cash in on the recycling refund.. you can’t just take one of those bottles and quench your first right away. Those bottles are empty!! You need to get to the cash register first, hand in the old empties and get the new 24 pack of beer that is full. Then you can happily digest your beer.
So in my flatcar ignition script I had the Kubeadm service being deployed and doing its pre boot magical setup BUT.. I put the netword section after the systemd section. DUH!! without any network settings the ignition process could not reach the network and go out to download the binaries. So like me trying in the past to drink an empty bottle before purchasing the new full one :). Flatcar Linux, being of positive spirit and “lets just get it done” attitude nevertheless provided a “best effort” and created the empty files which in turn were useless since they were boneless binary files!
So just remember check the simple stuff first like what order your commands are in during your script if the process that you are dealing with works sequentially!!
P.S. hope to have a post out soon in the Kubernetes Korner on how you can use an old laptop to leverage KVM and Flatcar Linux to bring up a Kubernetes cluster on an immutable OS.
