Skip to main content

Well, it feels inevitable that this change was coming. Red Hat is starting to withdraw the source code of Red Hat Enterprise Linux (RHEL) from public availability. Instead, only the source code of CentOS Stream will be publicly available.

What does this mean? And why do I say this was inevitable? Let’s start by looking at the RHEL project, and how Linux is maintained in general. First, we have the generic Linux kernel project, the basis of Linux. From here, we get “distributions” or “distros”, such as Ubuntu, CentOS, and RHEL. We call these “downstream” of the Linux project. The goal of these distributions is to take the Linux kernel and customise the kernel in ways that best suit the purpose of the distribution. Applications and specific patches related to the integration of the kernel and bundled/supported software will be maintained at this level too, ensuring the best level of compatibility between these various components & configurations.

But we can have more than one layer of upstream & downstream with regards to a Linux distribution. Some Linux distributions use another distribution of Linux as their upstream, and then apply further tweaks and changes for their specific purposes, rather than starting from scratch at the Kernel level. RHEL does this, with the upstream Fedora project.

By having multiple upstream/downstream layers, we can create what is commonly referred to in software development as “update rings” or “deployment rings”. No matter what you call it, the output is the same. You could have one upstream project that is a “bleeding edge” or alpha/nightly focused build. This build is designed to have constant updates from developers and is a good playground to test changes and new features. Then you leverage the stable branches of these development builds to create a less-frequent, more stable distribution, cascading down layers as you see fit.

This is where CentOS and RHEL fit into the picture. For a long time, the RHEL distribution has been Open Source, and allowed publicly to pull its builds, as an upstream distribution. This is how CentOS operated for many years, even after Red Hat started maintaining the distribution themselves.

Then, two years ago, Red Hat decided that CentOS shouldn’t be downstream. Red Hat’s view was that CentOS was piggybacking off the stability & testing of their paying customers and Red Hat’s invested resources such as staff costs. This scenario was providing a competitive distribution, without a price tag. Instead, a new version of the CentOS brand would be created, known as CentOS Stream.

CentOS Stream is an upstream build of RHEL, residing downstream of Fedora and upstream of RHEL. CentOS Stream offers early access to RHEL’s proposed changes, and when they’ve reached maturity, they’re merged and accepted downstream by the RHEL distribution. Unfortunately, this means users are more likely to find changes and issues between builds than before, with general grumblings from the community around this, and a lack of support for the CentOS Stream distribution for most commercial software due to it being seen as nothing more than a beta build of RHEL.

When this change happened, we saw a rise in distributions such as Rocky Linux and AlmaLinux, promising to fill the gap left behind, offering downstream builds comparable to CentOS. These aren’t the only two distributions created, but they are arguably two of the most famous/popular distributions forked from RHEL.

And for the past two years, nothing has happened, Red Hat have allowed this, which makes you think, why did they bother with changing CentOS to CentOS Stream? Well, Red Hat have been gathering feedback on the rumblings of the CentOS change and have now decided the time is right to push forward and ensure that no distribution can freely access the source code to RHEL. This isn’t to say that the source code is closed source, but it will only be available to Red Hat customers and partners moving forwards.

Instead, Red Hat will be ensuring that only the CentOS Stream project will be available publicly, this means that distributions will still be able to have a fork of Red Hat’s development efforts, but they can’t just rebuild RHEL for free anymore.

This brings me to two crucial points, the concept of free, which I’ll discuss later, and the fate of distributions currently downstream to RHEL. Both Rocky Linux & AlmaLinux have committed to continuing development efforts beyond this announcement. Both releases claim to be bug-for-bug compatible with RHEL, though this will no doubt remain their goal, I don’t know how much longer they can achieve this. An interesting comment has been made within the AlmaLinux blog post discussing this change however, as one question asked was ‘Can you just use CentOS Stream sources?’, to which AlmaLinux’s reply states that they won’t, as they are committed to being a downstream build of RHEL. Rocky Linux, however, states they’ll continue to produce RHEL-compatible versions so long as the option remains. It’s clear that neither project was consulted on the upcoming change and they’re scrabbling for responses & agreements with Red Hat, which brings us back to the concept of ‘free’.

Red Hat place huge value in the work that they do, and I won’t dispute this. Red Hat contributes to the Linux Kernel itself, the upstream Fedora project, the upstream CentOS Stream project, and RHEL itself. That is clearly a huge amount of effort required, and it allows them to justify compensation for those efforts when combined with the commercial support layer they offer.

But those aren’t the only efforts involved in the stream of distributions that ends up within the final RHEL distribution. If we look at the Rocky Linux distribution as an example, the open-source community behind this also actively & willingly contributes to Fedora, CentOS Stream, and other open-source projects.

This is where things get messy. One of the two Red Hat blog posts so far talks about how Red Hat sees no value in downstream rebuilds of RHEL, and rather than interpret the intent for you, I’ll include the original quote below:

Instead, we’ve found a group of users, many of whom belong to large or very large IT organizations, that want the stability, lifecycle and hardware ecosystem of RHEL without having to actually support the maintainers, engineers, writers, and many more roles that create it.

Mike McGrath – Red Hat. Source: https://www.redhat.com/en/blog/red-hats-commitment-open-source-response-gitcentosorg-changes

And this is what one furious developer interpreted as Red Hat calling downstream users nothing more than ‘freeloaders’ in quite a damning summary of this latest change. Source: https://www.jeffgeerling.com/blog/2023/im-done-red-hat-enterprise-linux

My own take on this subject is that Red Hat are absolutely within their rights to avoid supporting low-value rebuilds of RHEL that simply don’t offer Red Hat any perceived return-on-investment for their development efforts. But I also believe that Red Hat are downplaying and dismissing the value added by enabling access to a stable branch of RHEL, from both the perspective of dismissing the efforts of all those supporting applications in making them RHEL compliant, and from enabling a wider audience the ability to see how RHEL works.

Regarding my last point there of how RHEL works, Red Hat state that you can create a free developer account to get developer instances of RHEL for your work, but this introduces barriers to entry as it’ll discourage more casual users from registering (as why should they?) and instead potentially steer them to other readily-accessible distributions such as Ubuntu, reducing the number of people with experience using RHEL, which could diminish the long-term value of the distribution.

Finally, to ensure you’ve got all the facts, here’s the blog post that started it all: https://www.redhat.com/en/blog/furthering-evolution-centos-stream.

So, what do you think? Are you using any of the affected distributions? Are you now going to have to rearchitect and redeploy your Linux platforms? If so, will you do as Red Hat want and buy the RHEL distribution, or turn to another flavour of Linux? Let me know in the comments below!

This will be interesting to see how things go.  We use RHEL so I have passed this on for information.  Great post Michael.


Long as you’re using RHEL rather than a downstream like Rocky Linux you’ll be fine 🙂


Even though I’m just tapping into Linux the past yr or so, I was wondering how RH could do what they did with the ‘free’ versions….and for so long. As you stated Michael, doing so was at a real cost (develoopment, maintenance, updating, distribution, etc, etc.). I’m not surprised they ‘folded’ into what it seems they decided in moving forward. I can for sure see both sides’ arguments.

Thankfully, we don’t use RH in-house. So no worries at my org.


Yeah, it is mainly RHEL and Ubuntu typically.  We did use and still use I believe CentOS too.  I always liked the other flavors for testing in the homelab so will see what comes next there. 🤔


Great post @MicoolPaul with a clear explanation. I saw rumblings about Red Hat making some changes but didn’t really delve deeper so this explains it all. 

It’s going to be interesting and wonder if others may start following suit. 


meh.. not really an issue. The kernel is opensource so anyone can build on top of it. If anything this could be quite positive as developers will move over to Deb distributions and concentrate their focus there. RedHat was my first Linux distribution back in 1998 but later I wandered to Slackware, over to the BSD world, then back to debian and deb based. I only dealt with Centos at some jobs and for the LF Linux Administrator's exam :) 

All of this is just greedy IBM :) 

 

 


Thank you for the background/history lesson on this as I wasn’t aware of how this all worked.  I’ve not used CentOS heavily at all, and it’s been years since I’ve touched it.  At my old employer, we were a cloud service provider and hosted several RHEL VM’s, but I’m not sure there was much downstream.  It’s interesting to see how all of this works.  That said, when I’ve needed linux, I’ve generally used Ubuntu or even Photon OS in one instance.  


Great article, Michael. This move doesn’t surprise me either, but man it feels like it spits in the face of what GNU is all about. I take issue with Apple building an OS on top of Free BSD, and selling it at a profit. I see GNU as sacred and needs to in all forms be free. If you use it and build something off of it, I feel like you forfeit the right to profit from it. That could be ignorant thinking on my part, but I feel like profiting off a foundation of work that is designed to be free and open source, defies what it is all about.


Great article, Michael. This move doesn’t surprise me either, but man it feels like it spits in the face of what GNU is all about. I take issue with Apple building an OS on top of Free BSD, and selling it at a profit. I see GNU as sacred and needs to in all forms be free. If you use it and build something off of it, I feel like you forfeit the right to profit from it. That could be ignorant thinking on my part, but I feel like profiting off a foundation of work that is designed to be free and open source, defies what it is all about.

I thought Apple had moved off the FreeBSD kernel? Maybe not. Yeah I won’t even comment anymore, in some of the other communities that I am in the sycophants are out in force arguing that blah blah and blah blah just like when Elonian Muskrat blew 40 billion on Knitter in order to have tantrum like a 10 year old. 🙂 .  All baloney and ballet boots 🤣This is IBM so very simple like the Flying Wizards sang 

 


Wait you can tell I am too young.. it was Barret Strong actually 

 


Great article, Michael. This move doesn’t surprise me either, but man it feels like it spits in the face of what GNU is all about. I take issue with Apple building an OS on top of Free BSD, and selling it at a profit. I see GNU as sacred and needs to in all forms be free. If you use it and build something off of it, I feel like you forfeit the right to profit from it. That could be ignorant thinking on my part, but I feel like profiting off a foundation of work that is designed to be free and open source, defies what it is all about.

I thought Apple had moved off the FreeBSD kernel? Maybe not. Yeah I won’t even comment anymore, in some of the other communities that I am in the sycophants are out in force arguing that blah blah and blah blah just like when Elonian Muskrat blew 40 billion on Knitter in order to have tantrum like a 10 year old. 🙂 .  All baloney and ballet boots 🤣This is IBM so very simple like the Flying Wizards sang 

 

They may very well have, but they still made billions off of it in the past, and that bothers me immensely. I do see the flip side of this, though. RH puts tons of work into it’s product and they need to keep themselves afloat. “Highly suggest” donations. Maybe lock some enterprise features behind a paywall (similar to going from the home version of an OS, to the enterprsie/pro version). Make your base OS usable and sufficient for the average user, but have a paywall or donation need, for enterprise features. I think that’s fair. It walks that line of what open source is, but I get the bills have to be paid and the lights kept on.


Last one I promise. this topic reminds me of bike to work playlist that I had on this morning riding through the wildfire smog… when you get tired of all made up blarney that corporations put out 🙂 best Punk group ever…

 

 


@Geoff Burke has some eccentric playlists I think.  Need to share them out!  I love me some variety...plus I become a captive listener when I run, so I tend to discovery some new stuff.


Comment