Well, it feels inevitable that this change was coming. Red Hat is starting to withdraw the source code of Red Hat Enterprise Linux (RHEL) from public availability. Instead, only the source code of CentOS Stream will be publicly available.
What does this mean? And why do I say this was inevitable? Let’s start by looking at the RHEL project, and how Linux is maintained in general. First, we have the generic Linux kernel project, the basis of Linux. From here, we get “distributions” or “distros”, such as Ubuntu, CentOS, and RHEL. We call these “downstream” of the Linux project. The goal of these distributions is to take the Linux kernel and customise the kernel in ways that best suit the purpose of the distribution. Applications and specific patches related to the integration of the kernel and bundled/supported software will be maintained at this level too, ensuring the best level of compatibility between these various components & configurations.
But we can have more than one layer of upstream & downstream with regards to a Linux distribution. Some Linux distributions use another distribution of Linux as their upstream, and then apply further tweaks and changes for their specific purposes, rather than starting from scratch at the Kernel level. RHEL does this, with the upstream Fedora project.
By having multiple upstream/downstream layers, we can create what is commonly referred to in software development as “update rings” or “deployment rings”. No matter what you call it, the output is the same. You could have one upstream project that is a “bleeding edge” or alpha/nightly focused build. This build is designed to have constant updates from developers and is a good playground to test changes and new features. Then you leverage the stable branches of these development builds to create a less-frequent, more stable distribution, cascading down layers as you see fit.
This is where CentOS and RHEL fit into the picture. For a long time, the RHEL distribution has been Open Source, and allowed publicly to pull its builds, as an upstream distribution. This is how CentOS operated for many years, even after Red Hat started maintaining the distribution themselves.
Then, two years ago, Red Hat decided that CentOS shouldn’t be downstream. Red Hat’s view was that CentOS was piggybacking off the stability & testing of their paying customers and Red Hat’s invested resources such as staff costs. This scenario was providing a competitive distribution, without a price tag. Instead, a new version of the CentOS brand would be created, known as CentOS Stream.
CentOS Stream is an upstream build of RHEL, residing downstream of Fedora and upstream of RHEL. CentOS Stream offers early access to RHEL’s proposed changes, and when they’ve reached maturity, they’re merged and accepted downstream by the RHEL distribution. Unfortunately, this means users are more likely to find changes and issues between builds than before, with general grumblings from the community around this, and a lack of support for the CentOS Stream distribution for most commercial software due to it being seen as nothing more than a beta build of RHEL.
When this change happened, we saw a rise in distributions such as Rocky Linux and AlmaLinux, promising to fill the gap left behind, offering downstream builds comparable to CentOS. These aren’t the only two distributions created, but they are arguably two of the most famous/popular distributions forked from RHEL.
And for the past two years, nothing has happened, Red Hat have allowed this, which makes you think, why did they bother with changing CentOS to CentOS Stream? Well, Red Hat have been gathering feedback on the rumblings of the CentOS change and have now decided the time is right to push forward and ensure that no distribution can freely access the source code to RHEL. This isn’t to say that the source code is closed source, but it will only be available to Red Hat customers and partners moving forwards.
Instead, Red Hat will be ensuring that only the CentOS Stream project will be available publicly, this means that distributions will still be able to have a fork of Red Hat’s development efforts, but they can’t just rebuild RHEL for free anymore.
This brings me to two crucial points, the concept of free, which I’ll discuss later, and the fate of distributions currently downstream to RHEL. Both Rocky Linux & AlmaLinux have committed to continuing development efforts beyond this announcement. Both releases claim to be bug-for-bug compatible with RHEL, though this will no doubt remain their goal, I don’t know how much longer they can achieve this. An interesting comment has been made within the AlmaLinux blog post discussing this change however, as one question asked was ‘Can you just use CentOS Stream sources?’, to which AlmaLinux’s reply states that they won’t, as they are committed to being a downstream build of RHEL. Rocky Linux, however, states they’ll continue to produce RHEL-compatible versions so long as the option remains. It’s clear that neither project was consulted on the upcoming change and they’re scrabbling for responses & agreements with Red Hat, which brings us back to the concept of ‘free’.
Red Hat place huge value in the work that they do, and I won’t dispute this. Red Hat contributes to the Linux Kernel itself, the upstream Fedora project, the upstream CentOS Stream project, and RHEL itself. That is clearly a huge amount of effort required, and it allows them to justify compensation for those efforts when combined with the commercial support layer they offer.
But those aren’t the only efforts involved in the stream of distributions that ends up within the final RHEL distribution. If we look at the Rocky Linux distribution as an example, the open-source community behind this also actively & willingly contributes to Fedora, CentOS Stream, and other open-source projects.
This is where things get messy. One of the two Red Hat blog posts so far talks about how Red Hat sees no value in downstream rebuilds of RHEL, and rather than interpret the intent for you, I’ll include the original quote below:
Instead, we’ve found a group of users, many of whom belong to large or very large IT organizations, that want the stability, lifecycle and hardware ecosystem of RHEL without having to actually support the maintainers, engineers, writers, and many more roles that create it.
Mike McGrath – Red Hat. Source: https://www.redhat.com/en/blog/red-hats-commitment-open-source-response-gitcentosorg-changes
And this is what one furious developer interpreted as Red Hat calling downstream users nothing more than ‘freeloaders’ in quite a damning summary of this latest change. Source: https://www.jeffgeerling.com/blog/2023/im-done-red-hat-enterprise-linux
My own take on this subject is that Red Hat are absolutely within their rights to avoid supporting low-value rebuilds of RHEL that simply don’t offer Red Hat any perceived return-on-investment for their development efforts. But I also believe that Red Hat are downplaying and dismissing the value added by enabling access to a stable branch of RHEL, from both the perspective of dismissing the efforts of all those supporting applications in making them RHEL compliant, and from enabling a wider audience the ability to see how RHEL works.
Regarding my last point there of how RHEL works, Red Hat state that you can create a free developer account to get developer instances of RHEL for your work, but this introduces barriers to entry as it’ll discourage more casual users from registering (as why should they?) and instead potentially steer them to other readily-accessible distributions such as Ubuntu, reducing the number of people with experience using RHEL, which could diminish the long-term value of the distribution.
Finally, to ensure you’ve got all the facts, here’s the blog post that started it all: https://www.redhat.com/en/blog/furthering-evolution-centos-stream.
So, what do you think? Are you using any of the affected distributions? Are you now going to have to rearchitect and redeploy your Linux platforms? If so, will you do as Red Hat want and buy the RHEL distribution, or turn to another flavour of Linux? Let me know in the comments below!
