Deprecate support for RedHat 8 family hosts

[VannTen]

RHEL 8 (and derivates, like Almalinux 8, Rockylinux 8, and others) are starting to cause some problems:

• by default, they ship a kernel version which is below the ones recommended by kubeadm : [kubernetes] Support Kubernetes v1.32.0 #11862 (comment)
• the default python version is not supported by ansible-core 2.17 (and thus ansible 10) : Upgrade ansible from 9 to 10 (ansible-core from 2.16 to 2.17) #11519 (comment)
• (minor): they come with an old systemd version (v239) which would need some tweaks with containerd 2.0 for instance : [containerd] Support containerd v2.0.x #11845 (comment)

Another piece of information to consider is that the security support for RHEL8 and clones (alma, rocky) ends in 2029. Given the pace of kubernetes development, I don't think we will be able to support them until that date (cgroupsv2 is not mandated for now, but it might happen,)

So, this is to kick-off a discussion about a timeline to deprecate, and eventually remove, support for RHEL8 family (and also: Opensuse 15 I think ? -> in general OS with kernel < 5.8/10 and python < 3.7)

I see the following extreme possiblities:

• wait until 2029
• deprecate now as part of defaulting to K8s 1.32, remove when 1.31 is removed from our options (== 2.30 I think ?)

There is problably some room between those.

Slack thread on that subject on what other sig cluster-lifecycle projects are doing: https://kubernetes.slack.com/archives/C13J86Z63/p1736414039059959

@ant31 @floryut @yankay @tico88612 @mzaian

Others, feel free to chime in. I'm going to pin this issue for visibility.

[ant31]

Let's drop them and remove from CI.

Eventually we do as kops, have a flag to turn off preflight check with big warning -> close all future issues raised about it.

Not knowing what Kubespray users are using doesn't help.

[VannTen]

Yeah, that's basically why I meant by deprecation.
The removal would be: removing stuff from boostrap-os, preinstall, etc, stuff we know will definitely break on those distribs.

We already have the turn off prefligt, (with a variable) since #11710

[ant31]

let's go for it (removing support).
RHEL 8 and fam., can request to RH and other about bumping kernel in LTS or move on with newer version.

[vdveldet]

The best quote i have found grasping this issue through github PRs:
”we try not to special case distros, just handle Linux as a whole. if distro Foo does not comply with k8s you might want to move to a newer Foo or a different distro.”

[VannTen]

Yeah, I quoted this from neolit123 (one of kubeadm maintainers) in the 1.32 support PR. EDIT: Though kubespray does special case distro (to a point) because we need to install packages, etc. But this is another level.

[ErikJiang]

I believe many users are still using RHEL 8 and its derivatives, so I do not recommend an immediate deprecation. Users typically need time to adapt to such changes. Could we consider providing enough time for users to prepare for migration? During this period, we can maintain support for RHEL 8 as much as possible.

[ant31]

I believe many users are still using RHEL 8

Many user of Kubespray? we don't know.

The Kubespray maintainers are already short, with tones of distro supported, we just can't absorb such a high maintenance cost for others.
By the others I mean both the users of 'old' distro who are not upgrading, and the maintainers of those distro.

So far we've been doing best effort, go as far as possible when it doesn't add too much overload. Now, it hits a road blocker there's no need to continue. We did the same for debian 10 (#11347)

Users typically need time to adapt to such changes

They don't have to adapt, they'll won't be able to upgrade with kubespray or at their own risk (aka not tested).

In Kubernetes there's no LTS strategy

[VannTen]

I believe many users are still using RHEL 8 and its derivatives

Well, maybe ? We don't really have concrete data (getting that would help 👍)

Could we consider providing enough time for users to prepare for migration? During this period, we can maintain support for RHEL 8 as much as possible

That's what we are doing, considering. There are very concrete problems, though. In particular, the EOL of ansible-core 2.16 in 3 months is a bit alarming. We are not going to use an unsupported version for 5 years.

Users typically need time to adapt to such changes

Could you give examples of that ? In my experience (migrating clusters from rhel 7 to rhel 8) the impact of migration on applications was rather minimal (after all, that's the point of containers).

Migrating a cluster is some work, but it's the same magnitude than upgrading (while more involved). (And I'm only talking about live migration without downtime ; because cluster recreation or allowed downtime are certainly easier).

[tico88612]

@ErikJiang Kubernetes v1.32 only “suggests” a new version of the Linux kernel, and there are other ways to get around it. But the awkward thing is that RHEL 8 comes with Python 3.6, which can't be controlled by Ansible 10 (ansible-core 2.17) (which requires Python 3.7 or higher), and I don't think we can afford to stick with Ansible 9 for RHEL 8 for 5 years.

[ErikJiang]

from my perspective, there are still users using RHEL 8, although I apologize for not having specific data to support the scale of this user base. I also understand the burden of maintaining RHEL 8, and the compatibility issue between Python 3.6 and Ansible 10 is indeed a challenge. 😂 currently, it seems there isn't a good way to balance these factors.

[tmurakam]

Oops, some of my customers use RHEL8 so this will have a huge impact for me.
But I understand the situation and will talk to the customers...

EDIT: It is not difficult to avoid the Python version issue, because RHEL8 has 'python3.11' package. I'm using it with 'venv' to run the Kubespray on RHEL8.
Here are sample scripts: setup-py.sh, venv.sh

[tico88612]

@tmurakam I think your EDIT misunderstood this problem. It is related to the remote host's Python pre-install version, not the control host's.

ref: #11519 (comment)

[tmurakam]

@tico88612 Oh, I understood.

[vdveldet]

Given the fact that RHEL8 is still supported till 2029, this mean you can expect around 10 (or more) k8s releases.
Eventually support for the installation of k8s on RHEL8 will stop in my opinion, due to kernel incompatibility.
I just finished upgrading a customer from RHEL7 to RHEL9, using kubespray; really easy, so node upgrade is possible.
It is just work/time/effort/...
If reluctant against node upgrade is related to processes in place for VM upgrades, i can image companies implementing these processes will not tolerate, non supported software combinations.

[bbaassssiiee]

RHEL has long-time support, if you support it as a distro for Kubespray, then I suggest you adept if at all posssible, and yes I know that it can be a burden with all the moving parts.

You could create a vendor-branch rhel8 and backport useful features only if needed/asked for. Even if you stop working on it yourself, the rhel8 crowd can clone that branch to use Kubespray. With a branch-master for rhel8 it could be maintained.

[ant31]

RHEL has long-time support,

That's irrelevant for Kubespray's project or kubernetes.

Kubespray can't align with all LTS, it does't support RHEL as a whole but specific versions.
rhel8 will be removed as it's becoming incompatible as any other distribution that are in the same state.
If needed you can fork and maintain a rhel8 compatible version or use the latest kubespray release that supports it.

[yankay]

I'm not a Red Hat insider, but at my company, many of our users are on RHEL 8, and there are no immediate plans to upgrade to RHEL 9. Discontinuing support for RHEL 8 would significantly complicate matters as persuading customers to upgrade their Linux systems is extremely challenging and falls on me to manage.

While I believe it's best not to remove it before the end of life (EoL) of RHEL 8, but if the main branch decide to remove support for RHEL 8 and upgrade Ansible, we should still allow the community to contribute code snippets that ensure compatibility with RHEL 8 and Ansible versions earlier than 2.17, without disrupting the mainline.

This approach would allow Kubespray to use the new versions while also addressing legacy system compatibility issues.

[VannTen]

Well, yeah, release branches will still accept fixes, that's a given.

Discontinuing support for RHEL 8

This is effectively mostly the case already for upstream Kubernetes, anyway, given the kernel versions requirements mismatch with RHEL8

[yankay]

Well, yeah, release branches will still accept fixes, that's a given.

Discontinuing support for RHEL 8

This is effectively mostly the case already for upstream Kubernetes, anyway, given the kernel versions requirements mismatch with RHEL8

Thank you.

Given the unpredictability of the future, the best outcome would be for all the users to successfully upgrade their operating systems. :-)

If we decide to remove support for RHEL8 at last, I hope we can keep our options open. In the future, we might discuss how to maintain compatibility with RHEL8 for Kubespray (although I hope it doesn't come to that). For instance, we could consider merging a small portion of the code snippets into the main branch or maintaining them in a specific release version with minimal maintenance cost.

[tico88612]

However, based on a Kubernetes release cycle of about 4 months and 14 months of maintenance, the RHEL 8-based conservative estimate is that we will only maintain it for 1~2 years at most.

As Python 3.6 has been EOL for a long time, I think it makes sense for the Ansible community to drop RHEL8.

I can understand that RHEL 8 is used by a lot of people, but Kubespray is built on Kubernetes and Ansible, and we shouldn't get away from an actively maintained version.

With one month left before ansible-core 2.16 is deprecated, I thought 2.28 (K8s 1.32) could be a good time to notice and suggest Kubespray 2.29 (K8s 1.33) to remove all CIs for RHEL 8.

[ant31]

Yes @tico88612, let's plan that for next release.