[WIP] Release Proposal v2.27.1

[tico88612]

Changes by Kind

Feature

• Patch version updates for:
  calico: "v3.29.2"
  cilium_cli: "v0.16.24"
  cni: "v1.4.1"
  containerd: 1.7.26
  cri_dockerd: 0.3.16
  kubernetes: v1.31.7
  runc: v1.2.5 ([release-2.27] Patch versions update #12031, @VannTen)
• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. ([release-2.27] Add manual option to the external_cloud_provider variable #11884, @k8s-infra-cherrypick-robot)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. ([release-2.27] Bump ingress-nginx to 1.12.1 and certgen to 1.5.2 #12080, @tico88612)

Documentation

• Fix sample inventory for the reserved resource ([release-2.27] Updated sample in inventory #11922, @k8s-infra-cherrypick-robot)

Failing Test

• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl ([release-2.27] Fix CI by exclude the .ansible in .ansible-lint & remove ctr image pull workaround #11956, @k8s-infra-cherrypick-robot)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (Refactor control plane upgrades with reconfiguration support #12015, @chadswen)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary ([release-2.27] Fix incorrect syntax for secondary nodelocaldns manifest #11957, @k8s-infra-cherrypick-robot)
• [calico] Fix kubecontrollersconfigurations list permission ([release-2.27] [calico] fix: kubecontrollersconfigurations list permission #12039, @k8s-infra-cherrypick-robot)

[tico88612]

Blocking issue #12073

[VannTen]

I think we need to slip a backport of #12044 as well ; external etcd setup seems broken in some cases without it.

[farshadasadpour]

Blocking issue #12061

[arusa]

Currently control plane upgrades are failing and I hope this fix will be released soon with 2.27.1!

[VannTen]

Blocking issue #12061

This won't be backported: while a better default is good, this does not fix a bug.

@arusa You mean on release-2.27 ? Would you mind expanding on this preferably in a dedicated issue ? The linked PR is about reconfiguring during upgrade, and change quite a few things, so it's not an ideal target to backport (and I'm not sure how it would fix cluster upgrades 🤔 but maybe more details would clarify that ?).

[arusa]

@VannTen this is an issue from someone else, describing exactly what happened to me too when I tried to upgrade from 1.30.4 to 1.31.4: #12009

The problem is that the latest kubespray release 2.27.0 still uses kubeadm upgrade apply instead of kubeadm upgrade node to upgrade secondary control plane nodes.

So currently there is no kubespray release available that can successfully upgrade a cluster from 1.30 to 1.31.

This is why I commented here, just to make sure the bugfix will be part of the next release, to save others from upgrade problems.

[VannTen]

Thanks !
I wonder if a partial backport could fix the issue 🤔
@chadswen do you think we could backport 94deb52 only (part of #12015 ), or maybe the whole PR is not as risky to backport as I think ?

[chadswen]

Thanks ! I wonder if a partial backport could fix the issue 🤔 @chadswen do you think we could backport 94deb52 only (part of #12015 ), or maybe the whole PR is not as risky to backport as I think ?

We can backport it. I've tested the changes on older kubernetes releases already. For 2.27 we just need to revert the two changes where the v prefix was removed that were needed after #11890.

For 2.26, we may also need to remove the v1beta4 specific changes, but it should be straightforward as well.

[chadswen]

Thanks ! I wonder if a partial backport could fix the issue 🤔 @chadswen do you think we could backport 94deb52 only (part of #12015 ), or maybe the whole PR is not as risky to backport as I think ?

We can backport it. I've tested the changes on older kubernetes releases already. For 2.27 we just need to revert the two changes where the v prefix was removed that were needed after #11890.

For 2.26, we may also need to remove the v1beta4 specific changes, but it should be straightforward as well.

@VannTen I just created the upgrade reconfiguration fix backport PR for release-2.27 in #12103.

[tico88612]

Should we still need waiting for #12044?
v2.27.1 is ready to publish.