Skip to content

Controller: Correctly identify other pods on shutdown. #13444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Controller: Correctly identify other pods on shutdown. #13444

wants to merge 5 commits into from
+325 −24

Conversation

DerekTBrown
Copy link

@DerekTBrown DerekTBrown commented May 29, 2025 •
edited
Loading

What this PR does / why we need it:

  • Load balancer IP cleared from all ingresses when upgrading nginx-ingress-controller #11087 is a serious footgun: if a user upgrades the helm chart version in-place, or adds pod labels (either via the helm chart or some other mechanism), ingress-nginx may shutdown an ingress even though there are other ingress-nginx pods running and serving traffic.
  • The root cause of Load balancer IP cleared from all ingresses when upgrading nginx-ingress-controller #11087 is that ingress-nginx re-uses the set of pod labels on the terminating pod to find other pods belonging to the same controller group. This is a bug, since there are cases where we expect the set of labels on the terminating pod to differ from the labels on other pods belonging to the same controller group (for example, during an in-place helm upgrade, the chart version label will be different on old vs. new pods).
  • ingress-nginx already has an appropriate primitive to track the pods belonging to the same controller group: electionID. We expect a pod to tear down an ingress iff all pods with the same electionID have gone away (i.e. meaning there will be no future successful master election).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • CVE Report (Scanner found CVE and adding report)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation only

This is a breaking change in two respects:

  1. I would argue all helm chart upgrades are currently breaking changes because of Load balancer IP cleared from all ingresses when upgrading nginx-ingress-controller #11087. This is dangerous for users until resolved.
  2. The PR enables this new behavior by default. I don't believe this is a breaking change for users, but would like other eyes to validate my assumption.

Which issue/s this PR fixes

How Has This Been Tested?

  • I have added unit tests to the helm chart to validate the new label is defined.
  • I have added unit tests to the status.go component to validate the pod finding behavior.

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I've read the CONTRIBUTION guide
  • I have added unit and/or e2e tests to cover my changes.
  • All new and existing tests passed.

Footnotes

  1. See: https://github.com/kubernetes/ingress-nginx/issues/11087#issuecomment-2712118775 ↩ ↩2 ↩3

@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 29, 2025
@netlify Netlify
Copy link

netlify bot commented May 29, 2025 •
edited
Loading

✅ Deploy Preview for kubernetes-ingress-nginx canceled.

Name Link
🔨 Latest commit 59e0c50
🔍 Latest deploy log https://app.netlify.com/projects/kubernetes-ingress-nginx/deploys/683891bb0359140008abe1ef

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/helm Issues or PRs related to helm charts needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 29, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: DerekTBrown
Once this PR has been reviewed and has the lgtm label, please assign strongjz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Contributor

Welcome @DerekTBrown!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot requested a review from Gacko May 29, 2025 14:53
@k8s-ci-robot k8s-ci-robot added the needs-kind Indicates a PR lacks a `kind/foo` label and requires one. label May 29, 2025
@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority labels May 29, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @DerekTBrown. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 29, 2025
Gacko
Gacko requested changes May 29, 2025
View reviewed changes
@DerekTBrown DerekTBrown requested a review from Gacko May 29, 2025 15:12
@DerekTBrown DerekTBrown marked this pull request as ready for review May 29, 2025 18:53
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 29, 2025
Gacko
Gacko requested changes Jun 11, 2025
View reviewed changes
Copy link
Member

@Gacko Gacko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna review the Go code later.

/retitle Controller: Correctly identify other pods on shutdown.
/triage accepted
/kind feature
/priority backlog
/hold

charts/ingress-nginx/templates/controller-daemonset.yaml
@@ -34,6 +34,9 @@ spec:
labels:
{{- include "ingress-nginx.labels" . | nindent 8 }}
app.kubernetes.io/component: controller
{{- if not .Values.controller.disableLeaderElection }}
nginx.ingress.kubernetes.io/electionID: {{ include "ingress-nginx.controller.electionID" . }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if camel case is the way to go here...

Suggested change
nginx.ingress.kubernetes.io/electionID: {{ include "ingress-nginx.controller.electionID" . }}
nginx.ingress.kubernetes.io/election-id: {{ include "ingress-nginx.controller.electionID" . }}

charts/ingress-nginx/templates/controller-deployment.yaml
@@ -40,6 +40,9 @@ spec:
labels:
{{- include "ingress-nginx.labels" . | nindent 8 }}
app.kubernetes.io/component: controller
{{- if not .Values.controller.disableLeaderElection }}
nginx.ingress.kubernetes.io/electionID: {{ include "ingress-nginx.controller.electionID" . }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here:

Suggested change
nginx.ingress.kubernetes.io/electionID: {{ include "ingress-nginx.controller.electionID" . }}
nginx.ingress.kubernetes.io/election-id: {{ include "ingress-nginx.controller.electionID" . }}

charts/ingress-nginx/tests/controller-daemonset_test.yaml
@@ -208,3 +208,30 @@ tests:
- equal:
path: spec.template.spec.runtimeClassName
value: myClass

- it: should create a DaemonSet with the electionID label on the pod template when leader election is enabled
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- it: should create a DaemonSet with the electionID label on the pod template when leader election is enabled
- it: should create a DaemonSet with election ID label

charts/ingress-nginx/tests/controller-daemonset_test.yaml
controller.disableLeaderElection: false
asserts:
- equal:
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
path: spec.template.metadata.labels["nginx.ingress.kubernetes.io/election-id"]

charts/ingress-nginx/tests/controller-daemonset_test.yaml
- it: should create a DaemonSet with the electionID label on the pod template when leader election is enabled
set:
controller.kind: DaemonSet
controller.disableLeaderElection: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd actually omit explicitly setting this, so the test fails as soon as someone changes the default.

charts/ingress-nginx/tests/controller-daemonset_test.yaml
- it: should create a DaemonSet with the custom electionID label on the pod template when controller.electionID is set and leader election is enabled
set:
controller.kind: DaemonSet
controller.disableLeaderElection: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here: I'd actually omit explicitly setting this, so the test fails as soon as someone changes the default.

charts/ingress-nginx/tests/controller-daemonset_test.yaml
controller.electionID: custom-election-id
asserts:
- equal:
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
path: spec.template.metadata.labels["nginx.ingress.kubernetes.io/election-id"]

charts/ingress-nginx/tests/controller-daemonset_test.yaml
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
value: custom-election-id

- it: should create a DaemonSet without the electionID label on the pod template when leader election is disabled
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- it: should create a DaemonSet without the electionID label on the pod template when leader election is disabled
- it: should create a DaemonSet without election ID label if `controller.disableLeaderElection` is true

charts/ingress-nginx/tests/controller-daemonset_test.yaml
controller.disableLeaderElection: true
asserts:
- notExists:
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
path: spec.template.metadata.labels['nginx.ingress.kubernetes.io/electionID']
path: spec.template.metadata.labels["nginx.ingress.kubernetes.io/election-id"]

charts/ingress-nginx/tests/controller-deployment_test.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as for the DaemonSet tests.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2025
@k8s-ci-robot k8s-ci-robot changed the title [#11087] Determine if a given controller is the last pod using electionId label Controller: Correctly identify other pods on shutdown. Jun 11, 2025
@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. kind/feature Categorizes issue or PR as related to a new feature. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Jun 11, 2025
@k8s-ci-robot k8s-ci-robot removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority labels Jun 11, 2025
@Gacko
Copy link
Member

Gacko commented Jun 11, 2025 •
edited
Loading

What we are looking for here is basically a way to determine all the other pods targeted by either the Deployment or the DaemonSet, right? Why don't we hand in the selector labels used in the matchLabels property to the controller? I know this is not available from the pod details, but could be handed in as an argument. Because in some cases people want leader election to be disabled but still have features working which rely on identifying other pods, right?

@Gacko
Copy link
Member

Gacko commented Jun 11, 2025

Reviewing the code once again, I'm asking myself if the whole stuff in status.go is even called if there's not leader election. As in: The whole mess we currently have is around the status not being updated correctly on shutdown. For correctly updating the status on shutdown, the controller needs to have write access and only one controller pod should do that at the same time, so there needs to be a leader.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gacko Gacko Gacko requested changes

@tao12345666333 tao12345666333 Awaiting requested review from tao12345666333

Assignees
No one assigned
Labels
area/docs area/helm Issues or PRs related to helm charts cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. priority/backlog Higher priority than priority/awaiting-more-evidence. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
3 participants
@DerekTBrown @k8s-ci-robot @Gacko