KRaft controller pods restart in loop when having 3 controllers, but work fine with 1 controller

[artempol-dev]

KRaft controller pods restart in loop when having 3 controllers, but work fine with 1 controller

Description

I'm experiencing an issue with KRaft controller pods in my Strimzi Kafka cluster. When I configure the cluster with 3 controller pods, they continuously restart in a loop. However, when I use the exact same configuration but with only 1 controller pod, everything works perfectly fine with no restarts.

The issue happens during regular operation, not just during initial deployment. The restarts happen sequentially, so the cluster remains operational, but it's concerning behavior that shouldn't be happening in a properly configured cluster.

Environment

• Strimzi version: 0.45.0
• Kafka version: 3.9.0

Configuration

I'm using NodePools with the KRaft mode enabled. Here's my controller NodePool configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  annotations:
    meta.helm.sh/release-name: cdc
    meta.helm.sh/release-namespace: kafka2
  labels:
    app.kubernetes.io/component: kafka-cluster
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kafka-connector
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: kafka-connector-0.1.0
    strimzi.io/cluster: kafka-cluster-cdc
  name: controller
  namespace: kafka2
spec:
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 2048Mi
    requests:
      cpu: 100m
      memory: 2048Mi
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - class: standard-rwo-retain
        deleteClaim: true
        id: 0
        size: 30Gi
        type: persistent-claim
  template:
    pod:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: strimzi.io/pool-name
                    operator: In
                    values:
                      - controller
              topologyKey: kubernetes.io/hostname
      metadata: {}
      securityContext:
        fsGroup: 1001
        runAsNonRoot: true
        runAsUser: 1001

Events

LAST SEEN   TYPE      REASON      OBJECT                               MESSAGE
16m         Normal    Pulled      pod/kafka-cluster-cdc-controller-0   Container image ".../.../external/strimzi/kafka:0.45.0-kafka-3.9.0" already present on machine
16m         Normal    Created     pod/kafka-cluster-cdc-controller-0   Created container: kafka
16m         Normal    Started     pod/kafka-cluster-cdc-controller-0   Started container kafka
16m         Warning   Unhealthy   pod/kafka-cluster-cdc-controller-0   Liveness probe errored: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
16m         Warning   Unhealthy   pod/kafka-cluster-cdc-controller-0   Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
11m         Normal    Pulled      pod/kafka-cluster-cdc-controller-1   Container image ".../.../external/strimzi/kafka:0.45.0-kafka-3.9.0" already present on machine
11m         Normal    Created     pod/kafka-cluster-cdc-controller-1   Created container: kafka
11m         Normal    Started     pod/kafka-cluster-cdc-controller-1   Started container kafka
10m         Normal    Pulled      pod/kafka-cluster-cdc-controller-2   Container image ".../.../external/strimzi/kafka:0.45.0-kafka-3.9.0" already present on machine
10m         Normal    Created     pod/kafka-cluster-cdc-controller-2   Created container: kafka
10m         Normal    Started     pod/kafka-cluster-cdc-controller-2   Started container kafka

Troubleshooting Steps Taken

Verified that the broker pods are working fine - they don't have restart issues
Checked controller logs - no obvious errors explaining the restarts
Tried with a single controller - which works perfectly

Questions

Is this a known issue with multi-controller KRaft setup in Strimzi 0.45.0?
Could there be a problem with the quorum election or leader selection in the 3-controller setup?
Are there specific probe settings that need to be adjusted for a multi-controller setup?
Should I increase terminationGracePeriodSeconds or probe delays in controller pods?

Any help or guidance would be greatly appreciated!

[artempol-dev]

It was just a memory leak