[Bug]: Single node Kraft migration

[hamishforbes]

Bug Description

I am trying to migrate a single broker cluster to Kraft with dual role nodes and running into an error:
java.lang.NullPointerException: Cannot invoke "io.strimzi.operator.cluster.operator.resource.KafkaQuorumCheck.quorumLeaderId()" because "restartContext.quorumCheck" is null

Start state is a Kafka resource with node-pools enabled and kraft disabled.
1 broker replica and 3 zookeeper replicas

I followed the migration steps as in: https://strimzi.io/blog/2024/03/22/strimzi-kraft-migration/

Which leaves me with a broker nodepool with 1 replica and a controller nodepool with 3 replicas.

If i try to add the controller role to the broker nodepool I get the above error.

You also get this error if you try the migration process with a dual role pool

Steps to reproduce

1. Create a Zookeeper based cluster with a single broker
2. Follow the documented Kraft migration steps
3. Try to consolidate to a single combined broker/controller nodepool

Expected behavior

For this to work.
I get that its not a recommended configuration, but for dev/test environments it saves a lot of resources!

Strimzi version

0.41

Kubernetes version

v1.29

Installation method

Helm

Infrastructure

EKS

Configuration files and logs

This is the working Kraft configuration with 1 broker, 3 controllers.
Adding the controller role to the broker pool is where the problems start

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
  labels:
    app: kafka
  name: kafka
  namespace: dev
spec:
  kafka:
    config:
      auto.create.topics.enable: "true"
      default.replication.factor: 1
      delete.topic.enable: "true"
      log.flush.interval.messages: 1000
      log.retention.bytes: 1.073741824e+09
      log.retention.hours: 168
      num.partitions: 3
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
    - authentication:
        type: tls
      name: authtls
      port: 9093
      tls: true
      type: internal
    rack:
      topologyKey: topology.kubernetes.io/zone
    version: 3.7.0
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  labels:
    app: kafka
    strimzi.io/cluster: kafka
  name: kafka
  namespace: dev
spec:
  jvmOptions:
    -Xms: 128M
    -Xmx: 512M
  replicas: 1
  resources:
    limits:
      cpu: 1
      memory: 1536Mi
    requests:
      cpu: 150m
      memory: 768Mi
  roles:
  - broker
  storage:
    type: jbod
    volumes:
    - id: 0
      size: 50Gi
      type: persistent-claim
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  labels:
    app: kafka
    strimzi.io/cluster: kafka
  name: kafka-controller
  namespace: dev
spec:
  jvmOptions:
    -Xms: 64M
    -Xmx: 256M
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 300Mi
  roles:
  - controller
  storage:
    type: jbod
    volumes:
    - id: 0
      size: 50Gi
      type: persistent-claim

Operator error log

2024-05-30 02:01:53 INFO  KafkaRoller:397 - Reconciliation #934(watch) Kafka(dev/kafka): Will temporarily skip verifying pod kafka-kafka-0/0 is up-to-date due to java.lang.NullPointerException: Cannot invoke "io.strimzi.operator.cluster.operator.resource.KafkaQuorumCheck.quorumLeaderId()" because "restartContext.quorumCheck" is null, retrying after at least 4000ms

Additional context

No response

[ppatierno]

This is not a bug. Kafka doesn't support migration to dual-role node KRaft and it also stated in our documentation https://strimzi.io/docs/operators/latest/deploying#proc-deploy-migrate-kraft-str

[hamishforbes]

Hi, ah I did miss that, was just following the blog post.

Nevertheless, I've completed the migration to Kraft and the issue now is i want to downsize from 3 controllers to 1.
That gives the error I reported.

Is that not supported either? It's impossible to have a single controller without starting the cluster using Kraft from scratch?

[ppatierno]

Scaling controllers is again something missing in Kafka as well. So it's not possible to scale from 3 to 1.

[scholzj]

I assume you can migrate to a single controller node in the first place. But keep in mind that means no HA etc.

[ppatierno]

Yeah that's right. You can just have your controllers node pool created at the beginning of the migration with just one replica.

[veerendra2]

Hi @scholzj, we are also trying to migrate to KRaft, as next Strimzi version only supports KRaft. I see during the migrate we have to create dedicated controller KafkaNodePool.

My question is, can I remove this newly created controller KafkaNodePool after successful migration and then make the existing broker kafka node pool to duel role (both controller and broker)? Because I was hoping I can kafka cluster simple i.e. no extra dedicated pods for controller or zookeeper

[scholzj]

Hi @scholzj, we are also trying to migrate to KRaft, as next Strimzi version only supports KRaft. I see during the migrate we have to create dedicated controller KafkaNodePool.

That is correct. You will need to create a controller node pool. Just to clarify, this is how Kafka does it, so we just have to follow it. This is not a Strimzi limitation.

My question is, can I remove this newly created controller KafkaNodePool after successful migration and then make the existing broker kafka node pool to duel role (both controller and broker)? Because I was hoping I can kafka cluster simple i.e. no extra dedicated pods for controller or zookeeper

You can move to a mixed node(s) after you complete the KRaft migration. But given the Kafka limitations around the controller management, you should:

• Add the broker role to your controller(s) to create the mixed node(s)
• Move all data from the old broker(s) to the new mixed node(s) ... Cruise Control can be used for this (the remove broker feature of KafkaRebalance resource)
• Delete the old broker(s)