Operator up but can't get cluster pod up

[ncclementi]

I'm on an EKS cluster with all the right policies. Using Kubernetes 1.32

I did this to get the operator up

kubectl create -f 'https://strimzi.io/install/latest?namespace=default'

then when I try to apply the this yaml file to get the cluster up https://strimzi.io/examples/latest/kafka/kafka-single-node.yaml
but it hangs in pending.

➜ kubectl get pods                                                                 
NAME                                        READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-6f4fc4667c-wlgqr   1/1     Running   0          96s

 ➜ kubectl apply -f kafka-single-node.yaml
kafkanodepool.kafka.strimzi.io/dual-role created
kafka.kafka.strimzi.io/kafka-cluster created

 ➜ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
kafka-cluster-dual-role-0                   0/1     Pending   0          10s
strimzi-cluster-operator-6f4fc4667c-wlgqr   1/1     Running   0          118s

when I check

kubectl describe pod kafka-cluster-dual-role-0 

I see at the bottom

Warning  FailedScheduling  105s  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Then when I check it's not marked as default but I thought if it's the only one it should be used (but it's not)

kubectl get storageclass

NAME   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2    kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  16m

I patched it to be marked as default doing

kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"[storageclass.kubernetes.io/is-default-class](http://storageclass.kubernetes.io/is-default-class)":"true"}}}'

I can't trigger a restart, so I deleted the pod and it triggers a restart but still in PENDING

but now when I check

kubectl describe pod kafka-cluster-dual-role-0 

I don't see any event but now, checking

kubectl get pvc

NAME                               STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-0-kafka-cluster-dual-role-0   Pending                                      gp2            <unset>                 16m

and

kubectl describe pvc data-0-kafka-cluster-dual-role-0

I see this event

Events:
  Type    Reason                Age                     From                         Message
  ----    ------                ----                    ----                         -------
  Normal  FailedBinding         11m (x26 over 17m)      persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
  Normal  ExternalProvisioning  2m19s (x23 over 7m49s)  persistentvolume-controller  Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Summarizing, the problem seems to be:

The Kafka pod (kafka-cluster-dual-role-0) is stuck in a Pending state because it cannot find a matching Persistent Volume (PV) for the PVC (data-0-kafka-cluster-dual-role-0).

The PVC is using the gp2 StorageClass, which is configured with the in-tree AWS EBS provisioner (kubernetes.io/aws-ebs).

However, my PVC has an annotation that expects the CSI AWS EBS provisioner (ebs.csi.aws.com), which it seems to be incompatible with the in-tree provisioner.

I'm fairly new to kubernetes and strimzi so I'm not sure how to solve this. Does anyone know why this happen and how can I solve this?

[scholzj]

What Strimzi does, is that it creates the PVC with the corresponding storage class (if you configured it in the Strimzi custom resource). The rest is up to your infrastructure.

In your case, it seems the PVC got assigned the storage class gp2. But it looks like your storage provisioner does not work. What is the exact cause is hard to say from what you provided. You can always try to delete the PVC and wait for Strimzi to recreate it - maybe it helps if this is just some one-time issue on the Kubernetes level. But otherwise you would need to talk check the Kubernetes / Storage driver logs or check with some Kubernetes cluster admin to see what the issue might be.

[ncclementi]

I wonder if having the quickstart using storage ephemeral but adding a note mentioning this is not encouraged for production, and put a disclaimer about when the pod dies you lose data ("ephemeral"). Or have maybe topics with 2 replicas might help?

I'm not sure if adding something just for the EKS scenario would be beneficial, as there can be different issues everywhere else (and we would not be able to cover all, right?).

This makes sense, maybe point out that you need to have access to create PV in the K8s deployment of your choice.

[katheris]

For the quickstart we could give more than one option for the command to deploy Kafka. I think making sure it works on EKS is useful, since that is probably a popular infrastructure choice, and we want the first experience to be a good one. @scholzj @im-konge are you happy for me to go ahead and raise an issue around this and we can decide there?

[scholzj]

It is not about working on EKS. It is about working on Kube clusters without (default) storage class. I'm fine if you go ahead. But the question is .... would users know if they have properly configured storage provisioning?

[ppatierno]

Tbh for a quick start guide I would have just ephemeral with more nodes. It should work out of the box without any issue related to the persistent storage. Losing data on pods restarting is fine imho, it's a quick start not a full production ready example. If we use persistent storage in the quick start, there are more possibilities that things go wrong (like in this case) and your quick start experience is awful.
Also I would not add specific docs around EKS, I agree with Jakub it's not EKS, it's Kubernetes itself. If you are using EKS you should be aware of how Kube works/is configured in EKS.

[im-konge]

+1 for having more nodes in the quick start together with ephemeral storage (as I anyway mentioned in my first comment).