[Bug]: Strimzi operator (v0.39) removes metadata from PersistentVolumeClaim on reconcile

[taylormgeorge91]

Bug Description

When certain metadata properties (e.g. finalizers, annotations, labels) are added to PersistentVolumeClaim managed by Strimzi, they are removed when the reconcile runs and patches the object.

These metadata properties are being added by Disaster Recovery services and the removal of these attributes are compromising those solutions working with these Strimzi services.

Desire: The additional metadata (finalizers, annotations, labels) added by the DR services should be preserved when Strimizi makes updates.

Strimzi version tested: 0.39

Steps to reproduce

1. Deploy Strimzi Kafka (0.39 prior to KRaft, etc -- provides Kafka w/ Zookeeper)
2. Verify Strimzi Kafka is ready and running with PVCs in place.
3. Add additional annotations, finalizers, labels to the Kafka/Zookeeper PVC(s).
4. Watch the Kafka/Zookeeper PVC(s) and observe the custom annotations/finalizers/labels are removed

Expected behavior

When additional metadata is added to the Kafka/Zookeeper PVCs, they are persisted and not reverted by Strimzi operator.

Strimzi version

0.39

Kubernetes version

OpenShift 4.18.8

Installation method

OLM

Infrastructure

IBM internal machines (VMs hosting OCP)

Configuration files and logs

Increased the Audit policy on OCP to provide WriteRequestBodies to surface the information involved in the API requests. For example:

# Set the Openshift cluster configuration to enable more details in the audit logs for CP4AIOps services - change the custom rule group property based on the namespace
$ oc get apiserver cluster -o yaml

apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    oauth-apiserver.openshift.io/secure-token-storage: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2025-05-06T18:25:58Z"
  generation: 2
  name: cluster
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    name: version
    uid: 29484658-2300-4659-889c-e45b6e1e5a30
  resourceVersion: "22887062"
  uid: 61014138-2fcc-4686-8ae1-793030b83a9d
spec:
  audit:
    customRules:
    - group: system:serviceaccounts:cp4aiops
      profile: WriteRequestBodies
    profile: Default

Then queried the API request logs as such:

# Extract the audit events from the OCP apiserver and filter by desired resource - pipe the output to a file for example
export MASTERS=$(oc get no -l node-role.kubernetes.io/master -o name --no-headers)
export NAMESPACE=$(oc config view --minify | grep namespace | cut -d" " -f6) # Uses the current context namespace 
export RESOURCE_TYPE="persistentvolumeclaims"
export RESOURCE_NAME="data-iaf"
for n in $MASTERS; do
	oc adm node-logs $n --path=kube-apiserver/audit.log | jq "select(.objectRef.namespace==\"$NAMESPACE\")" | jq "select(.objectRef.resource==\"$RESOURCE_TYPE\")" | jq "select(.verb==\"get\" | not)" | jq "select(.objectRef.name | contains(\"$RESOURCE_NAME\"))"
done

One such extracted API call:

events-audit-kafkapvc.json

Kafka CR Spec:

kafka-spec.txt

Additional context

No response

[Frawless]

hi, this is currently a known limitation and it requires implementation of server-side apply in Strimzi. We have proposal for that https://github.com/strimzi/proposals/blob/main/052-k8s-server-side-apply.md. However, we don't have any ETA for this feature yet (the behavior is the same in latest Strimzi release - 0.46).

[taylormgeorge91]

Thank you for the quick reply.

[scholzj]

It is not really "known limitation". This works as designed. Any labels and annotations you want to have on the resources managed by Strimzi can be set through the template section in the Strimzi CRs.

While server-side-apply might make things work differently in respect to this, it is an enhancement rather than fixing some known issues. (it is tracked in #6938, so I think this should be closed as duplicate or comverted to discussion)