PVCs are shown as 'Out of Sync' in ArgoCD

[bonzonkim]

Using Strimzi Operator version 0.45.0
with this yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: log-aggregator-kafka
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        kraftMetadata: shared
        deleteClaim: false
---

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  labels:
    strimzi.io/cluster: log-aggregator-kafka
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        kraftMetadata: shared
        deleteClaim: false
---

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: log-aggregator-kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    metadataVersion: 3.9-IV0
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          bootstrap:
            host: kafka.my-kafka.com
          brokers:
            - broker: 0
              host: broker-0.my-kafka.com
            - broker: 1
              host: broker-1.my-kafka.com
            - broker: 2
              host: broker-2.my-kafka.com
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
  entityOperator:
    topicOperator: {}
    userOperator: {}

Everything deployed well, but the PVCs are shown as 'Out of Sync'.

I've searched for similar issues, but there doesn't seem to be a clear solution yet.
Has anyone else run into this?
If you have, I'd appreciate if you share with me.

[see-quick]

First, please share the logs of the Kafka cluster and Cluster Operator so we can look at the issue. Thanks @bonzonkim. Now I can only guess what could be a problem :).

[bonzonkim]

Thank you for your reply here are some logs from kafka nodes and strimzi operater

broker logs

k logs -f -l strimzi.io/broker-role=true | tspin
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-1-raft-io-thread]
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-2-raft-io-thread]
2025-05-07 10:06:59,935 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=187247, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-1-raft-scheduler0]
2025-05-07 10:06:56,014 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=187247, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-0-raft-scheduler0]
2025-05-07 10:07:09,621 INFO [RaftManager id=0] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-0-raft-outbound-request-thread]
2025-05-07 10:39:07,630 INFO [LocalLog partition=mcmp-logs-0, dir=/var/lib/kafka/data-0/kafka-log0] Rolled new log segment at offset 4174061623 in 10 ms. (kafka.log.LocalLog) [data-plane-kafka-request-handler-4]
2025-05-07 10:39:07,921 INFO [LocalLog partition=mcmp-logs-0, dir=/var/lib/kafka/data-0/kafka-log1] Rolled new log segment at offset 4174061623 in 279 ms. (kafka.log.LocalLog) [ReplicaFetcherThread-0-0]
2025-05-07 10:39:07,922 INFO [ProducerStateManager partition=mcmp-logs-0] Wrote producer snapshot at offset 4174061623 with 0 producer ids in 1 ms. (org.apache.kafka.storage.internals.log.ProducerStateManager) [ReplicaFetcherThread-0-0]
2025-05-07 10:39:07,631 INFO [ProducerStateManager partition=mcmp-logs-0] Wrote producer snapshot at offset 4174061623 with 0 producer ids in 1 ms. (org.apache.kafka.storage.internals.log.ProducerStateManager) [data-plane-kafka-request-handler-4]
2025-05-07 10:06:59,011 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=187247, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-2-raft-scheduler0]
2025-05-07 10:39:07,923 INFO [LocalLog partition=mcmp-logs-0, dir=/var/lib/kafka/data-0/kafka-log2] Rolled new log segment at offset 4174061623 in 280 ms. (kafka.log.LocalLog) [ReplicaFetcherThread-0-0]
2025-05-07 10:39:07,924 INFO [ProducerStateManager partition=mcmp-logs-0] Wrote producer snapshot at offset 4174061623 with 0 producer ids in 1 ms. (org.apache.kafka.storage.internals.log.ProducerStateManager) [ReplicaFetcherThread-0-0]
2025-05-07 11:05:29,002 INFO [SnapshotGenerator id=0] Creating new KRaft snapshot file snapshot 00000000000001403884-0000000001 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-0-snapshot-generator-event-handler]
2025-05-07 11:05:29,002 INFO [SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001403884-0000000001 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-1-snapshot-generator-event-handler]
2025-05-07 11:05:29,002 INFO [SnapshotGenerator id=2] Creating new KRaft snapshot file snapshot 00000000000001403884-0000000001 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-2-snapshot-generator-event-handler]
2025-05-07 11:05:29,021 INFO [SnapshotEmitter id=2] Successfully wrote snapshot 00000000000001403884-0000000001 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-2-snapshot-generator-event-handler]
2025-05-07 11:05:59,014 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log2] Incremented log start offset to 201645 due to snapshot generated (kafka.log.UnifiedLog) [kafka-2-raft-io-thread]
2025-05-07 11:05:59,014 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=2] Marking snapshot OffsetAndEpoch(offset=194446, epoch=1) for deletion because its timestamp (1746011101722) is now (1746615959014) older than the
2025-05-07 11:05:29,021 INFO [SnapshotEmitter id=1] Successfully wrote snapshot 00000000000001403884-0000000001 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-1-snapshot-generator-event-handler]
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-2-raft-io-thread]
2025-05-07 11:05:29,022 INFO [SnapshotEmitter id=0] Successfully wrote snapshot 00000000000001403884-0000000001 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-0-snapshot-generator-event-handler]
2025-05-07 11:05:59,933 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log1] Incremented log start offset to 201645 due to snapshot generated (kafka.log.UnifiedLog) [kafka-1-raft-io-thread]
2025-05-07 11:05:59,934 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=1] Marking snapshot OffsetAndEpoch(offset=194446, epoch=1) for deletion because its timestamp (1746011101722) is now (1746615959933) older than the
2025-05-07 11:05:56,014 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Incremented log start offset to 201645 due to snapshot generated (kafka.log.UnifiedLog) [kafka-0-raft-io-thread]
2025-05-07 11:05:56,015 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=0] Marking snapshot OffsetAndEpoch(offset=194446, epoch=1) for deletion because its timestamp (1746011101722) is now (1746615956014) older than the
2025-05-07 11:06:59,021 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=194446, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-2-raft-scheduler0]
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-1-raft-io-thread]
2025-05-07 11:06:59,940 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=194446, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-1-raft-scheduler0]
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-0-raft-io-thread]
2025-05-07 11:06:56,020 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=194446, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-0-raft-scheduler0]

controller logs

k logs -f -l strimzi.io/broker-role=false | tspin
2025-05-07 10:05:55,742 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log3] Incremented log start offset to 194446 due to snapshot generated (kafka.log.UnifiedLog) [kafka-3-raft-io-thread]
2025-05-07 10:05:55,742 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=3] Marking snapshot OffsetAndEpoch(offset=187247, epoch=1) for deletion because its timestamp (1746007501560) is now (1746612355742) older than the
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-3-raft-io-thread]
2025-05-07 10:06:55,748 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=187247, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-3-raft-scheduler0]
2025-05-07 11:05:29,003 INFO [SnapshotGenerator id=3] Creating new KRaft snapshot file snapshot 00000000000001403884-0000000001 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-3-snapshot-generator-event-handler]
2025-05-07 11:05:29,172 INFO [SnapshotEmitter id=3] Successfully wrote snapshot 00000000000001403884-0000000001 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-3-snapshot-generator-event-handler]
2025-05-07 11:05:55,748 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log3] Incremented log start offset to 201645 due to snapshot generated (kafka.log.UnifiedLog) [kafka-3-raft-io-thread]
2025-05-07 11:05:55,749 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=3] Marking snapshot OffsetAndEpoch(offset=194446, epoch=1) for deletion because its timestamp (1746011101722) is now (1746615955748) older than the
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-3-raft-io-thread]
2025-05-07 10:05:55,492 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log5] Incremented log start offset to 194446 due to snapshot generated (kafka.log.UnifiedLog) [kafka-5-raft-io-thread]
2025-05-07 10:05:55,492 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=5] Marking snapshot OffsetAndEpoch(offset=187247, epoch=1) for deletion because its timestamp (1746007501560) is now (1746612355492) older than the
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-5-raft-io-thread]
2025-05-07 10:06:55,498 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=187247, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-5-raft-scheduler0]
2025-05-07 11:05:29,003 INFO [SnapshotGenerator id=5] Creating new KRaft snapshot file snapshot 00000000000001403884-0000000001 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-5-snapshot-generator-event-handler]
2025-05-07 11:05:29,172 INFO [SnapshotEmitter id=5] Successfully wrote snapshot 00000000000001403884-0000000001 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-5-snapshot-generator-event-handler]
2025-05-07 11:05:55,497 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log5] Incremented log start offset to 201645 due to snapshot generated (kafka.log.UnifiedLog) [kafka-5-raft-io-thread]
2025-05-07 11:05:55,497 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=5] Marking snapshot OffsetAndEpoch(offset=194446, epoch=1) for deletion because its timestamp (1746011101722) is now (1746615955497) older than the
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-5-raft-io-thread]
2025-05-07 11:06:55,755 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=194446, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-3-raft-scheduler0]
2025-05-07 11:06:55,513 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=194446, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-5-raft-scheduler0]
2025-05-07 10:05:55,229 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log4] Incremented log start offset to 194447 due to snapshot generated (kafka.log.UnifiedLog) [kafka-4-raft-io-thread]
2025-05-07 10:05:55,229 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=4] Marking snapshot OffsetAndEpoch(offset=187248, epoch=1) for deletion because its timestamp (1746007502060) is now (1746612355228) older than the
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-4-raft-io-thread]
2025-05-07 10:06:55,235 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=187248, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-4-raft-scheduler0]
2025-05-07 11:05:29,004 INFO [SnapshotGenerator id=4] Creating new KRaft snapshot file snapshot 00000000000001403885-0000000001 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-4-snapshot-generator-event-handler]
2025-05-07 11:05:29,018 INFO [SnapshotEmitter id=4] Successfully wrote snapshot 00000000000001403885-0000000001 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-4-snapshot-generator-event-handler]
2025-05-07 11:05:55,236 INFO [UnifiedLog partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log4] Incremented log start offset to 201646 due to snapshot generated (kafka.log.UnifiedLog) [kafka-4-raft-io-thread]
2025-05-07 11:05:55,237 INFO [MetadataLog partition=__cluster_metadata-0, nodeId=4] Marking snapshot OffsetAndEpoch(offset=194447, epoch=1) for deletion because its timestamp (1746011102223) is now (1746615955236) older than the
retention (604800000) (kafka.raft.KafkaMetadataLog) [kafka-4-raft-io-thread]
2025-05-07 11:06:55,243 INFO Deleted snapshot files for snapshot OffsetAndEpoch(offset=194447, epoch=1). (org.apache.kafka.snapshot.Snapshots) [kafka-4-raft-scheduler0]

strimzi operator logs

2025-05-07 11:00:31 INFO  AbstractOperator:546 - Reconciliation #17332(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:02:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:02:30 INFO  AbstractOperator:266 - Reconciliation #17333(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:02:31 INFO  AbstractOperator:546 - Reconciliation #17333(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:349 - Reconciliation #17334(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:385 - Reconciliation #17334(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:349 - Reconciliation #17335(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:385 - Reconciliation #17335(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:349 - Reconciliation #17336(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:385 - Reconciliation #17336(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:349 - Reconciliation #17337(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:385 - Reconciliation #17337(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:349 - Reconciliation #17338(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:03:29 INFO  StrimziPodSetController:385 - Reconciliation #17338(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:04:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:04:30 INFO  AbstractOperator:266 - Reconciliation #17339(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:04:31 INFO  AbstractOperator:546 - Reconciliation #17339(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:06:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:06:30 INFO  AbstractOperator:266 - Reconciliation #17340(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:06:31 INFO  AbstractOperator:546 - Reconciliation #17340(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:349 - Reconciliation #17341(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:385 - Reconciliation #17341(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:349 - Reconciliation #17342(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:385 - Reconciliation #17342(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:349 - Reconciliation #17343(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:385 - Reconciliation #17343(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:349 - Reconciliation #17344(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:385 - Reconciliation #17344(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:349 - Reconciliation #17345(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:08:29 INFO  StrimziPodSetController:385 - Reconciliation #17345(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:08:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:08:30 INFO  AbstractOperator:266 - Reconciliation #17346(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:08:31 INFO  AbstractOperator:546 - Reconciliation #17346(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:10:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:10:30 INFO  AbstractOperator:266 - Reconciliation #17347(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:10:31 INFO  AbstractOperator:546 - Reconciliation #17347(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:12:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:12:30 INFO  AbstractOperator:266 - Reconciliation #17348(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:12:31 INFO  AbstractOperator:546 - Reconciliation #17348(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:13:29 INFO  StrimziPodSetController:349 - Reconciliation #17349(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:13:29 INFO  StrimziPodSetController:385 - Reconciliation #17349(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:13:29 INFO  StrimziPodSetController:349 - Reconciliation #17350(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:13:29 INFO  StrimziPodSetController:385 - Reconciliation #17350(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:14:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:14:30 INFO  AbstractOperator:266 - Reconciliation #17351(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:14:31 INFO  AbstractOperator:546 - Reconciliation #17351(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:16:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:16:30 INFO  AbstractOperator:266 - Reconciliation #17352(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:16:31 INFO  AbstractOperator:546 - Reconciliation #17352(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:18:29 INFO  StrimziPodSetController:349 - Reconciliation #17353(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:18:29 INFO  StrimziPodSetController:385 - Reconciliation #17353(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:18:29 INFO  StrimziPodSetController:349 - Reconciliation #17354(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:18:29 INFO  StrimziPodSetController:385 - Reconciliation #17354(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:18:29 INFO  StrimziPodSetController:349 - Reconciliation #17355(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:18:29 INFO  StrimziPodSetController:385 - Reconciliation #17355(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:18:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:18:30 INFO  AbstractOperator:266 - Reconciliation #17356(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:18:31 INFO  AbstractOperator:546 - Reconciliation #17356(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:20:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:20:30 INFO  AbstractOperator:266 - Reconciliation #17357(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:20:31 INFO  AbstractOperator:546 - Reconciliation #17357(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:22:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:22:30 INFO  AbstractOperator:266 - Reconciliation #17358(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:22:31 INFO  AbstractOperator:546 - Reconciliation #17358(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:349 - Reconciliation #17359(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:385 - Reconciliation #17359(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:349 - Reconciliation #17360(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:385 - Reconciliation #17360(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:349 - Reconciliation #17361(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:385 - Reconciliation #17361(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:349 - Reconciliation #17362(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:385 - Reconciliation #17362(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:349 - Reconciliation #17363(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:23:29 INFO  StrimziPodSetController:385 - Reconciliation #17363(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:24:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:24:30 INFO  AbstractOperator:266 - Reconciliation #17364(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:24:31 INFO  AbstractOperator:546 - Reconciliation #17364(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:26:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:26:30 INFO  AbstractOperator:266 - Reconciliation #17365(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:26:31 INFO  AbstractOperator:546 - Reconciliation #17365(timer) Kafka(kafka/log-aggregator-kafka): reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:349 - Reconciliation #17366(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:385 - Reconciliation #17366(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:349 - Reconciliation #17367(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:385 - Reconciliation #17367(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:349 - Reconciliation #17368(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:385 - Reconciliation #17368(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:349 - Reconciliation #17369(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): StrimziPodSet will be reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:385 - Reconciliation #17369(watch) StrimziPodSet(kafka/log-aggregator-kafka-broker): reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:349 - Reconciliation #17370(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): StrimziPodSet will be reconciled
2025-05-07 11:28:29 INFO  StrimziPodSetController:385 - Reconciliation #17370(watch) StrimziPodSet(kafka/log-aggregator-kafka-controller): reconciled
2025-05-07 11:28:30 INFO  ClusterOperator:140 - Triggering periodic reconciliation for namespace kafka
2025-05-07 11:28:30 INFO  AbstractOperator:266 - Reconciliation #17371(timer) Kafka(kafka/log-aggregator-kafka): Kafka log-aggregator-kafka will be checked for creation or modification
2025-05-07 11:28:31 INFO  AbstractOperator:546 - Reconciliation #17371(timer) Kafka(kafka/log-aggregator-kafka): reconciled

same logs are repeated

[see-quick]

From the logs, I don't see any suspicious. Maybe you can set logging to DEBUG so we know more? Also, I think this would be a more Argocd-specific problem than Strimzi. Can you post the Application yaml to be able to better understand what's going on? There should be some error or warning why that out-of-sync is happening.

[bonzonkim]

Hi @see-quick , here is my argocd application resource manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kafka-cluster
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: default
  destination:
    namespace: kafka
    server: https://kubernetes.default.svc
  source:
    path: path/to/kafka/config
    repoURL: git@github.com:mygithub/githubRepo.git
    targetRevision: main

I thougth #7205 would work for me but it wasn't

[bonzonkim]

Now I can confirm it's an ArgoCD issue, not Strimzi.
I followed #7205, and initially the PVCs appear healthy, but after a short while, they disappear even though the application is marked as synced.
This suggests that ArgoCD considers these PVCs as resources that should not be deployed.

[Frawless]

hi @bonzonkim , is it really Out of Sync or just _There are unexpected resources not managed by Argo_ ? If the second than it is your misconfiguration of Argo App I would say.

I am using this configuration https://github.com/skodjob/deployment-hub/blob/main/strimzi/mirror/kafka/00-kafka-route.yaml (which looks similar to me as yours) that is synced by Argo CD and I don't see PVCs in Argo's UI. In general, PVCs with deleteClaim=false doesn't have owner reference so I think Argo in such cases doesn't show the resource as part of the Argo App. Maybe could you share Argo App yaml and also one of the PVCs please? Maybe there is some annotation added by argo or something that could cause this? Or did you have deleteClaim ever set to true ?

[bonzonkim]

Hi @Frawless , Thanks for your reply.
I am using this configurations https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kafka-persistent.yaml with this config, PVCs are deployed. and PVCs status appears OutOfSync (This resource is not present in the application's source. It will be deleted from Kubernetes if the prune option is enabled during sync.)

ArgoCD Application yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kafka-cluster
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: default
  destination:
    namespace: kafka
    server: https://kubernetes.default.svc
  source:
    path: path/to/kafka/config
    repoURL: git@github.com:mygithub/githubRepo.git
    targetRevision: main

[bonzonkim]

and this is one of my PVCs live manifest that shown as 'Out of Sync'

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: 'kafka-cluster:argoproj.io/Application:argocd/kafka-cluster'
    pv.kubernetes.io/bind-completed: 'yes'
    strimzi.io/delete-claim: 'false'
  creationTimestamp: '2025-04-29T08:04:14Z'
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/instance: log-aggregator-kafka
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-log-aggregator-kafka
    argocd.argoproj.io/instance: kafka-cluster
    strimzi.io/cluster: log-aggregator-kafka
    strimzi.io/component-type: kafka
    strimzi.io/kind: Kafka
    strimzi.io/name: log-aggregator-kafka-kafka
    strimzi.io/pool-name: broker
  name: data-0-log-aggregator-kafka-broker-2
  namespace: kafka
  resourceVersion: '24281769'
  uid: de7e91b4-0a83-4203-b72a-68c88f111b5c
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: nfs
  volumeMode: Filesystem
  volumeName: pvc-de7e91b4-0a83-4203-b72a-68c88f111b5c
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 100Gi
  phase: Bound

I thought this was app.kubernetes.io/managed-by problem. but other resources that keeping status synced and health has the same app.kubernetes.io/managed-by and argocd.argoproj.io/instance value.

[Frawless]

I do not have any argocd* annotation/label on mine PVCs, I wonder why. My Argo App additionaly have this

syncPolicy:
    automated:
      selfHeal: true

but I am not sure if this could be one of the workarounds Jakub mentioned below. But in general he is right, I don't think it is something we can do with on our side.

[scholzj]

I've searched for similar issues, but there doesn't seem to be a clear solution yet. Has anyone else run into this? If you have, I'd appreciate if you share with me.

I do not think there will be any clear solution ever. Tracking resources through owner references has its limitations because it interferes with garbage collection. So this problem will be always there and you have to use one of the workarounds to deal with it.

[bonzonkim]

You're right. I'll try workarounds if one of those works for me. Thank you