[Bug]: The operator restarts 1-2 times due to an internal locking / timeout mechanism related to informers sync:

[ekowsal]

Bug Description

The strimzi operator pod restarts 1-2 times due to an internal locking / timeout mechanism related to informers sync. One suspect we have is related to the recent strimzi uplift to 0.39. we don't have hard evidence that the uplift is the cause of the issue, or if it's just a red herring. That being said, the ShutdownHook calls (introduced in 6bd9728#diff-afffcd80290af3211aded91d5d8ff9187c88279d92295dd9aafe242bcd8d45d0 ) are now more apparent.

Operator logs:

2025-04-09T06:16:01.924+00:00 Health and metrics server is ready on port 8080)
2025-04-09T06:16:02.024+00:00 Waiting to become a leader
2025-04-09T06:16:02.024+00:00 Starting the Leader Elector
2025-04-09T06:16:06.423+00:00 The new leader is eric-data-message-bus-kf-operator-5776cb85dc-m72cm
2025-04-09T06:16:07.013+00:00 The new leader is eric-data-message-bus-kf-operator-5776cb85dc-cf28r
2025-04-09T06:16:07.014+00:00 Started being a leader
2025-04-09T06:16:07.014+00:00 I'm the new leader
2025-04-09T06:16:08.212+00:00 API Group route.openshift.io is not supported
2025-04-09T06:16:08.314+00:00 API Group build.openshift.io is not supported
2025-04-09T06:16:08.328+00:00 API Group image.openshift.io is not supported
2025-04-09T06:16:08.329+00:00 Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.28,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2025-04-09T06:16:08.608+00:00 Found PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2025-04-09T06:16:08.609+00:00 Initializing PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2025-04-09T06:16:08.916+00:00 Creating ClusterOperator for namespace oamfw-ci-testing-tls
2025-04-09T06:16:09.112+00:00 Starting ClusterOperator for namespace oamfw-ci-testing-tls
2025-04-09T06:16:09.427+00:00 Starting the StrimziPodSet controller
2025-04-09T06:16:09.428+00:00 Starting StrimziPodSet controller for namespace oamfw-ci-testing-tls
2025-04-09T06:16:09.711+00:00 The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2025-04-09T06:16:09.810+00:00 The client is using resource type 'kafkas' with unstable version 'v1beta2'
2025-04-09T06:16:09.817+00:00 The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2025-04-09T06:16:09.819+00:00 The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2025-04-09T06:16:09.913+00:00 Waiting for informers to sync
2025-04-09T06:16:21.115+00:00 Shutdown hook started

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.39.0

Kubernetes version

Kubernetes 1.22

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Not sure what locking are you talking about. From the log I would say the most likely issue is resource starvation.

[ekowsal]

@scholzj we are talking about LeaderElectionManager lock. How to address this issue. Could you please elaborate more on resource starvation and how to improve it.

Improve LeaderElectionManager to release the lock at shutdown by scholzj · Pull Request #8940 · strimzi/strimzi-kafka-operator · GitHub

[scholzj]

I do not think anything in the log snippet you shared suggests that there is a problem with the leader election. Nor I see any relation to the PR you linked which is about something completely different.