pg operator creates empty disk

[uk1988]

Hej all

First of all, I have to say we are a happy user of the pg operator until recent upgrade it worked perfectly.

Ever since we have upgraded postgres-operator from 1.7.1 to 1.9.0 we are experiencing a funny issue.
Let say for every 10-deployment operator fails to create a database.
By that I mean that pvc where database should be created is empty.

We are using https://github.com/rancher/local-path-provisioner as the default storage class for our pg clusters.

Example of our reqeust to operator

apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: keycloak-postgresql
spec:
  teamId: keycloak
  resources:
    requests:
      cpu: 50m
      memory: 200Mi
    limits:
      cpu: 250m
      memory: 500Mi
  volume:
    size: {{ .Values.persistence.size }}
    storageClass: {{ .Values.persistence.storageClass }}
  numberOfInstances: 1
  users:
    keycloak: # database owner
      - login
  enableMasterLoadBalancer: false
  enableReplicaLoadBalancer: false
  allowedSourceRanges: # load balancers' source ranges for both master and replica services
    - 0.0.0.0/0
  databases:
    keycloak: keycloak # dbname: owner
  postgresql:
    version: "14"
    parameters:
      log_destination: stderr
      logging_collector: "off"
  spiloRunAsUser: 101
  spiloRunAsGroup: 103
  spiloFSGroup: 103

Logs from the pg pod and operator:
keycloak-postgresql-0.log
postgres-operator-667568dcc9-vwxt8.log

When we did upgrade of the operator, we have also finally updated our k8s clusters from 1.23.x to 1.26.x
We are experiencing similar issue with postgres-operator 1.10.1.

Any idea what is the issue?

[uk1988]

I think I have figured out the issue, kube scheduler is just too busy:

2024-04-12 13:23:10,407 INFO: Lock owner: None; I am keycloak-postgresql-0
2024-04-12 13:23:16,437 ERROR: Request to server https://10.43.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.43.0.1', port=443): Read timed out. (read timeout=4.999674700000014)")
2024-04-12 13:23:19,389 ERROR: Request to server https://10.43.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.43.0.1', port=443): Read timed out. (read timeout=2.042445346348586)")
2024-04-12 13:23:21,437 ERROR: Request to server https://10.43.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.43.0.1', port=443): Read timed out. (read timeout=1.4978268808044959)")
2024-04-12 13:23:25,571 INFO: failed to acquire initialize lock

why does operator try only 3 times?
I guess after 3 times is goes to standby mode and does not retry at all

where can we increase that retry?

[uk1988]

@FxKu could please help me out here?
does pod try only 3 times to get a Lock owner?