Calico 3.29.2 apiserver fails to start due to missing validatingadmission* rbac rules

[philroche]

Code @ https://github.com/tigera/operator/blob/ed26c42e1e232e9cc604a9a010d4ca9111b6f388/pkg/render/apiserver.go#L654C11-L654C28 conditionally adds rule based on kuberetes version

if c.cfg.KubernetesVersion == nil || !(c.cfg.KubernetesVersion != nil && c.cfg.KubernetesVersion.Major < 2 && c.cfg.KubernetesVersion.Minor < 30) {
		// If the kubernetes version is higher than 1.30, we add extra RBAC permissions to allow establishing watches.
		// https://v1-30.docs.kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
		rules = append(rules, rbacv1.PolicyRule{
			// Kubernetes validating admission policy resources.
			APIGroups: []string{"admissionregistration.k8s.io"},
			Resources: []string{
				"validatingadmissionpolicies",
				"validatingadmissionpolicybindings",
			},
			Verbs: []string{
				"get",
				"list",
				"watch",
			},
		})
	}

With calico 3.29.2 with commit projectcalico/calico@8eacb02 there was a new default set to EnableValidatingAdmissionPolicy which depends on the validatingadmissionpolicies and validatingadmissionpolicybindings being present. When deploying tigera operator via a helm chart install the rules required to successfully make those requests are missing resulting in calico-apiserver error:

0219 12:38:01.076291       1 reflector.go:150] k8s.io/client-go@v0.30.9/tools/cache/reflector.go:232: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: validatingadmissionpolicies.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:calico-apiserver:calico-apiserver" cannot list resource "validatingadmissionpolicies" in API group "admissionregistration.k8s.io" at the cluster scope

The logic of the if statement above is reversed for the c.cfg.KubernetesVersion.Minor check and should be checking c.cfg.KubernetesVersion.Minor > 30 and not < 30 as the comment suggests it is. I tested with kuberenetes 1.32 and observed these errors. If I manually add these rules the the calico-apiserver starts as expected.

[philroche]

PR #3781 fixes this issue

[philroche]

Discussion in the PR #3781 shows my assumptions on the logic of this check were incorrect but the issue does still persist.

[philroche]

Some further debug with a deployment via helm chart of tigera-operator 1.37

# kubectl get pods --all-namespaces
NAMESPACE          NAME                                      READY   STATUS                 RESTARTS       AGE
calico-apiserver   calico-apiserver-6f5cf9d797-g8pqc         1/1     Running                0              32m
calico-apiserver   calico-apiserver-6f5cf9d797-phw4g         1/1     Running                0              32m
calico-system      calico-kube-controllers-fd489b6b5-l5sdc   1/1     Running                0              33m
calico-system      calico-node-7rqj2                         1/1     Running                0              33m
calico-system      calico-typha-54cb7c654d-dlwrt             1/1     Running                0              33m
calico-system      csi-node-driver-dhxtw                     0/2     CreateContainerError   10 (83s ago)   33m
kube-system        coredns-ff8999cc5-584tl                   1/1     Running                0              34m
kube-system        local-path-provisioner-698b58967b-jrpvq   1/1     Running                0              34m
tigera-operator    tigera-operator-7dc4f84c95-rb62d          1/1     Running                0              33m

# kubectl auth can-i list validatingadmissionpolicies --as=system:serviceaccount:calico-apiserver:calico-apiserver
Warning: resource 'validatingadmissionpolicies' is not namespace scoped in group 'admissionregistration.k8s.io'

no
bash-5.2# kubectl get clusterrole calico-crds -o yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2025-02-21T11:10:30Z"
  name: calico-crds
  ownerReferences:
  - apiVersion: operator.tigera.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: APIServer
    name: default
    uid: 89344af0-4d8c-4f28-8a98-b6c59c5be132
  resourceVersion: "778"
  uid: 98e67a25-ccba-4cdf-9506-54e1f8b27bc9
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - pods
  - serviceaccounts
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - crd.projectcalico.org
  resources:
  - globalnetworkpolicies
  - networkpolicies
  - caliconodestatuses
  - clusterinformations
  - hostendpoints
  - globalnetworksets
  - networksets
  - bgpconfigurations
  - bgpfilters
  - bgppeers
  - felixconfigurations
  - kubecontrollersconfigurations
  - ippools
  - ipreservations
  - ipamblocks
  - blockaffinities
  - ipamconfigs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - delete
  - patch
# kubectl version --output=yaml
clientVersion:
  buildDate: "2025-02-13T21:25:31Z"
  compiler: gc
  gitCommit: 67a30c0adcf52bd3f56ff0893ce19966be12991f
  gitTreeState: clean
  gitVersion: v1.32.2
  goVersion: go1.24.0
  major: "1"
  minor: "32"
  platform: linux/amd64
kustomizeVersion: v5.5.0
serverVersion:
  buildDate: "2025-02-09T10:06:28Z"
  compiler: gc
  gitCommit: 6a322f122729e0e668ca67fd9f0e993541bdce49
  gitTreeState: dirty
  gitVersion: v1.32.1+k3s1
  goVersion: go1.23.6
  major: "1"
  minor: "32"
  platform: linux/amd64