Failed to unmarshal "config.yaml": <nil>

[obervinov]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
Through the helm chart, argo-cd should deploy the following manifests to kubernetes

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: nodepool-1
spec:
  limits:
    memory: 192Gi
  template:
    metadata:
      labels:
        node-group: nodepool-1
    spec:
      expireAfter: 720h
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: generic
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values:
            - r6a
        - key: karpenter.k8s.aws/instance-memory
          operator: In
          values:
            - '4096'
      startupTaints:
        - effect: NoExecute
          key: node.cilium.io/agent-not-ready
          value: 'true'
      taints:
        - effect: NoSchedule
          key: node-group
          value: nodepool-1

We also use this helm template in our helm chart

{{- if .Values.karpenterNodePools }}
{{-   range .Values.karpenterNodePools }}
{{-     $namespace := $.Release.Namespace }}
{{-     $apiVersion := .apiVersion }}
{{-     if not $apiVersion }}
{{-         $apiVersion = "karpenter.sh/v1beta1" }}
{{-     end }}
{{-     $name := .name }}
{{-     $spec := .spec }}
---
apiVersion: {{ $apiVersion }}
kind: NodePool
metadata:
  name: {{ $namespace }}-{{ $name }}
spec:
{{-      if $spec }}
  {{- toYaml .spec | nindent 2 }}
{{-      end }}
...
{{-   end }}
{{- end }}

and this config.yaml

karpenterNodePools:
  - name: nodepool-1
    apiVersion: karpenter.sh/v1
    spec:
      template:
        metadata:
          labels:
            node-group: nodepool-1
        spec:
          nodeClassRef:
            name: generic
            kind: EC2NodeClass
            group: karpenter.k8s.aws
          startupTaints:
            - key: node.cilium.io/agent-not-ready
              value: "true"
              effect: NoExecute
          taints:
            - key: node-group
              effect: NoSchedule
              value: nodepool-1
          requirements:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["on-demand"]
            - key: karpenter.k8s.aws/instance-family
              operator: In
              values: ["r6a"]
            - key: karpenter.k8s.aws/instance-memory
              operator: In
              values: ["4096"]
      limits:
        memory: 192Gi

As soon as we try to start synchronization, the application freezes with an error

ComparisonError: Failed to load target state: failed to generate manifest for source 2 of 2: rpc error: code = FailedPrecondition desc = Failed to unmarshal "config.yaml": <nil>

Experimentally, we found out that reading config.yml starts to break as soon as we add these two parameters.

            kind: EC2NodeClass
            group: karpenter.k8s.aws

If we remove them, the application starts to work correctly.
We also tried to reproduce the problem directly by running the helm template with this config.yaml and our diagram for argo-cd, but we were unable to reproduce this problem.

To Reproduce

1. Make helm chart with the following template

{{- if .Values.karpenterNodePools }}
{{-   range .Values.karpenterNodePools }}
{{-     $namespace := $.Release.Namespace }}
{{-     $apiVersion := .apiVersion }}
{{-     if not $apiVersion }}
{{-         $apiVersion = "karpenter.sh/v1beta1" }}
{{-     end }}
{{-     $name := .name }}
{{-     $spec := .spec }}
---
apiVersion: {{ $apiVersion }}
kind: NodePool
metadata:
  name: {{ $namespace }}-{{ $name }}
spec:
{{-      if $spec }}
  {{- toYaml .spec | nindent 2 }}
{{-      end }}
...
{{-   end }}
{{- end }}

2. Make config.yaml with the following content

karpenterNodePools:
  - name: nodepool-1
    apiVersion: karpenter.sh/v1
    spec:
      template:
        metadata:
          labels:
            node-group: nodepool-1
        spec:
          nodeClassRef:
            name: generic
            kind: EC2NodeClass
            group: karpenter.k8s.aws
          startupTaints:
            - key: node.cilium.io/agent-not-ready
              value: "true"
              effect: NoExecute
          taints:
            - key: node-group
              effect: NoSchedule
              value: nodepool-1
          requirements:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["on-demand"]
            - key: karpenter.k8s.aws/instance-family
              operator: In
              values: ["r6a"]
            - key: karpenter.k8s.aws/instance-memory
              operator: In
              values: ["4096"]
      limits:
        memory: 192Gi

3. Add the application to argo-cd
4. Start synchronization

Expected behavior
Argo-CD should correctly process the following config.yaml file

karpenterNodePools:
  - name: nodepool-1
    apiVersion: karpenter.sh/v1
    spec:
      template:
        metadata:
          labels:
            node-group: nodepool-1
        spec:
          nodeClassRef:
            name: generic
            kind: EC2NodeClass
            group: karpenter.k8s.aws
          startupTaints:
            - key: node.cilium.io/agent-not-ready
              value: "true"
              effect: NoExecute
          taints:
            - key: node-group
              effect: NoSchedule
              value: nodepool-1
          requirements:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["on-demand"]
            - key: karpenter.k8s.aws/instance-family
              operator: In
              values: ["r6a"]
            - key: karpenter.k8s.aws/instance-memory
              operator: In
              values: ["4096"]
      limits:
        memory: 192Gi

Screenshots

Version

argocd: v2.14.1+3345d05
  BuildDate: 2025-02-03T21:23:31Z
  GitCommit: 3345d05a43d8edd7ec42ec71ffe8b5f95bc68dbe
  GitTreeState: clean
  GoVersion: go1.23.3
  Compiler: gc
  Platform: linux/amd64

Logs

time="2025-02-21T13:44:42Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): rpc error: code = FailedPrecondition desc = Failed to unmarshal \"config.yaml\": <nil>" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2025-02-21T13:44:42Z" grpc.time_ms=13.638 span.kind=server system=grpc

[crenshaw-dev]

Can you post your Application manifest?

[obervinov]

@crenshaw-dev sure, here's

project: dalaran
destination:
  namespace: test-hephaestus
  name: dalaran
syncPolicy:
  automated:
    selfHeal: true
  syncOptions:
    - CreateNamespace=true
    - PruneLast=true
sources:
  - repoURL: https://nexus.exmaple.com/repository/helm
    targetRevision: 1.*.*
    helm:
      valueFiles:
        - $config/dalaran/test-hephaestus/config.yaml
    chart: argo
  - repoURL: https://gitlab.exmaple.com/ops/k8s-cluster-state
    path: dalaran/test-hephaestus
    targetRevision: master
    ref: config

[rumstead]

This feels semi-similar to #11001.

IIRC, the code tries to be smart and looks to see if the yaml/json contains "apiVersion,kind,metadata" and if it does, it treats the yaml as a Kubernetes resource.