[Fleet] Simplified package policy schema validation fails both in preconfiguration and api

[criamico]

Kibana version:
Versions >= 8.15.1

Describe the bug:
When trying to generate a package policy using the simplified schema, the validation fails on the streams portion of the policy. This happens both with preconfiguration and with regular API create.

Steps to reproduce:

Preconfiguration

• Load this sample policy in kibana.dev.yml - integration used is elasticsearch-1.15.2

Policy

xpack.fleet.agentPolicies:
- name: dev-elastic-monitoring
  id: dev-elastic-monitoring
  namespace: dev
  monitoring_enabled: ["logs", "metrics"]
  inactivity_timeout: 1209600
  is_protected: false
  package_policies:
  - id: elasticsearch-dev-elastic-monitoring
    package:
      name: elasticsearch
    name: elasticsearch-dev-elastic-monitoring
    description: ''
    namespace: ''
    inputs:
      elasticsearch-logfile:
        enabled: true
        vars: {}
        streams:
          elasticsearch.audit:
            enabled: true
            vars:
              paths:
              - "/var/log/elasticsearch/*_audit.json"
              tags: []
          elasticsearch.deprecation:
            enabled: true
            vars:
              paths:
              - "/var/log/elasticsearch/*_deprecation.json"
              tags: []
          elasticsearch.gc:
            enabled: true
            vars:
              paths:
              - "/var/log/elasticsearch/gc.log.[0-9]*"
              - "/var/log/elasticsearch/gc.log"
              tags: []
          elasticsearch.server:
            enabled: true
            vars:
              paths:
              - "/var/log/elasticsearch/*_server.json"
              tags: []
          elasticsearch.slowlog:
            enabled: true
            vars:
              paths:
              - "/var/log/elasticsearch/*_index_search_slowlog.json"
              - "/var/log/elasticsearch/*_index_indexing_slowlog.json"
              tags: []
      elasticsearch-elasticsearch/metrics:
        enabled: true
        vars:
          scope: cluster
          leaderelection: false
          hosts:
          - https://test:9200

        streams:
          elasticsearch.stack_monitoring.ccr:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.cluster_stats:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.enrich:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.index:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.index_recovery:
            enabled: true
            vars:
              active.only: true
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.index_summary:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.ingest_pipeline:
            enabled: true
            vars:
              ingest_pipeline_processor_sampling_rate: '0.25'
              tags: []
          elasticsearch.stack_monitoring.ml_job:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.node:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.node_stats:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.pending_tasks:
            enabled: true
            vars:
              period: 10s
              tags: []
          elasticsearch.stack_monitoring.shard:
            enabled: true
            vars:
              period: 10s
              tags: []

• The preconfiguration fails with error

[2025-03-04T11:48:39.453+01:00][FATAL][root] Reason: [config validation of [xpack.fleet].agentPolicies.0.package_policies.0.inputs.elasticsearch-logfile.streams]: definition for this key is missing Error: [config validation of [xpack.fleet].agentPolicies.0.package_policies.0.inputs.elasticsearch-logfile.streams]: definition for this key is missing

API

Tested the same behavior when trying to create the policy via api, I loaded the POST command from the page "add integration":

POST

POST kbn:/api/fleet/package_policies
{
  "policy_ids": [
    ""
  ],
  "package": {
    "name": "elasticsearch",
    "version": "1.15.2"
  },
  "name": "elasticsearch-1",
  "description": "",
  "namespace": "",
  "inputs": {
    "elasticsearch-logfile": {
      "enabled": true,
      "vars": {},
      "streams": {
        "elasticsearch.audit": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/elasticsearch/*_audit.json"
            ],
            "tags": []
          }
        },
        "elasticsearch.deprecation": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/elasticsearch/*_deprecation.json"
            ],
            "tags": []
          }
        },
        "elasticsearch.gc": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/elasticsearch/gc.log.[0-9]*",
              "/var/log/elasticsearch/gc.log"
            ],
            "tags": []
          }
        },
        "elasticsearch.server": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/elasticsearch/*_server.json"
            ],
            "tags": []
          }
        },
        "elasticsearch.slowlog": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/elasticsearch/*_index_search_slowlog.json",
              "/var/log/elasticsearch/*_index_indexing_slowlog.json"
            ],
            "tags": []
          }
        }
      }
    },
    "elasticsearch-elasticsearch/metrics": {
      "enabled": true,
      "vars": {
        "hosts": [
          "http://localhost:9200"
        ],
        "scope": "node",
        "timeout": "10s",
        "ssl": "#certificate_authorities: [\"/etc/ca.crt\"]\n#certificate: \"/etc/client.crt\"\n#key: \"/etc/client.key\"\n",
        "leaderelection": false
      },
      "streams": {
        "elasticsearch.stack_monitoring.ccr": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.cluster_stats": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.enrich": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.index": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.index_recovery": {
          "enabled": true,
          "vars": {
            "active.only": true,
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.index_summary": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.ingest_pipeline": {
          "enabled": true,
          "vars": {
            "ingest_pipeline_processor_sampling_rate": "0.25",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.ml_job": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.node": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.node_stats": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.pending_tasks": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        },
        "elasticsearch.stack_monitoring.shard": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "tags": []
          }
        }
      }
    }
  }
}

In this case I saw the error as well:

Expected behavior:
In both cases the policy should be created correctly.

Any additional context:
The error is probably in this schema. I noticed that commenting the streams portion of the policy it passes validation, so I think that the issue lies there.

Let's make sure to add some unit tests to avoid breaking it again.
This fix would also require backporting as it is present on several versions of kibana.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[criamico]

@nchaulet suggested that the issue could depend on the fact that kibana tries to auto-expand properties that have a dot in the name, so they should be instead escaped like this:
'[elasticsearch-logfile.streams]'

I tested a portion of the preconfigured policy above with the escaping but it fails with the same error:

xpack.fleet.agentPolicies:
- name: dev-elastic-monitoring
  id: dev-elastic-monitoring
  namespace: dev
  monitoring_enabled: ["logs", "metrics"]
  inactivity_timeout: 1209600
  is_protected: false
  package_policies:
  - id: elasticsearch-dev-elastic-monitoring
    package:
      name: elasticsearch
    name: elasticsearch-dev-elastic-monitoring
    description: ''
    namespace: ''
    inputs:
      elasticsearch-logfile:
        enabled: true
        vars: {}
        streams:
          'elasticsearch.audit': 
            enabled: true
            vars:
              paths:
              - "/var/log/elasticsearch/*_audit.json"
              tags: []

error:

Error: [config validation of [xpack.fleet].agentPolicies.0.package_policies.0]: types that failed validation:
- [config validation of [xpack.fleet].agentPolicies.0.package_policies.0.0.inputs]: expected value of type [array] but got [Object]
- [config validation of [xpack.fleet].agentPolicies.0.package_policies.0.1.inputs.elasticsearch-logfile.streams.elasticsearch.audit]: definition for this key is missing