[system tests] Validate fields are documented via the mappings generated after ingesting docs comparing them to the preview mappings

[mrodm]

In order to try to be agnostic to the structure of the documents ingested to run the validation in system tests. It would be helpful to run validations comparing the mapping definitions instead.

These validations should be done with these two sets of mappings:

• Mappings and dynamic templates installed by Fleet before ingesting any doc.
  • These preview mappings can be retrieved using this simulated API:

    POST /_index_template/_simulate/<index_template_name>
    
    # Example
    POST /_index_template/_simulate/logs-microsoft_dhcp.log
    

• Mappings and dynamic templates that are present after ingesting the docs as part of the system tests.
  • These mappings can be retrieved using this API:

    GET /<data_stream_test>/_mapping/
    

Being available these two sets of mappings, it would be needed to validate whether or not each mapping present in the data stream (after ingesting docs) has a corresponding mapping in the preview.

It should be taken into account that:

• Some fields should be skipped (e.g. agent.*, host.*, ...) as it is done nowadays (link)
• Ignore for now those mappings that are not found in the preview (to be done in the next step).
  • To be done in a follow-up issue.
• TBD check if any mapping that does not exist in the preview is part of the ECS.

For now, the errors that could be raised as part of this validation should not make fail the validation process, therefore:

• Ignore at this step mappings not found in the preview, to be done in the next step.
• Enable via environment variable this mapping or show just just warning messages.

To be tested:

• Run these validations in stack 7.x
• Run these validations in stack 8.x
• Run these validations in input and integration packages.
• Run these validations in Stacks with LogsDB enabled (synthetics).

[mrodm]

Some assumptions made when adding this new validation based on mappings:

• Objects with type array are skipped.
  • These field definition do not generate any mapping.
  • Type array is checked using the local field definitions in the package.
    • Requires to load in memory also the field definitions from the data stream or input package.
  • These kind of definitions were allowed in spec versions <= 2.0.0 (link)
• Empty objects present in the actual mappings are skipped:

  "_tmp": {
    "type": "object"  
  }

• If a field is defined as a constant_keyword, just check/validate the value if both (preview and actual mappings) have "value" parameter.
• If an object contains dynamic: true:
  • skip the validation of those field mappings under a mapping like
    • should they be validated using the dynamic template definitions?

      "field": {
        "dynamic": true,
        "type": "object"
      }

  • The other field mappings must be checked as regular field mappings.

[mrodm]

While working on #2214 , there have been some issues found while testing with the packages in the integration repository that are not related to the dynamic templates. Support to validate mappings with the dynamic templates is going to be tackle in #2207

Issues found are detailed in #2214 (comment)

In summary, those issues are:

• Failures related to different mappings found in ECS
  • Due to the embedded ECS or the ecs@mappings component template
  • Some examples:
    • Add initial support to validate the mappings in system tests #2214 (comment)
    • Add initial support to validate the mappings in system tests #2214 (comment)
• Failures while validating mappings with type long, float or double Add initial support to validate the mappings in system tests #2214 (comment)
• Nested objects Add initial support to validate the mappings in system tests #2214 (comment)
  • Skip for spec versions <3.0.1
• Array of objects defined as group in the package Add initial support to validate the mappings in system tests #2214 (comment)

[mrodm]

With the latest changes introduced in the PR, the missing failures are related to different mappings found in ECS. That could probably be checked as part of the following issue where dynamic templates are going to be considered #2207

WDYT @jsoriano ?

Nested and groups field/mappings should be fixed , testing it in elastic/integrations#11828 (build https://buildkite.com/elastic/integrations/builds/19362).

Found an unexpected error for tines package, described in #2214 (comment)

[jsoriano]

With the latest changes introduced in the PR, the missing failures are related to different mappings found in ECS. That could probably be checked as part of the following issue where dynamic templates are going to be considered #2207

Yes, I think we can go on with current implementation and polish it in follow ups.

Found an unexpected error for tines package, described in #2214 (comment)

This looks like an issue in Fleet (see #2214 (comment)), I will take a deeper look tomorrow.