[Fleet][Discuss] Add removal of event.original to the Fleet final pipeline

[jsoriano]

Describe the feature:

Add the removal of event.original to the Fleet final pipeline, so this removal doesn't need to be included in ingest pipelines in packages.

It would be to add a processor like this one to the final pipeline:

  {
    "remove": {
      "field": "event.original",
      "if": "ctx?.tags == null || !(ctx.tags.contains('preserve_original_event'))",
      "ignore_failure": true,
      "ignore_missing": true
    }
  }

Fleet could also have an advanced setting that can be used to inject the tag in specific data streams. If enabled, it would inject a pipeline processor to add the tag if not already present. So this logic would neither need to be included in packages.

Describe a specific use case for the feature:

Users have access to event.original in custom pipelines.

Current approach is to remove this field from the pipeline included in the package, that is executed before the custom pipelines, so users don't have access to this field in custom processing.

Eventually packages could remove the removal from their pipelines.

Current workaround:

Users can enable "Preserve original event", then they have access to original.event in custom processing and can also decide to delete the field there.

Related issues:

• Make event.original available to the custom pipeline integrations#7636
• [Change Proposal] Validating event.original implementation + ECS version in pipeline package-spec#177
• Add new validation rule to avoid rename processor without checking event.original package-spec#583

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[termcap]

Another solution could be making the "Remove" processor the last processor in the default pipeline. Currently the last processor is the custom pipeline.