Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Change Proposal] Validating event.original implementation + ECS version in pipeline #177

Closed
P1llus opened this issue May 31, 2021 · 1 comment
Labels
discuss Issue needs discussion

Comments

@P1llus
Copy link
Member

P1llus commented May 31, 2021

This Issue is to track the possibility to both add these specific fields to the package-spec for newly created packages + linting to make sure they exist.

Currently these are the changes:
Event.original implementation: (Example can be found on the apache package)
The manifest.yml should have this option for all non-metric packages (is this possible to differentiate?)

    vars:
      - name: preserve_original_event
        required: true
        show_user: true
        title: Preserve original event
        description: Preserves a raw copy of the original event, added to the field `event.original`
        type: bool
        multi: false
        default: false

The HBS file should include:

tags:
{{#if preserve_original_event}}
  - preserve_original_event
{{/if}}
{{#each tags as |tag i|}}
- {{tag}}
{{/each}}

In the pipeline these two processors should be included:
At the top

  - rename:
      field: message
      target_field: event.original
      ignore_missing: true

At the bottom:

  - remove:
      field: event.original
      if: "ctx?.tags == null || !(ctx.tags.contains('preserve_original_event'))"
      ignore_failure: true
      ignore_missing: true

The common pipeline test file should at least set the preservation tag:

fields:
  tags:
    - preserve_original_event

ECS versions implementation
All pipelines for packages that are non-metrics (since they put ECS versions in code instead) should have a pipeline processor like this:

  - set:
      field: ecs.version
      value: "1.10.0"

Allow Custom processors
To be able to allow custom processors, all packages should have these changes:
HBS files:

processors:
{{processors}}

Menu item in manifests:

      - name: processors
        type: yaml
        title: Processors
        multi: false
        required: false
        show_user: false
        description: >
          Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
@jsoriano
Copy link
Member

I am going to close this one because we have other more specific issues about the problems described here.

Default variables, such as preserve_original_event will be added as part of elastic/kibana#170336.

Consolidation of ecs.version to be done in #737.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Issue needs discussion
Projects
None yet
Development

No branches or pull requests

2 participants