Merge branch 'main' into memfd_stage0

CHANGELOG.next.md

@@ -13,12 +13,14 @@ Thanks, you're awesome :-) -->
 #### Breaking changes
 
 #### Bugfixes
+* Fix broken link in docs for vulnerability.id. #2328
 
 #### Added
 
 * Added `volume.*` as beta field set. #2269
 * Advanced `process.env_vars` to GA. #2315
 * Advanced `process.io` and `process.tty` fields to GA. #2317
+* Added `threat.indicator.id`. #2324
 
 #### Improvements
 
@@ -32,6 +34,8 @@ Thanks, you're awesome :-) -->
 
 #### Added
 
+* Documentation in README.md providing instruction on contributions to ECS during the OTel donation #2325
+
 #### Improvements
 
 #### Deprecated

CONTRIBUTING.md

@@ -7,7 +7,8 @@ ECS is an open source project and we love to receive contributions from our comm
 
 ## Table of Contents
 
-- [How to Contribute](#how-to-contribute)
+- [How to contribute](#how-to-contribute)
+- - [Special guidance during OTel donation of ECS](#special-guidance-during-otel-donation-of-ecs)
   - [Dev Tools](#dev-tools)
   - [Submitting Changes](#submitting-changes)
 - [Git and Github Guidelines](#git-and-github-guidelines)
@@ -22,12 +23,67 @@ ECS is an open source project and we love to receive contributions from our comm
 - [Schema Files](#schema-files)
 - [Additional Resources](#additional-resources)
 
-## How to Contribute
 
-There are two primary ways in which you can contribute to ECS.
+## How to contribute
 
-1. The [RFC process](./rfcs/README.md) is used for significant additions or breaking changes to the schema itself.
-2. For bug fixes or incremental, non-controversial additions to ECS, changes can be made directly to the ECS project and submitted as pull request.
+## ECS donation to OpenTelemetry
+In April 2023, OpenTelemetry and Elastic made an [important joint announcement](https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/). In this announcement Elastic
+shared its intention to achieve convergence of ECS and OTel Semantic Conventions into a single standard maintained
+by OpenTelemetry.
+
+The stated plan has been to keep ECS in a frozen state during the transition. However, it is also apparent that these
+things take time. It takes time for the OTel community to adopt donated fields, and it will take time for development
+teams to build OTel native constructs in the Elastic stack. In the meantime, ECS users need to be able to develop
+features for Elastic that rely on continued contributions to the schemas that drive our technology.
+
+For these reasons, we need a process and guidelines for contributing to these data schemas during this period that
+allows us to avoid breaking changes.
+
+### ECS releases during the donation to OpenTelemetry
+Historically, ECS has shipped a new version with every minor release of the Elastic stack. While the schema is
+effectively frozen during the Otel donation period, this approach has been halted.
+
+Moving forward, we will release ECS at the team's discretion as new material changes to the schema are adopted.
+
+While the decision to release will be discretionary, any release will still coincide with a minor Elastic stack release;
+however, not every minor version will warrant a new release of the stack.
+
+### How to contribute during OTel donation of ECS
+
+Bug fixes or minor field addition changes can be made directly to the ECS project and submitted as pull requests.
+
+Significant changes that add new use cases, top-level fieldsets, or could be considered controversial are
+considered material. The general rule for contributing new material changes to schemas during the transition period is
+
+- First, merge a pull request to
+[OTel Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/CONTRIBUTING.md) with new
+fields, namespaces or schemas
+- Second, to backport those changes to ECS at the starting point indicated in the table below
+- Finally, once the Semantic Conventions changes are marked as stable, remove the Beta designation in ECS
+
+This will ensure that the latest changes are included in OTel Semantic Conventions, where schema evolution will continue
+as the merger proceeds. It will also allow teams and users to continue using ECS while OTel migration tools and guidance
+are being developed. Finally, this will reduce the risk of breaking changes if new fields are merged first to ECS, and
+then require changes before being adopted in Semantic Conventions.
+
+_There are some exceptions to this rule._
+
+1. My contribution to OTel Semantic Conventions is stalled. We are waiting for a sign-off from a second company.
+In the meantime, our Elastic feature is blocked.
+2. I want to build a workflow in Elastic, and the fields I need to proceed are already in OTel but not in ECS where I
+need them today.
+
+In these cases, the recommendation is to make a contribution to ECS to unblock development. The appropriate ECS starting
+point can be an [RFC](./rfcs/README.md) or pull request based on the maturity of the Otel changes. Please see the
+following table.
+
+| OTel submission maturity                                                                                                                                                             | Breaking changes expected | ECS starting point                                   |
+|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|------------------------------------------------------|
+| OTel working groups accepts the premise of the addition and commits to considering this proposal as it advances.                                                                     | Major                     | RFC Stage 1                                          |
+| The initial field definitions comprehensively model the addition to the schema. Fundamental questions and concerns are resolved, though some less significant questions remain open. | Iterative                 | RFC Stage 2                                          |
+| All requested changes from codeowners have been addressed, and there are no open discussions.                                                                                        | Iterative                 | Open an ECS pull request with new fields marked Beta |
+| Fields, schema, namespace exists in OTel and are designated experimental                                                                                                             | Iterative                 | Open an ECS pull request with new fields marked Beta |
+| Fields, schema, namespace exists in OTel and are designated stable                                                                                                                   | None                      | Open an ECS pull request with new fields marked GA   |
 
 ### Dev Tools
 

README.md

@@ -9,9 +9,18 @@ ingesting data into Elasticsearch. A common schema helps you correlate
 data from sources like logs and metrics or IT operations
 analytics and security analytics.
 
+## ECS Donation to OpenTelemetry
+In April 2023, OpenTelemetry and Elastic made an
+[important joint announcement](https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/).
+In this announcement, we shared our intention to achieve convergence of ECS and OTel
+Semantic Conventions into a single standard maintained by OpenTelemetry.
+
+Special guidance is provided during the donation period. Please review the [contribution guide](CONTRIBUTING.md).
+
 ## Documentation
 
 The ECS reference is published on the main Elastic documentation website.
+
 Visit [the official ECS Reference Documentation](https://www.elastic.co/guide/en/ecs/current/index.html).
 
 ## Getting Started

docs/fields/field-details.asciidoc

@@ -10953,6 +10953,27 @@ example: `2020-11-05T17:25:47.000Z`
 
 // ===============================================================
 
+|
+[[field-threat-indicator-id]]
+<<field-threat-indicator-id, threat.indicator.id>>
+
+a| The ID of the indicator used by this threat to conduct behavior commonly modeled using MITRE ATT&CK®. This field can have multiple values to allow for the identification of the same indicator across systems that use different ID formats.
+
+While not required, a common approach is to use a STIX 2.x indicator ID.
+
+type: keyword
+
+
+Note: this field should contain an array of values.
+
+
+
+example: `[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]`
+
+| extended
+
+// ===============================================================
+
 |
 [[field-threat-indicator-ip]]
 <<field-threat-indicator-ip, threat.indicator.ip>>
@@ -13300,7 +13321,7 @@ example: `CVE`
 [[field-vulnerability-id]]
 <<field-vulnerability-id, vulnerability.id>>
 
-a| The identification (ID) is the number portion of a vulnerability entry. It includes a unique identification number for the vulnerability. For example (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities and Exposure CVE ID]
+a| The identification (ID) is the number portion of a vulnerability entry. It includes a unique identification number for the vulnerability. For example (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities and Exposure CVE ID])
 
 type: keyword
 

experimental/generated/beats/fields.ecs.yml

@@ -11645,6 +11645,17 @@
       description: The time zone of the location, such as IANA time zone name.
       example: America/Argentina/Buenos_Aires
       default_field: false
+    - name: indicator.id
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: "The ID of the indicator used by this threat to conduct behavior\
+        \ commonly modeled using MITRE ATT&CK\xAE. This field can have multiple values\
+        \ to allow for the identification of the same indicator across systems that\
+        \ use different ID formats.\nWhile not required, a common approach is to use\
+        \ a STIX 2.x indicator ID."
+      example: '[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]'
+      default_field: false
     - name: indicator.ip
       level: extended
       type: ip
@@ -13636,8 +13647,8 @@
       ignore_above: 1024
       description: The identification (ID) is the number portion of a vulnerability
         entry. It includes a unique identification number for the vulnerability. For
-        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities
-        and Exposure CVE ID]
+        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities
+        and Exposure CVE ID])
       example: CVE-2019-00001
       default_field: false
     - name: reference

experimental/generated/csv/fields.csv

@@ -1500,6 +1500,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 8.12.0-dev+exp,true,threat,threat.indicator.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
 8.12.0-dev+exp,true,threat,threat.indicator.geo.region_name,keyword,core,,Quebec,Region name.
 8.12.0-dev+exp,true,threat,threat.indicator.geo.timezone,keyword,core,,America/Argentina/Buenos_Aires,Time zone.
+8.12.0-dev+exp,true,threat,threat.indicator.id,keyword,extended,array,[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37],ID of the indicator
 8.12.0-dev+exp,true,threat,threat.indicator.ip,ip,extended,,1.2.3.4,Indicator IP address
 8.12.0-dev+exp,true,threat,threat.indicator.last_seen,date,extended,,2020-11-05T17:25:47.000Z,Date/time indicator was last reported.
 8.12.0-dev+exp,true,threat,threat.indicator.marking.tlp,keyword,extended,,CLEAR,Indicator TLP marking

experimental/generated/ecs/ecs_flat.yml

@@ -18960,6 +18960,22 @@ threat.indicator.geo.timezone:
   original_fieldset: geo
   short: Time zone.
   type: keyword
+threat.indicator.id:
+  dashed_name: threat-indicator-id
+  description: "The ID of the indicator used by this threat to conduct behavior commonly\
+    \ modeled using MITRE ATT&CK\xAE. This field can have multiple values to allow\
+    \ for the identification of the same indicator across systems that use different\
+    \ ID formats.\nWhile not required, a common approach is to use a STIX 2.x indicator\
+    \ ID."
+  example: '[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]'
+  flat_name: threat.indicator.id
+  ignore_above: 1024
+  level: extended
+  name: indicator.id
+  normalize:
+  - array
+  short: ID of the indicator
+  type: keyword
 threat.indicator.ip:
   dashed_name: threat-indicator-ip
   description: Identifies a threat indicator as an IP address (irrespective of direction).
@@ -22158,8 +22174,8 @@ vulnerability.id:
   dashed_name: vulnerability-id
   description: The identification (ID) is the number portion of a vulnerability entry.
     It includes a unique identification number for the vulnerability. For example
-    (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities
-    and Exposure CVE ID]
+    (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities and
+    Exposure CVE ID])
   example: CVE-2019-00001
   flat_name: vulnerability.id
   ignore_above: 1024

experimental/generated/ecs/ecs_nested.yml

@@ -21632,6 +21632,22 @@ threat:
       original_fieldset: geo
       short: Time zone.
       type: keyword
+    threat.indicator.id:
+      dashed_name: threat-indicator-id
+      description: "The ID of the indicator used by this threat to conduct behavior\
+        \ commonly modeled using MITRE ATT&CK\xAE. This field can have multiple values\
+        \ to allow for the identification of the same indicator across systems that\
+        \ use different ID formats.\nWhile not required, a common approach is to use\
+        \ a STIX 2.x indicator ID."
+      example: '[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]'
+      flat_name: threat.indicator.id
+      ignore_above: 1024
+      level: extended
+      name: indicator.id
+      normalize:
+      - array
+      short: ID of the indicator
+      type: keyword
     threat.indicator.ip:
       dashed_name: threat-indicator-ip
       description: Identifies a threat indicator as an IP address (irrespective of
@@ -25142,8 +25158,8 @@ vulnerability:
       dashed_name: vulnerability-id
       description: The identification (ID) is the number portion of a vulnerability
         entry. It includes a unique identification number for the vulnerability. For
-        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities
-        and Exposure CVE ID]
+        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities
+        and Exposure CVE ID])
       example: CVE-2019-00001
       flat_name: vulnerability.id
       ignore_above: 1024

experimental/generated/elasticsearch/composable/component/threat.json

@@ -1522,6 +1522,10 @@
                     }
                   }
                 },
+                "id": {
+                  "ignore_above": 1024,
+                  "type": "keyword"
+                },
                 "ip": {
                   "type": "ip"
                 },

experimental/generated/elasticsearch/legacy/template.json

@@ -6727,6 +6727,10 @@
                   }
                 }
               },
+              "id": {
+                "ignore_above": 1024,
+                "type": "keyword"
+              },
               "ip": {
                 "type": "ip"
               },

generated/beats/fields.ecs.yml

@@ -11595,6 +11595,17 @@
       description: The time zone of the location, such as IANA time zone name.
       example: America/Argentina/Buenos_Aires
       default_field: false
+    - name: indicator.id
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: "The ID of the indicator used by this threat to conduct behavior\
+        \ commonly modeled using MITRE ATT&CK\xAE. This field can have multiple values\
+        \ to allow for the identification of the same indicator across systems that\
+        \ use different ID formats.\nWhile not required, a common approach is to use\
+        \ a STIX 2.x indicator ID."
+      example: '[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]'
+      default_field: false
     - name: indicator.ip
       level: extended
       type: ip
@@ -13586,8 +13597,8 @@
       ignore_above: 1024
       description: The identification (ID) is the number portion of a vulnerability
         entry. It includes a unique identification number for the vulnerability. For
-        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities
-        and Exposure CVE ID]
+        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities
+        and Exposure CVE ID])
       example: CVE-2019-00001
       default_field: false
     - name: reference

generated/csv/fields.csv

@@ -1493,6 +1493,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 8.12.0-dev,true,threat,threat.indicator.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
 8.12.0-dev,true,threat,threat.indicator.geo.region_name,keyword,core,,Quebec,Region name.
 8.12.0-dev,true,threat,threat.indicator.geo.timezone,keyword,core,,America/Argentina/Buenos_Aires,Time zone.
+8.12.0-dev,true,threat,threat.indicator.id,keyword,extended,array,[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37],ID of the indicator
 8.12.0-dev,true,threat,threat.indicator.ip,ip,extended,,1.2.3.4,Indicator IP address
 8.12.0-dev,true,threat,threat.indicator.last_seen,date,extended,,2020-11-05T17:25:47.000Z,Date/time indicator was last reported.
 8.12.0-dev,true,threat,threat.indicator.marking.tlp,keyword,extended,,CLEAR,Indicator TLP marking

generated/ecs/ecs_flat.yml

@@ -18891,6 +18891,22 @@ threat.indicator.geo.timezone:
   original_fieldset: geo
   short: Time zone.
   type: keyword
+threat.indicator.id:
+  dashed_name: threat-indicator-id
+  description: "The ID of the indicator used by this threat to conduct behavior commonly\
+    \ modeled using MITRE ATT&CK\xAE. This field can have multiple values to allow\
+    \ for the identification of the same indicator across systems that use different\
+    \ ID formats.\nWhile not required, a common approach is to use a STIX 2.x indicator\
+    \ ID."
+  example: '[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]'
+  flat_name: threat.indicator.id
+  ignore_above: 1024
+  level: extended
+  name: indicator.id
+  normalize:
+  - array
+  short: ID of the indicator
+  type: keyword
 threat.indicator.ip:
   dashed_name: threat-indicator-ip
   description: Identifies a threat indicator as an IP address (irrespective of direction).
@@ -22089,8 +22105,8 @@ vulnerability.id:
   dashed_name: vulnerability-id
   description: The identification (ID) is the number portion of a vulnerability entry.
     It includes a unique identification number for the vulnerability. For example
-    (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities
-    and Exposure CVE ID]
+    (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities and
+    Exposure CVE ID])
   example: CVE-2019-00001
   flat_name: vulnerability.id
   ignore_above: 1024

generated/ecs/ecs_nested.yml

@@ -21552,6 +21552,22 @@ threat:
       original_fieldset: geo
       short: Time zone.
       type: keyword
+    threat.indicator.id:
+      dashed_name: threat-indicator-id
+      description: "The ID of the indicator used by this threat to conduct behavior\
+        \ commonly modeled using MITRE ATT&CK\xAE. This field can have multiple values\
+        \ to allow for the identification of the same indicator across systems that\
+        \ use different ID formats.\nWhile not required, a common approach is to use\
+        \ a STIX 2.x indicator ID."
+      example: '[indicator--d7008e06-ab86-415a-9803-3c81ce2d3c37]'
+      flat_name: threat.indicator.id
+      ignore_above: 1024
+      level: extended
+      name: indicator.id
+      normalize:
+      - array
+      short: ID of the indicator
+      type: keyword
     threat.indicator.ip:
       dashed_name: threat-indicator-ip
       description: Identifies a threat indicator as an IP address (irrespective of
@@ -25062,8 +25078,8 @@ vulnerability:
       dashed_name: vulnerability-id
       description: The identification (ID) is the number portion of a vulnerability
         entry. It includes a unique identification number for the vulnerability. For
-        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id)[Common Vulnerabilities
-        and Exposure CVE ID]
+        example (https://cve.mitre.org/about/faqs.html#what_is_cve_id[Common Vulnerabilities
+        and Exposure CVE ID])
       example: CVE-2019-00001
       flat_name: vulnerability.id
       ignore_above: 1024