[google_secops] Initial release of the google secops #12767

janvi-elastic · 2025-02-13T07:58:18Z

Proposed commit message

Create new integration package google_secops.

Added data stream.
Added data collection logic for alert data stream.
Added the ingest pipeline for alert data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for alert data stream.
Added system test cases for alert data stream.
Add agentless deployment

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/google_secops directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: google_secops - START ---
╭───────────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                                                              │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ google_secops │             │ asset     │ dashboard google_secops-517f96c9-1d44-420a-9ff4-96cf28de00d7 is loaded │ PASS   │      1.619µs │
│ google_secops │             │ asset     │ search google_secops-5407845c-e294-4f99-a342-9dc7e05dc8e0 is loaded    │ PASS   │        273ns │
│ google_secops │             │ asset     │ search google_secops-6a9de76e-e146-484e-ad87-fdb29c937fa2 is loaded    │ PASS   │        205ns │
│ google_secops │ alert       │ asset     │ index_template logs-google_secops.alert is loaded                      │ PASS   │        244ns │
│ google_secops │ alert       │ asset     │ ingest_pipeline logs-google_secops.alert-0.1.0 is loaded               │ PASS   │        253ns │
╰───────────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: google_secops - END   ---
Done
--- Test results for package: google_secops - START ---
╭───────────────┬─────────────┬───────────┬───────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼───────────────────────────────────────────┼────────┼──────────────┤
│ google_secops │ alert       │ pipeline  │ (ingest pipeline warnings test-alert.log) │ PASS   │ 288.418232ms │
│ google_secops │ alert       │ pipeline  │ test-alert.log                            │ PASS   │ 342.148692ms │
╰───────────────┴─────────────┴───────────┴───────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: google_secops - END   ---
Done
--- Test results for package: google_secops - START ---
╭───────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ google_secops │ alert       │ static    │ Verify sample_event.json │ PASS   │ 239.740171ms │
╰───────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: google_secops - END   ---
Done
--- Test results for package: google_secops - START ---
╭───────────────┬─────────────┬───────────┬───────────┬────────┬────────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │   TIME ELAPSED │
├───────────────┼─────────────┼───────────┼───────────┼────────┼────────────────┤
│ google_secops │ alert       │ system    │ default   │ PASS   │ 2m8.093218093s │
╰───────────────┴─────────────┴───────────┴───────────┴────────┴────────────────╯
--- Test results for package: google_secops - END   ---
Done

Screenshot

elastic-vault-github-plugin-prod · 2025-02-13T08:18:00Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

jamiehynds · 2025-02-13T09:26:58Z

packages/google_secops/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

+      description: Script to set event.severity.
+      tag: set_event_severity
+      source: |-
+        if (ctx.google_secops?.alert?.event?.securityResult instanceof List) {


We've discovered an issue across all of our integrations, whereby event.severity is not mapped to the values expected by Elastic Security. This is a result of ambiguity in the ECS field description that we're also working to address (https://www.elastic.co/guide/en/ecs/current/ecs-event.html#field-event-severity)

Could we map the event.severity values to the values expected by Elastic Security, to ensure the severity from 3rd party alerts will be assigned the correct severity within Elastic? It's a numeric scale:

Low - 21
Medium -47
High- 73
Critical - 99

cc @raqueltabuyo @cpascale43

@jamiehynds We have mapped event.severity to the values given in the Google SecOps documentation.

Let me know if we have to update the values as the values expected by Elastic Security.

@janvi-elastic , yes please update according to Elastic Security guidelines stated above.
The custom field will always be there if users wants the source severity.

You can add a note to the README to mentioning the same.

elasticmachine · 2025-02-13T10:25:51Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy

Please check if we are able to ingest alert status updates from the API.
For example from ALERTING -> NOT_ALERTING.

If so, we will need a latest transform to store the last state.

packages/google_secops/_dev/build/docs/README.md

kcreddy · 2025-02-14T07:33:39Z

packages/google_secops/img/google-secops-overview-screenshot.png

Can you resolve an alert and check if you are getting the updated document?

ALERTING -> NOT_ALERTING

packages/google_secops/_dev/deploy/docker/files/config.yml

packages/google_secops/data_stream/alert/sample_event.json

packages/google_secops/data_stream/alert/_dev/test/system/test-default-config.yml

packages/google_secops/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

jamiehynds · 2025-02-18T09:12:30Z

packages/google_secops/_dev/build/docs/README.md

+
+For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html)
+
+> Note: Currently Agentless support for Microsoft Sentinel integration is limited to API and not supported for Azure Event Hub input.


Microsoft Sentinel shouldn't be mentioned in the Google SecOps docs, presume a copy/paste error.

jamiehynds · 2025-02-18T09:14:44Z

packages/google_secops/_dev/build/docs/README.md

+
+> Note: Currently Agentless support for Microsoft Sentinel integration is limited to API and not supported for Azure Event Hub input.
+
+### Agent-based deployment


@alaudazzi can you advise how we're handing these agentless going forward? With this issue in mind, I want to ensure we adhere for any new integrations developed. https://github.com/elastic/integration-docs/issues/653

jamiehynds · 2025-02-18T09:15:24Z

packages/google_secops/manifest.yml

+        enabled: true
+        organization: security
+        division: engineering
+        team: security-service-integrations
    inputs:
      - type: cel
        title: Collect Google SecOps logs via API


Can we change logs to alerts, as the scope of the integration is limited to alerts from SecOps.

jamiehynds · 2025-02-18T09:15:36Z

packages/google_secops/manifest.yml

@@ -26,6 +26,14 @@ policy_templates:
  - name: google_secops
    title: Google SecOps logs


Change from logs to alerts.

jamiehynds · 2025-02-18T09:15:42Z

packages/google_secops/manifest.yml

@@ -26,6 +26,14 @@ policy_templates:
  - name: google_secops
    title: Google SecOps logs
    description: Collect Google SecOps logs.


Change from logs to alerts.

packages/google_secops/_dev/build/docs/README.md

packages/google_secops/_dev/deploy/docker/files/config.yml

packages/google_secops/data_stream/alert/_dev/test/pipeline/test-alert.log

packages/google_secops/data_stream/alert/agent/stream/cel.yml.hbs

packages/google_secops/data_stream/alert/fields/beats.yml

packages/google_secops/data_stream/alert/fields/fields.yml

packages/google_secops/manifest.yml

efd6 · 2025-02-20T02:56:00Z

packages/google_secops/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

+                            "ImpersonationLevel": "%33",
+                            "KeyLength": "0",
+                            "Keywords": "-9214364837600035000",
+                            "LmPackageName": "-",


Do we want to remove "-" fields?

efd6

LGTM but please wait for @kcreddy.

kcreddy

My primary concern is from #12767 (comment), we didn't test for events with alert's state change.
This might mean the alerts will keep growing in Kibana, and no way to find Current Active Alerts.

packages/google_secops/manifest.yml

janvi-elastic · 2025-02-25T11:11:30Z

My primary concern is from #12767 (comment), we didn't test for events with alert's state change. This might mean the alerts will keep growing in Kibana, and no way to find Current Active Alerts.

In our instance, we don't have an option to update the state of alert.

elasticmachine · 2025-02-25T11:29:35Z

💚 Build Succeeded

Buildkite Build
Commit: 5eb85c9

History

💚 Build #22588 succeeded 4b079da
💚 Build #22504 succeeded 8dc68c2
💚 Build #22443 succeeded f0061aa
💚 Build #22439 succeeded 98d6886
💚 Build #22240 succeeded 58bdca1

elastic-sonarqube · 2025-02-25T11:29:41Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

kcreddy · 2025-02-25T12:30:21Z

My primary concern is from #12767 (comment), we didn't test for events with alert's state change. This might mean the alerts will keep growing in Kibana, and no way to find Current Active Alerts.

In our instance, we don't have an option to update the state of alert.

@jamiehynds,
As per @janvi-elastic, changing alert state is possible in SOAR, but they have only SIEM access. Due to this we are unable to test by closing an alert and whether we ingest the updated state.

If the API doesn't send updated states, we are okay.
If the API does send updated states, then we ingest the alert again in its new state (because we don't have fingerprint processor), thus leading to duplicates of same alert inside the dashboard. i.e., an alert can be in both Active and Close states. The only way to fix this is through latest transform.

I will leave it to you to take the call on merging the PR without this test.

piyush-elastic · 2025-02-26T06:32:47Z

My primary concern is from #12767 (comment), we didn't test for events with alert's state change. This might mean the alerts will keep growing in Kibana, and no way to find Current Active Alerts.

In our instance, we don't have an option to update the state of alert.

@jamiehynds, As per @janvi-elastic, changing alert state is possible in SOAR, but they have only SIEM access. Due to this we are unable to test by closing an alert and whether we ingest the updated state.

If the API doesn't send updated states, we are okay. If the API does send updated states, then we ingest the alert again in its new state (because we don't have fingerprint processor), thus leading to duplicates of same alert inside the dashboard. i.e., an alert can be in both Active and Close states. The only way to fix this is through latest transform.

I will leave it to you to take the call on merging the PR without this test.

@kcreddy , @jamiehynds -Unfortunately, we don't have access to the SOAR component. We can add transform as a precaution, but we are unable to determine which field leads to updated_detection. Alternatively, if we can confirm the relevant fields, we can test it using a Mockserver. Please find the list of supported response fields here.

kcreddy · 2025-02-27T07:29:27Z

@kcreddy , @jamiehynds -Unfortunately, we don't have access to the SOAR component. We can add transform as a precaution, but we are unable to determine which field leads to updated_detection. Alternatively, if we can confirm the relevant fields, we can test it using a Mockserver. Please find the list of supported response fields here.

@piyush-elastic, thanks for the info.

If we can't get access to SOAR, here's my suggestion:
The field responsible is lastUpdatedTime, but it is not available through the ListDetections API, i.e., not available in the rule-based detections (type: RULE_DETECTION).
It is available in API response of curated detections, i.e., threat intel detections (type: GCTI_FINDING).

At this point, we are unsure if the updated alert state comes through the List Detections API. We may not want to add transform as precaution, because we don't have a proper way to delete the transform and its destination index automatically.
In case later we find the updated alerts are not available through the API and there is no need for transform, then users will have to delete their existing transform and also the index manually, which doesn't seem ideal experience.

We can go ahead with what we have right now and collect user feedback on any duplicate alerts (with different states) in their indices. We can add transform if duplicate alerts are ingested.

@jamiehynds does that sound good?

kcreddy

Waiting on #12767 (comment) for merging.

janvi-elastic added 2 commits February 13, 2025 13:19

Initial Release

9a0e550

Update changelog

58bdca1

jamiehynds reviewed Feb 13, 2025

View reviewed changes

kcreddy added New Integration Issue or pull request for creating a new integration package. Crest Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Feb 13, 2025

kcreddy added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Feb 13, 2025

kcreddy reviewed Feb 14, 2025

View reviewed changes

Resolve review comments

98d6886

jamiehynds reviewed Feb 18, 2025

View reviewed changes

janvi-elastic added 2 commits February 18, 2025 15:59

Resolve review comments

f0061aa

Add alert friendly name and update event.severity

8dc68c2

efd6 reviewed Feb 20, 2025

View reviewed changes

Resolve review comments

4b079da

efd6 approved these changes Feb 24, 2025

View reviewed changes

kcreddy reviewed Feb 25, 2025

View reviewed changes

packages/google_secops/manifest.yml Show resolved Hide resolved

packages/google_secops/manifest.yml Show resolved Hide resolved

Update categories in manifest

5eb85c9

janvi-elastic requested a review from kcreddy February 25, 2025 11:31

kcreddy reviewed Mar 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[google_secops] Initial release of the google secops #12767

[google_secops] Initial release of the google secops #12767

janvi-elastic commented Feb 13, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 13, 2025

jamiehynds Feb 13, 2025

janvi-elastic Feb 14, 2025

kcreddy Feb 14, 2025

elasticmachine commented Feb 13, 2025

kcreddy left a comment

kcreddy Feb 14, 2025

jamiehynds Feb 18, 2025

jamiehynds Feb 18, 2025

jamiehynds Feb 18, 2025

jamiehynds Feb 18, 2025

jamiehynds Feb 18, 2025

efd6 Feb 20, 2025

efd6 left a comment

kcreddy left a comment

janvi-elastic commented Feb 25, 2025

elasticmachine commented Feb 25, 2025

elastic-sonarqube bot commented Feb 25, 2025

kcreddy commented Feb 25, 2025 •

edited

Loading

piyush-elastic commented Feb 26, 2025

kcreddy commented Feb 27, 2025 •

edited

Loading

kcreddy left a comment


		For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html)

		> Note: Currently Agentless support for Microsoft Sentinel integration is limited to API and not supported for Azure Event Hub input.


		> Note: Currently Agentless support for Microsoft Sentinel integration is limited to API and not supported for Azure Event Hub input.

		### Agent-based deployment

		@@ -26,6 +26,14 @@ policy_templates:
		- name: google_secops
		title: Google SecOps logs

[google_secops] Initial release of the google secops #12767

Are you sure you want to change the base?

[google_secops] Initial release of the google secops #12767

Conversation

janvi-elastic commented Feb 13, 2025 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Screenshot

elastic-vault-github-plugin-prod bot commented Feb 13, 2025

🚀 Benchmarks report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Feb 13, 2025

kcreddy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

janvi-elastic commented Feb 25, 2025

elasticmachine commented Feb 25, 2025

💚 Build Succeeded

History

elastic-sonarqube bot commented Feb 25, 2025

Quality Gate passed

kcreddy commented Feb 25, 2025 • edited Loading

piyush-elastic commented Feb 26, 2025

kcreddy commented Feb 27, 2025 • edited Loading

kcreddy left a comment

Choose a reason for hiding this comment

janvi-elastic commented Feb 13, 2025 •

edited

Loading

kcreddy commented Feb 25, 2025 •

edited

Loading

kcreddy commented Feb 27, 2025 •

edited

Loading