microsoft_sentinel: Add agentless deployment

[kcreddy]

Proposed commit message

As part of onboarding integrations into agentless deployments, 
this PR adds agentless deployment to Microsoft Sentinel integration.

Ref:
- https://docs.elastic.dev/security-solution/cloud-security/agentless/onboard-integration

cc: @jamiehynds

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Screenshots

Successful ingestion in Cloud environment (8.18.0):

Successful ingestion in Serverless environment:

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 26e5fdeaa77f369e7cc9b67cce0a89363f267aa2

History

• 💔 Build #21543 failed 33a051c532f931067332dc21173dd53412f49c06

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

Thanks for reviewing @chemamartinez 😄
We would like to run some smoke-tests with valid Microsoft Sentinel credentials. So just waiting for them and other reviews before merging.

[qcorporation]

@kcreddy do you need to update the kibana constraint to ^8.18 || ^9.0 ?

[qcorporation]

@kcreddy can you please see if it works with both inputs azure-eventhub ?

Good catch @qcorporation, I think agentless support is limited to CEL/httpjson at the moment, but Azure Event Hub is on the roadmap (@nimarezainia).

We'll need to ensure it's clear in docs/UI that agentless support for Sentinel is limited to API and not supported for Event Hub.

@kcreddy we should be testing on a 8.18 install. Within Kibana this input for agentless should not be available. If it is available then we need to file a ticket with Kibana as this should have been disabled by this ticket: elastic/kibana#202091

[kcreddy]

@kcreddy we should be testing on a 8.18 install. Within Kibana this input for agentless should not be available. If it is available then we need to file a ticket with Kibana as this should have been disabled by this ticket: elastic/kibana#202091

@qcorporation, I created: elastic/kibana#211092 for removing Azure Eventhub input from Agentless.

cc: @jamiehynds

[kcreddy]

@qcorporation @jamiehynds ,
I was able to successfully test with API (CEL input) in both Cloud (8.18.0) and Serverless environments.

Successful ingestion in Cloud environment (8.18.0)

Successful ingestion in Serverless environment

[nimarezainia]

@kcreddy thanks for this work.
@qcorporation @kcreddy @smriti0321 what's the level of testing that will be performed on this integration? We have also been modeling the impact on COGs, at the moment portion of the cogs will depend on volume of traffic ingested.

Seems that for this integration the default container sizes are used, which means that there may not be COGS related concern. However do we know that the default (1cpu, 1GiB) container is sufficient for MSFT sentinel?

I think as we start onboarding more integrations to agentless some level of stress testing would be good just to see if the sizing of the infrastructure is suitable. let me know your thoughts.

[nimarezainia]

@kcreddy once the eventhub inout is blocked, what would the behavior be here for this integration? the MSFT sentinel events are to be ingested via Eventhub. if at the top level the integration is nominated to be agentless, would the user be allowed to enable collection of events?

[kcreddy]

Moved the PR to draft once again as we are waiting on elastic/kibana#211092.

[kcreddy]

@kcreddy once the eventhub inout is blocked, what would the behavior be here for this integration? the MSFT sentinel events are to be ingested via Eventhub. if at the top level the integration is nominated to be agentless, would the user be allowed to enable collection of events?

@nimarezainia, the Microsoft Sentinel integration currently supports 2 inputs: API (CEL input) and EventHub. As we block EventHub input, the users can ingest alerts and incidents via API. The other data-stream Microsoft Sentinel Events, i.e., Collecting Events from Microsoft Sentinel via Azure Event Hub will not show up and remain hidden to the users.

[nimarezainia]

@kcreddy once the eventhub inout is blocked, what would the behavior be here for this integration? the MSFT sentinel events are to be ingested via Eventhub. if at the top level the integration is nominated to be agentless, would the user be allowed to enable collection of events?

@nimarezainia, The Microsoft Sentinel integration currently supports 2 inputs: API (CEL input) and EventHub. As we block EventHub, the users can ingest alerts and incidents via API.

My question is around the integrations UI. Once the user has chosen Agentless - I assume the toggle for sentinel events will stay off and user unable to enable it. is this correct?

[kcreddy]

My question is around the integrations UI. Once the user has chosen Agentless - I assume the toggle for sentinel events will stay off and user unable to enable it. is this correct?

@nimarezainia,
Looking at elastic/kibana#206074 how the earlier inputs disablement is handled, when users select Agentless, this toggle for Eventhub, thus Microsoft Sentinel Events is not shown to the users.

[kcreddy]

@qcorporation, I updated the README as per suggestion from docs team.

azure-eventhub is still present in latest 8.18.0 snapshot. However it is now removed in 9.0.0 and also Serverless.

Azure Eventhub present in latest 8.18.0

Azure Eventhub removed in 9.0.0

Azure Eventhub removed in Serverless

It seems that the change removing Eventhub input: elastic/kibana#211262 is merged into 9.0.0 and 9.1.0. Should this be also backported to 8.18.0 or 8.19.0?

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 311b345

History

• 💚 Build #22959 succeeded 12fefe4
• 💔 Build #22958 failed bb589a5
• 💚 Build #22275 succeeded b388c01
• 💚 Build #21900 succeeded 393c46a
• 💚 Build #21762 succeeded f059f98581936ff4e75b49e6cfdbb4f60fd5ef01
• 💚 Build #21746 succeeded 384ba0aa4385e56775d1d0e167c96b046c3acd7d

cc @kcreddy

[kcreddy]

Retested after elastic/kibana#213104 in 8.18.0 and the Eventhub input is no longer visible in Agentless mode.

[elastic-vault-github-plugin-prod]

Package microsoft_sentinel - 0.5.0 containing this change is available at https://epr.elastic.co/package/microsoft_sentinel/0.5.0/