[Fleet] Allow collection CPU profiles when collecting agent diagnostics

[cmacknz]

elastic/elastic-agent#2140 restored the ability of the elastic-agent diagnostics command to collect CPU profiles, which is critically important when diagnosing CPU performance problems.

CPU profile collection is optional and is done when the --cpu-profiles option is provided like elastic-agent diagnostics --cpu-profile.

Add a check box or toggle allowing users to opt into collecting CPU profiles when collecting diagnostics through Fleet. The diagnostics action will need to be modified to include this new boolean switch.

Also note that each CPU profile takes approximately 30s to capture, so it is likely that the timeout for the diagnostics action will need to be increased when this option is selected.

Depends on elastic/elastic-agent#3491

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jlind23]

@zombieFox could you please help us out here with the design?
cc @nimarezainia

[jlind23]

@nimarezainia @zombieFox I don't see any update here, we are we at? Are we just going down the path of adding a simple checkbox in the UI below?

[nimarezainia]

We should use the ability to generate UI from config to do this. I'll add it to the list.

(fyi @amitkanfer @kpollich )

[amitkanfer]

where's the list ? :)

[jlind23]

Discussed today with @zombieFox and @nimarezainia, we all agreed to start with a checkbox to enable CPU profile collection. Hence moving this to a ready state.

[jlind23]

@kpollich @juliaElastic for the request diagnostic, can we send the --cpu-profiles options as part of the action data here ? Or do we need to change the method signature to supply external options?

[juliaElastic]

I think we have to extend the REQUEST_DIAGNOSTICS action with a parameter to pass if cpu-profiles should be included. This requires a change in kibana and fleet-server.

[jlind23]

So this should be available only if fleet server is in the right version though.

[juliaElastic]

Checked this further and I think it also needs changes on elastic-agent side, to add a cpuProfile flag to the request diagnostics action and apply the flag when collecting diagnostics in the handle function.
@cmacknz I think the agent changes are a prerequisite of the kibana and fleet-server changes.

[cmacknz]

Correct the change needs to be made across Fleet, Fleet Server, and Elastic Agent. Prior to making the Elastic Agent change the new parameter in the diagnostics action needs to be defined.

You can't test any of this without implementing the whole system, the easiest option is to have whoever makes the Fleet Server (and possibly Fleet) change also change the agent. This is just passing in a function argument that already exists in the right place. We could also have multiple people do this, that's just more coordination overhead.

I wrote up how to do this in detail in elastic/elastic-agent#3491.

If the Fleet changes come first and are merged into a snapshot image we can write an E2E test for it in the agent repository. This could be done after manual verification end to end though.

[jlind23]

@cmacknz changed the area for: elastic/elastic-agent#3491 As I believe one of the fleet engineer can do both Elastic Agent and Fleet Server change at the same time.

[nimarezainia]

moving this to sprint 27 as the item blocking it is in sprint 26

[michel-laterman]

Changes to fleet-server are here: elastic/fleet-server#3333
the openapi spec has been updated so diagnostics actions can have data.additional_metrics: []string.
In order to gather CPU profiles "CPU" needs to be included in the list.

[kpollich]

We should add a confirmation modal to the request diagnostics flow that allows the user to check a checkbox for Collect additional CPU metrics. This should follow the conventions elsewhere where we create Agent actions.