Add additional_parameters to diagnostics action

[michel-laterman]

What is the problem this PR solves?

Allow fleet-server to specify if a diagnostics bundle should contain additional CPU metrics.

How does this PR solve the problem?

Add an optional parameter to the RequestDiagnostics action. This parameter is a list of strings (enum) that has a single value "CPU". If this parameter is specified, the agent should collect a CPU trace.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates Allow collecting CPU diagnostics in the Fleet diagnostics action handler elastic-agent#3491

[juliaElastic]

LGTM
Is there an elastic-agent change needed to make use of the CPU option?

[michel-laterman]

The agent change is elastic/elastic-agent#4394

[michel-laterman]

I'm going to hold off on merging this until I can get an e2e test working

[michel-laterman]

I managed to do an e2e test.

I started 8.14.0-SNAPSHOT builds locally then enrolled a new fleet-server running under the elastic agent with changes from elastic/elastic-agent#4394

First I validated that the request diagnostics button on the UI works as intended (changes in the eefb64c commit were needed for that).

Then I created a user with permissions to index documents in .fleet-actions and created a new document with:

POST .fleet-actions-7/_doc
{
  "@timestamp": "2024-03-11T16:36:08.791Z",
  "expiration": "2024-03-11T16:39:08.791Z",
  "agents": [
    "717e2602-3901-4ca2-b310-7d488a7631b0"
  ],
  "action_id": "cpu-tester",
  "type": "REQUEST_DIAGNOSTICS",
  "data": {
    "additional_metrics": [
      "CPU"
    ]
  }
}

The resulting diagnostics bundle was available through the UI, and it contains cpu.pprof files

 elastic-agent-diagnostics-2024-03-11T16-36-29Z-00|⇒ ls -la
total 2120
drwxr-xr-x@ 21 laterman  staff     672 11 Mar 17:38 .
drwx------@ 47 laterman  staff    1504 11 Mar 17:37 ..
-rw-r--r--@  1 laterman  staff    6148 11 Mar 17:38 .DS_Store
-rwxr-xr-x@  1 laterman  staff   32297 11 Mar 17:36 allocs.pprof.gz
-rwxr-xr-x@  1 laterman  staff     591 11 Mar 17:36 block.pprof.gz
drwxr-xr-x@  8 laterman  staff     256 11 Mar 17:37 components
-rwxr-xr-x@  1 laterman  staff  455938 11 Mar 17:36 components-actual.yaml
-rwxr-xr-x@  1 laterman  staff  455938 11 Mar 17:36 components-expected.yaml
-rwxr-xr-x@  1 laterman  staff   16510 11 Mar 17:36 computed-config.yaml
-rwxr-xr-x@  1 laterman  staff    2718 11 Mar 17:36 cpu.pprof
-rwxr-xr-x@  1 laterman  staff    7199 11 Mar 17:36 goroutine.pprof.gz
-rwxr-xr-x@  1 laterman  staff   32224 11 Mar 17:36 heap.pprof.gz
-rwxr-xr-x@  1 laterman  staff    6029 11 Mar 17:36 local-config.yaml
drwxr-xr-x@  3 laterman  staff      96 11 Mar 17:37 logs
-rwxr-xr-x@  1 laterman  staff     592 11 Mar 17:36 mutex.pprof.gz
-rwxr-xr-x@  1 laterman  staff       7 11 Mar 17:36 package.version
-rwxr-xr-x@  1 laterman  staff   17677 11 Mar 17:36 pre-config.yaml
-rwxr-xr-x@  1 laterman  staff    5872 11 Mar 17:36 state.yaml
-rwxr-xr-x@  1 laterman  staff     748 11 Mar 17:36 threadcreate.pprof.gz
-rwxr-xr-x@  1 laterman  staff    1226 11 Mar 17:36 variables.yaml
-rwxr-xr-x@  1 laterman  staff     113 11 Mar 17:36 version.txt

elastic-agent-diagnostics-2024-03-11T16-36-29Z-00|⇒ ls -la components/fleet-server-default
total 144
drwxr-xr-x@ 12 laterman  staff    384 11 Mar 17:37 .
drwxr-xr-x@  8 laterman  staff    256 11 Mar 17:37 ..
-rwxr-xr-x@  1 laterman  staff  19046 11 Mar 17:37 allocs.pprof.gz
-rwxr-xr-x@  1 laterman  staff    395 11 Mar 17:37 block.pprof.gz
-rwxr-xr-x@  1 laterman  staff   3139 11 Mar 17:37 cpu.pprof
drwxr-xr-x@  2 laterman  staff     64 11 Mar 17:37 fleet-server-default
drwxr-xr-x@  2 laterman  staff     64 11 Mar 17:37 fleet-server-fleet_server-d314b1f8-223b-4212-8bd4-1b6183179d20
-rwxr-xr-x@  1 laterman  staff   7855 11 Mar 17:37 fleet-server.yml
-rwxr-xr-x@  1 laterman  staff   7513 11 Mar 17:37 goroutine.pprof.gz
-rwxr-xr-x@  1 laterman  staff  18964 11 Mar 17:37 heap.pprof.gz
-rwxr-xr-x@  1 laterman  staff    395 11 Mar 17:37 mutex.pprof.gz
-rwxr-xr-x@  1 laterman  staff    590 11 Mar 17:37 threadcreate.pprof.gz

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
90.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[cmacknz]

Just to double check, this is backwards compatible with older agents that don't understand this yet? They'll just ignore it?

[michel-laterman]

Yes.
I ran another e2e test by enrolling an agent without the changes in elastic/elastic-agent#4394 to a fleet-server instance build with the changes in this PR and manually inserting an action with CPU in additional_metrics.
A diagnostics bundle was generated with no cpu.pprof as a part of it

[cmacknz]

Thanks!