[APM] Service map new API #212550

crespocarlos · 2025-02-26T16:33:27Z

Summary

This PR replaces the scripted_metric aggregation used to retrieve the data for the service map.

The new solution relies on samples of exit spans - each representing a unique combination of service.name and span.destination.service.resource - along with their child transactions. The Service Map is now built entirely on the client side to reduce server-side load and prevent excessive event loop utilization.

Note

transform_service_map_responses.ts was refactored to improve readability and performance, The file was renamed to get_service_map_nodes.ts
group_resource_nodes.ts was refactored to improve readability and performance

Consequences

The new solution requires all exit spans to have the span.destination.service.resource field populated — with the exception of messaging systems, which may rely on span.links (not addressed in this PR)
- A warning will be added to the trace waterfall for exit spans without span.destination.sevice.resource #212638

When multiple services point to load balancers, they will share the same span.destination.service.resource. This could lead to incomplete paths in the map, as the path is built for the first service.name + span.destination.service.resource pair returned processed.
- This can't be addressed, but we'll look into ways to inform the user when the logic identifies this scenario #213124

current	new

Analysis

The performance analysis below uses data from the edge cluster and the service_map_oom synthtrace scenario, simulating long traces. The selected date range was 24h.

Current solution

numeric_labels.event_loop_active: 4085.601743
numeric_labels.event_loop_utilization: 0.28716

New solution

numeric_labels.event_loop_active: 887.149512
numeric_labels.event_loop_utilization: 0.123929

On the client side, the most CPU-intensive operation is performed by cytoscape. The creation of service connections performs efficiently.

How to test

Add xpack.apm.serviceMapV2Enabled: true to kibana.dev.yml
Navigate to APM > Services Inventory > Service Map

simplified DSL for service map change aggregation Improvements

crespocarlos · 2025-03-04T15:27:06Z

src/platform/packages/shared/kbn-apm-synthtrace-client/src/lib/dsl/service_map.ts

-      return [childSpan.destination(next.serviceInstance.fields['service.name']!)];
+      return [
+        childSpan.destination(next.serviceInstance.fields['service.name']!).children(
+          next.serviceInstance


Exit spans are followed by a transaction. This makes the synthtrace scenario adhere to the real data

…apping'

Bluefinger · 2025-03-05T12:09:03Z

x-pack/solutions/observability/plugins/apm/common/service_map/get_service_map_nodes.ts

+    if (reversedConnection) {
+      reversedConnection.bidirectional = true;
+      return prev.concat({
+        ...connection,


Does this have to be a new object with different property set? Just trying to understand some of the perf impact of some of these operations

Well, considering that reverseConnection is mutated, I would say that we don't really need to create a new object here either. I doubt that there will be any noticeable performance improvement though. I'll change it anyways.

elasticmachine · 2025-03-05T12:09:08Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

github-actions · 2025-03-05T12:09:16Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2025-03-05T17:09:26Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f4893d5
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-212550-f4893d5ec3b8

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`apm`	1890	1894	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	2.5MB	2.5MB	+6.4KB

History

💚 Build #280974 succeeded ae73512
💔 Build #280914 failed 8afc9a8
💔 Build #280846 failed 4bc9dd8
💔 Build #280807 failed b58aaaa
💔 Build #280237 failed 2f2be9a

Bluefinger · 2025-03-05T17:18:02Z

x-pack/solutions/observability/plugins/apm/common/service_map/get_service_map_nodes.ts

+
+  // List of nodes that are services
+  const serviceNodes = new Map(
+    [...Array.from(allNodes.values()), ...connectionFromDiscoveredServices]


.values() returns an iterator, so with spreading into an array, Array.from is redundant and adds an extra allocation that can be avoided.

Suggested change

[...Array.from(allNodes.values()), ...connectionFromDiscoveredServices]

[...allNodes.values(), ...connectionFromDiscoveredServices]

Bluefinger · 2025-03-05T17:21:41Z

x-pack/solutions/observability/plugins/apm/common/service_map/get_service_map_nodes.ts

+
+  const uniqueNodes = mappedEdges
+    .flatMap((connection) => [connection.sourceData, connection.targetData])
+    .concat(Array.from(allServices.values()))


Suggested change

.concat(Array.from(allServices.values()))

.concat(...allServices.values())

We can just spread instead of doing an additional array allocation

crespocarlos force-pushed the 212252-refactor-service-map branch 9 times, most recently from a7df664 to 5fa4ec0 Compare February 28, 2025 17:09

crespocarlos changed the title ~~212252 refactor service map~~ [APM] Service map new API Mar 3, 2025

crespocarlos force-pushed the 212252-refactor-service-map branch 3 times, most recently from 2f2be9a to b648b47 Compare March 3, 2025 11:06

crespocarlos added 6 commits March 4, 2025 16:06

refactor service map

f88c011

simplified DSL for service map change aggregation Improvements

fix load balancer scenario

05f947d

improve readability

9b49fc2

add feature flag for new service map

bedf604

change to enable the new api on the kibana config

e387cc2

Add tests

e357f9e

crespocarlos force-pushed the 212252-refactor-service-map branch 2 times, most recently from 87ef10e to b58aaaa Compare March 4, 2025 15:25

crespocarlos commented Mar 4, 2025

View reviewed changes

crespocarlos mentioned this pull request Mar 4, 2025

[APM] Create a feature flag for the new Service Map API #213125

Open

crespocarlos force-pushed the 212252-refactor-service-map branch from 524f34f to ffe63c9 Compare March 4, 2025 16:55

Return emtpy map when trace.ids are not found

8afc9a8

crespocarlos force-pushed the 212252-refactor-service-map branch from 4bc9dd8 to 8afc9a8 Compare March 4, 2025 22:07

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

ae73512

…apping'

crespocarlos marked this pull request as ready for review March 5, 2025 11:59

crespocarlos requested review from a team as code owners March 5, 2025 11:59

crespocarlos requested a review from a team as a code owner March 5, 2025 11:59

Bluefinger reviewed Mar 5, 2025

View reviewed changes

botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Mar 5, 2025

crespocarlos added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels Mar 5, 2025

CR adjustment

f4893d5

Bluefinger reviewed Mar 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Service map new API #212550

[APM] Service map new API #212550

crespocarlos commented Feb 26, 2025 •

edited

Loading

crespocarlos Mar 4, 2025

Bluefinger Mar 5, 2025

crespocarlos Mar 5, 2025 •

edited

Loading

elasticmachine commented Mar 5, 2025

github-actions bot commented Mar 5, 2025

elasticmachine commented Mar 5, 2025 •

edited

Loading

Bluefinger Mar 5, 2025

Bluefinger Mar 5, 2025

	[...Array.from(allNodes.values()), ...connectionFromDiscoveredServices]
	[...allNodes.values(), ...connectionFromDiscoveredServices]

	.concat(Array.from(allServices.values()))
	.concat(...allServices.values())

[APM] Service map new API #212550

Are you sure you want to change the base?

[APM] Service map new API #212550

Conversation

crespocarlos commented Feb 26, 2025 • edited Loading

Summary

Consequences

Analysis

Current solution

New solution

How to test

crespocarlos Mar 4, 2025

Choose a reason for hiding this comment

Bluefinger Mar 5, 2025

Choose a reason for hiding this comment

crespocarlos Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Mar 5, 2025

github-actions bot commented Mar 5, 2025

🤖 GitHub comments

elasticmachine commented Mar 5, 2025 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Async chunks

History

Bluefinger Mar 5, 2025

Choose a reason for hiding this comment

Bluefinger Mar 5, 2025

Choose a reason for hiding this comment

crespocarlos commented Feb 26, 2025 •

edited

Loading

crespocarlos Mar 5, 2025 •

edited

Loading

elasticmachine commented Mar 5, 2025 •

edited

Loading