[Streams] Replay loghub data with synthtrace #212120

dgieselaar · 2025-02-21T16:31:09Z

Download, parse and replay loghub data with Synthtrace, for use in the Streams project. In summary:

adds a @kbn/sample-log-parser package which parses Loghub sample data, creates valid parsers for extracting and replacing timestamps, using the LLM
add a sample_logs scenario which uses the parsed data sets to replay Loghub data continuously as if it were live data
refactor some parts of Synthtrace (follow-up work captured in [Synthtrace] Consolidate clients #212179)

Synthtrace changes

Replace custom Logger object with Kibana-standard ToolingLog
Report progress and estimated time to completion for long-running jobs
Simplify scenarioOpts (allow comma-separated key-value pairs instead of just JSON)
Simplify client initialization
When using workers, only bootstrap once (in the main thread)
Allow workers to gracefully shutdown
Downgrade some logging levels for less noise

flash1293

Implementation-wise this looks pretty good to me. Some meta questions:

Should we rely on the public loghub repo or fork it off? I'm a little worried this breaking at some point because loghub changes its layout. This would also make it easier to expand it by our own means. In both cases we should cite loghub and the paper somewhere appropriate (like a readme file) as by the license
I'm not so sure about the different speeds. I'm running via node scripts/synthtrace.js sample_logs --live --kibana=http://localhost:5601 --target=http://localhost:9200 --liveBucketSize=1000 and the liveBucketSize is essentially not considered because it computes its own speed. Can we make it taken into account? Different speeds for different data sets are a nice touch as it mirrors reality, but I would like to control the factor of data intake (and speed everything up by a factor of 1k for example). Maybe that's already possible and I just don't know the right command
I spot-checked some aspects of the refactoring and it makes sense to me, but I didn't dig through everything and as I'm not super familiar with the code base it's likely I'm missing something in there

github-actions · 2025-02-24T15:24:38Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

dgieselaar · 2025-02-24T15:38:32Z

Should we rely on the public loghub repo or fork it off? I'm a little worried this breaking at some point because loghub changes its layout. This would also make it easier to expand it by our own means. In both cases we should cite loghub and the paper somewhere appropriate (like a readme file) as by the license

I'm fine with either - but maybe good to do that as a follow-up, I'm not sure what the legal ramifications are.

I'm not so sure about the different speeds. I'm running via node scripts/synthtrace.js sample_logs --live --kibana=http://localhost:5601/ --target=http://localhost:9200/ --liveBucketSize=1000 and the liveBucketSize is essentially not considered because it computes its own speed. Can we make it taken into account? Different speeds for different data sets are a nice touch as it mirrors reality, but I would like to control the factor of data intake (and speed everything up by a factor of 1k for example). Maybe that's already possible and I just don't know the right command

Yes, totally forgot about this setting, I should be able to use it. Would we use a constant indexing rate for each generator, or keep the relative rate per generator (e.g. Android indexes at a way higher rate than Macbook)?

flash1293 · 2025-02-24T17:48:29Z

I'm fine with either - but maybe good to do that as a follow-up, I'm not sure what the legal ramifications are.

Sounds good, then we should add a backlink to the repo and paper and follow up later.

Would we use a constant indexing rate for each generator, or keep the relative rate per generator (e.g. Android indexes at a way higher rate than Macbook)?

I would prefer the latter, in practice this kind of thing happens all the time.

elasticmachine · 2025-02-25T10:47:27Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

dgieselaar · 2025-02-25T10:49:32Z

@flash1293 I've added runOptions.rpm to set a target rpm for all the generators. Unless I'm mistaken or misunderstand what you mean, I don't think liveBucketSize matters - that defines the time range that synthtrace requests data for. interval('5s') defines the granularity of the timestamps. We don't really have a native concept for rpm in synthtrace, each scenario itself defines the volume of requests.

MiriamAparicio

LGTM 🌟
I've been thinking about the clients in synthtrace, makes more sense consolidating them, there was a lot of repeated code. Thanks for this 👏

flash1293 · 2025-02-26T14:38:01Z

Discussed offline:

Requests shouldn't just shift original timestamps, but instead be smoothed out over the target time frame (because otherwise some data sets log very little docs)
The scenario should not disable streams as part of the setup routine as this drops existing configuration

pheyos

src/platform/packages/private/kbn-journeys/services/synthtrace.ts changes LGTM

achyutjhunjhunwala · 2025-03-04T11:19:52Z

...atform/packages/shared/kbn-apm-synthtrace/src/lib/apm/client/apm_synthtrace_kibana_client.ts

-      if (responseJson.response && responseJson.response.latestVersion) {
-        return responseJson.response.latestVersion as string;
+      if (!response.item.latestVersion) {
+        throw new Error(`Failed to fetch APM package version`);


Are we not supporting Synthtrace for 7.x ?

I totally accidentally removed it, but yeah, let's forget about 7.x, or do you have concerns?

…er # Please enter a commit message to explain why this merge is necessary,

elasticmachine · 2025-03-06T09:40:59Z

💚 Build Succeeded

Buildkite Build
Commit: ca6805e
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-212120-ca6805e7786b

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/apm-synthtrace`	97	98	+1
`@kbn/apm-synthtrace-client`	272	274	+2
`@kbn/sample-log-parser`	-	18	+18
total			+21

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/sample-log-parser`	-	1	+1

Unknown metric groups

API count

id	before	after	diff
`@kbn/apm-synthtrace`	97	98	+1
`@kbn/apm-synthtrace-client`	272	274	+2
`@kbn/sample-log-parser`	-	18	+18
total			+21

ESLint disabled in files

id	before	after	diff
`@kbn/apm-synthtrace`	2	3	+1

ESLint disabled line counts

id	before	after	diff
`@kbn/apm-synthtrace`	6	2	-4

Total ESLint disabled count

id	before	after	diff
`@kbn/apm-synthtrace`	8	5	-3

History

💔 Build #281170 failed 8d977ef
💔 Build #279401 failed 38a6cd6
💔 Build #279048 failed 74681f2
💚 Build #278662 succeeded f02eafb
💔 Build #278653 failed 40ca31b

cc @dgieselaar

dgieselaar added 2 commits February 21, 2025 17:30

[Streams] Replay loghub data with synthtrace

57bbd8e

Add tests & ensure repo for synthtrace

fbf8063

dgieselaar mentioned this pull request Feb 22, 2025

[Synthtrace] Consolidate clients #212179

Open

dgieselaar and others added 5 commits February 22, 2025 14:18

Fix type issues

b17ae1c

Merge branch 'main' of github.com:elastic/kibana into sample-log-parser

57909f0

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

be38355

Fix type issues

40ca31b

Use kibanaClient.fetch for entities client

f02eafb

dgieselaar added v9.0.0 v9.1.0 v8.19.0 Feature:Streams This is the label for the Streams Project labels Feb 23, 2025

dgieselaar self-assigned this Feb 23, 2025

dgieselaar marked this pull request as ready for review February 23, 2025 14:22

dgieselaar requested review from a team as code owners February 23, 2025 14:22

dgieselaar added backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes labels Feb 23, 2025

flash1293 reviewed Feb 24, 2025

View reviewed changes

botelastic bot added the ci:project-deploy-observability Create an Observability project label Feb 24, 2025

dgieselaar added 2 commits February 24, 2025 19:20

Merge branch 'main' of github.com:elastic/kibana into sample-log-parser

0d6ade8

Distribute RPM amongst scenarios

6d8253a

botelastic bot added the Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team label Feb 25, 2025

Merge branch 'main' of github.com:elastic/kibana into sample-log-parser

74681f2

Ikuni17 approved these changes Feb 25, 2025

View reviewed changes

dgieselaar added 2 commits February 26, 2025 12:10

Merge branch 'main' of github.com:elastic/kibana into sample-log-parser

e58b73e

Fix type issues

38a6cd6

MiriamAparicio approved these changes Feb 26, 2025

View reviewed changes

viduni94 approved these changes Feb 26, 2025

View reviewed changes

pheyos approved these changes Feb 26, 2025

View reviewed changes

achyutjhunjhunwala reviewed Mar 4, 2025

View reviewed changes

dgieselaar added 6 commits March 5, 2025 17:10

Query generation, graceful worker shutdown, progress reporters

2e69704

Removed comment for 7.x

7af77c5

Don't clean sample_logs

133a749

Merge branch 'main' of github.com:elastic/kibana into sample-log-pars…

269425c

…er # Please enter a commit message to explain why this merge is necessary,

Enable stream manager before bootstrap

8d977ef

Make logging optional

ca6805e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Streams] Replay loghub data with synthtrace #212120

[Streams] Replay loghub data with synthtrace #212120

dgieselaar commented Feb 21, 2025 •

edited

Loading

flash1293 left a comment •

edited

Loading

github-actions bot commented Feb 24, 2025

dgieselaar commented Feb 24, 2025

flash1293 commented Feb 24, 2025

elasticmachine commented Feb 25, 2025

dgieselaar commented Feb 25, 2025

MiriamAparicio left a comment

flash1293 commented Feb 26, 2025

pheyos left a comment

achyutjhunjhunwala Mar 4, 2025

dgieselaar Mar 5, 2025

elasticmachine commented Mar 6, 2025 •

edited

Loading

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

[Streams] Replay loghub data with synthtrace #212120

Are you sure you want to change the base?

[Streams] Replay loghub data with synthtrace #212120

Conversation

dgieselaar commented Feb 21, 2025 • edited Loading

Synthtrace changes

flash1293 left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Feb 24, 2025

🤖 GitHub comments

dgieselaar commented Feb 24, 2025

flash1293 commented Feb 24, 2025

elasticmachine commented Feb 25, 2025

dgieselaar commented Feb 25, 2025

MiriamAparicio left a comment

Choose a reason for hiding this comment

flash1293 commented Feb 26, 2025

pheyos left a comment

Choose a reason for hiding this comment

achyutjhunjhunwala Mar 4, 2025

Choose a reason for hiding this comment

dgieselaar Mar 5, 2025

Choose a reason for hiding this comment

elasticmachine commented Mar 6, 2025 • edited Loading

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

dgieselaar commented Feb 21, 2025 •

edited

Loading

flash1293 left a comment •

edited

Loading

elasticmachine commented Mar 6, 2025 •

edited

Loading