Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[O11y][MySQL] Rally benchmark mysql.slowlog #9024

Merged
merged 3 commits into from
Feb 15, 2024

Conversation

ali786XI
Copy link
Contributor

Proposed commit message

  • This PR adds benchmarking templates to the slowlog data stream of MySQL

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

  • elastic-package benchmark rally --benchmark slowlog-benchmark -v
  • elastic-package benchmark stream --benchmark slowlog-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: mysql - START ---
╭────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                               │
├────────────────────────┬───────────────────────────────────────────────────────────┤
│ benchmark              │                                         slowlog-benchmark │
│ description            │             Benchmark 20000 mysql.slowlog events ingested │
│ run ID                 │                      025ba584-2ff5-4e43-8ff7-14f9126c2b07 │
│ package                │                                                     mysql │
│ start ts (s)           │                                                1706695746 │
│ end ts (s)             │                                                1706695785 │
│ duration               │                                                       39s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-1555256379 │
╰────────────────────────┴───────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────╮
│ parameters                                                            │
├─────────────────────────────────┬─────────────────────────────────────┤
│ package version                 │                              1.17.0 │
│ data_stream.name                │                             slowlog │
│ corpora.generator.total_events  │                               20000 │
│ corpora.generator.template.path │ ./slowlog-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                     │
│ corpora.generator.template.type │                              gotext │
│ corpora.generator.config.path   │      ./slowlog-benchmark/config.yml │
│ corpora.generator.config.raw    │                               map[] │
│ corpora.generator.fields.path   │      ./slowlog-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                               map[] │
╰─────────────────────────────────┴─────────────────────────────────────╯
╭───────────────────────╮
│ cluster info          │
├───────┬───────────────┤
│ name  │ elasticsearch │
│ nodes │             1 │
╰───────┴───────────────╯
╭────────────────────────────────────────────────────╮
│ data stream stats                                  │
├────────────────────────────┬───────────────────────┤
│ data stream                │ logs-mysql.slowlog-ep │
│ approx total docs ingested │                 20000 │
│ backing indices            │                     1 │
│ store size bytes           │               9797811 │
│ maximum ts (ms)            │         1706695741000 │
╰────────────────────────────┴───────────────────────╯
╭───────────────────────────────────────╮
│ disk usage for index .ds-logs-mysql.s │
│ lowlog-ep-2024.01.30-000001 (for all  │
│ fields)                               │
├──────────────────────────────┬────────┤
│ total                        │ 9.6 MB │
│ inverted_index.total         │ 1.3 MB │
│ inverted_index.stored_fields │ 4.3 MB │
│ inverted_index.doc_values    │ 2.1 MB │
│ inverted_index.points        │ 1.9 MB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭────────────────────────────────────────────────────────────────────────────────────╮
│ pipeline logs-mysql.slowlog-1.17.0 stats in node 7isCMBINTqSiSa82qwK9Ag            │
├──────────────────────────────────────────┬─────────────────────────────────────────┤
│ Totals                                   │ Count: 20000 | Failed: 0 | Time: 2.119s │
│ drop ()                                  │         Count: 0 | Failed: 0 | Time: 0s │
│ set ()                                   │   Count: 20000 | Failed: 0 | Time: 14ms │
│ set ()                                   │   Count: 20000 | Failed: 0 | Time: 87ms │
│ grok ()                                  │ Count: 20000 | Failed: 0 | Time: 1.684s │
│ rename ()                                │   Count: 20000 | Failed: 0 | Time: 28ms │
│ script ()                                │    Count: 20000 | Failed: 0 | Time: 8ms │
│ script ()                                │    Count: 20000 | Failed: 0 | Time: 7ms │
│ script ()                                │   Count: 20000 | Failed: 0 | Time: 37ms │
│ script ()                                │    Count: 20000 | Failed: 0 | Time: 6ms │
│ remove ()                                │   Count: 20000 | Failed: 0 | Time: 17ms │
│ date ()                                  │   Count: 20000 | Failed: 0 | Time: 48ms │
│ remove ()                                │   Count: 20000 | Failed: 0 | Time: 20ms │
│ set ()                                   │    Count: 20000 | Failed: 0 | Time: 9ms │
│ append ()                                │   Count: 20000 | Failed: 0 | Time: 10ms │
│ append ()                                │    Count: 20000 | Failed: 0 | Time: 8ms │
│ remove ()                                │   Count: 20000 | Failed: 0 | Time: 13ms │
│ pipeline (global@custom)                 │    Count: 20000 | Failed: 0 | Time: 3ms │
│ pipeline (logs@custom)                   │    Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (logs-mysql.integration@custom) │    Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (logs-mysql.slowlog@custom)     │    Count: 20000 | Failed: 0 | Time: 2ms │
╰──────────────────────────────────────────┴─────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │      63.17803333333333 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │  0.0024416666666666666 min │
│ Max cumulative indexing time across primary shards             │                18.8743 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │                 2.1645 min │
│ Cumulative merge count of primary shards                       │                       1473 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                    0.0 min │
│ Max cumulative merge time across primary shards                │    0.27523333333333333 min │
│ Cumulative merge throttle time of primary shards               │                      0 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                      0 min │
│ Cumulative refresh time of primary shards                      │     2.4740166666666665 min │
│ Cumulative refresh count of primary shards                     │                      50735 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │   0.004833333333333333 min │
│ Max cumulative refresh time across primary shards              │    0.47914999999999996 min │
│ Cumulative flush time of primary shards                        │     132.20303333333334 min │
│ Cumulative flush count of primary shards                       │                      45336 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │   0.012258333333333335 min │
│ Max cumulative flush time across primary shards                │      22.56978333333333 min │
│ Total Young Gen GC time                                        │                    0.048 s │
│ Total Young Gen GC count                                       │                          5 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │      0.1612674044445157 GB │
│ Translog size                                                  │     0.04449701029807329 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        855 │
│ Total Ingest Pipeline count                                    │                      20027 │
│ Total Ingest Pipeline time                                     │                    3.408 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │             2634.09 docs/s │
│ Mean Throughput                                                │             2692.55 docs/s │
│ Median Throughput                                              │             2689.76 docs/s │
│ Max Throughput                                                 │             2736.26 docs/s │
│ 50th percentile latency                                        │      167.69859398482367 ms │
│ 90th percentile latency                                        │        198.707362083951 ms │
│ 100th percentile latency                                       │      219.81315198354423 ms │
│ 50th percentile service time                                   │      167.69859398482367 ms │
│ 90th percentile service time                                   │        198.707362083951 ms │
│ 100th percentile service time                                  │      219.81315198354423 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: mysql - END   ---
Done

@ali786XI ali786XI self-assigned this Jan 31, 2024
@ali786XI ali786XI added enhancement New feature or request Integration:mysql MySQL labels Jan 31, 2024
@ali786XI ali786XI changed the title [O11y][MySQL] Rally benchmark mysql.slowlog [O11y][MySQL] Rally benchmark mysql.slowlog Jan 31, 2024
@elasticmachine
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@ali786XI ali786XI marked this pull request as ready for review January 31, 2024 11:10
@ali786XI ali786XI requested a review from a team as a code owner January 31, 2024 11:10
fields:
- name: timestamp
period: -24h # one day
- name: start_time
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean start time could be after end time? also, should timestamp and start_time be the same?

it would be good if we could generate durations, so we could have the start time, then say we want end times between 100-1000 ms later, for example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tommyers-elastic for this catch. @aspacca also thanks for your suggestion link.
If we are giving multiple date fields the timestamps would be same which would not make sense. Hence changed the start_time to before 5m and end_time to before 2m the timestamp generated. Let me know if this makes sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perfect, thanks! if these aren't documented we should add something.

only one more comment (not a blocker), could we make these have some variation?

{{- $startTime := generate "timestamp" | date_modify "-5m" }}
{{- $endTime := generate "timestamp" | date_modify "-2m" }}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tommyers-elastic Agree. Generated some range of enum values which the start and end time can take.
The date_modify is documented here as a part of this documentation. Is that what you were referring to?

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @aliabbas-elastic

Copy link

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@ali786XI
Copy link
Contributor Author

@tommyers-elastic if all good can we go ahead merging this PR. Let me know if there are any comments. Thanks !!

Copy link
Member

@shmsr shmsr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the other comments I see in the PR; approving this.

@ali786XI ali786XI merged commit 577e9e5 into elastic:main Feb 15, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:mysql MySQL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants