Skip to content

Support Min/Max aggregation for partitioned stats (i.e. First/Last message timestamps) #18142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 8, 2025

Conversation

Hor911
Copy link
Collaborator

@Hor911 Hor911 commented May 8, 2025

When we agggregate partitioned statistics (from different parts, i.e. when single shards send data through several read actors) we should support different aggregations functions, i.e.:

  • FirstMessageMs should be aggregated by MIN
  • LastMessageMs should be aggregated by MAX

Before this change, all aggregations were processed with SUM and result incorrect statistics

P.S. Also minor fix to assign HistorySampleCount and restore time series recordings is included

@Hor911 Hor911 requested a review from a team as a code owner May 8, 2025 09:28
Copy link

github-actions bot commented May 8, 2025

🟢 2025-05-08 09:42:29 UTC The validation of the Pull Request description is successful.

Copy link

github-actions bot commented May 8, 2025

2025-05-08 09:32:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for c10605e has started.
2025-05-08 09:32:33 UTC Artifacts will be uploaded here
2025-05-08 09:35:21 UTC ya make is running...
🟡 2025-05-08 10:42:31 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20778 19412 0 6 1323 37

2025-05-08 10:44:16 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-05-08 11:08:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
135 (only retried tests) 100 0 1 2 32

2025-05-08 11:08:18 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-05-08 11:31:56 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
115 (only retried tests) 78 0 0 8 29

🟢 2025-05-08 11:32:03 UTC Build successful.
🟢 2025-05-08 11:32:25 UTC ydbd size 2.2 GiB changed* by +18.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e4d3ee7 merge: c10605e diff diff %
ydbd size 2 346 283 080 Bytes 2 346 302 176 Bytes +18.6 KiB +0.001%
ydbd stripped size 493 372 816 Bytes 493 376 400 Bytes +3.5 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented May 8, 2025

2025-05-08 09:32:33 UTC Pre-commit check linux-x86_64-release-asan for c10605e has started.
2025-05-08 09:32:44 UTC Artifacts will be uploaded here
2025-05-08 09:35:42 UTC ya make is running...
🟡 2025-05-08 11:09:50 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12792 12558 0 155 51 28

2025-05-08 11:10:57 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-05-08 11:45:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1588 (only retried tests) 1443 0 89 31 25

2025-05-08 11:45:43 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-05-08 12:18:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1288 (only retried tests) 1159 0 67 38 24

🟢 2025-05-08 12:19:12 UTC Build successful.
🟢 2025-05-08 12:19:44 UTC ydbd size 3.8 GiB changed* by +34.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e4d3ee7 merge: c10605e diff diff %
ydbd size 4 126 362 640 Bytes 4 126 397 920 Bytes +34.5 KiB +0.001%
ydbd stripped size 1 432 730 464 Bytes 1 432 740 832 Bytes +10.1 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@Hor911 Hor911 changed the title Support Min/Max aggregation for partitioned stats (i.e. First/Last message timstamps) Support Min/Max aggregation for partitioned stats (i.e. First/Last message timestamps) May 8, 2025
@Hor911 Hor911 merged commit b2f257e into ydb-platform:main May 8, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants