Skip to content

Commit

Permalink
Update TestingConcepts.md
Browse files Browse the repository at this point in the history
  • Loading branch information
stanbrub authored Jan 18, 2025
1 parent c409850 commit 0131e1d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/TestingConcepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Want to know what tables and operations the test uses? Go to the test. Want to k
Tests are developed by test-writers, so why not make it easy for them? Run tests from the IDE for ease of debugging. Point the tests to a local or a remote Deephaven Server instance. Or package tests in a jar and run them locally or remotely from the Benchmark uber-jar. The same tests should work whether running everything on the same system or different system.

### Measure Where It Matters
The Benchmark framework allows the test-writer to set each benchmark measurement from the test code instead of relying on a mechanism that measures automatically behind the scenes. Measurements can be taken across the execution of the test locally with a *Timer* like in the [JoinTablesFromKafkaStreamTest](src/it/java/io/deephaven/benchmark/tests/internal/examples/stream/JoinTablesFromKafkaStreamTest.java) example test or fetched from the remote Deephaven instance where the test is running as is done in the [StandardTestRunner](src/it/java/io/deephaven/benchmark/tests/standard/StandardTestRunner.java) used for nightly Deephaven benchmarks. Either way the submission of the result to the Benchmark framework is under the test-writer's control.
The Benchmark framework allows the test-writer to set each benchmark measurement from the test code instead of relying on a mechanism that measures automatically behind the scenes. Measurements can be taken across the execution of the test locally with a *Timer* like in the [JoinTablesFromKafkaStreamTest](../src/it/java/io/deephaven/benchmark/tests/internal/examples/stream/JoinTablesFromKafkaStreamTest.java) example test or fetched from the remote Deephaven instance where the test is running as is done in the [StandardTestRunner](../src/it/java/io/deephaven/benchmark/tests/standard/StandardTestRunner.java) used for nightly Deephaven benchmarks. Either way the submission of the result to the Benchmark framework is under the test-writer's control.

### Preserve and Compare
Most benchmarking efforts involve a fixed timeline for "improving performance" rather than tracking the performance impacts of code changes day-to-day. This can be effective, but how do you know if future code changes are degrading performance unless benchmarking is done every day. A better way is to preserve benchmarking results every day and compare to the results from the previous week. To avoid death by a thousand cuts, compare release to release as well for the same benchmarks. Also, compare benchmarks for your product with other products for equivalent operations.
Expand Down

0 comments on commit 0131e1d

Please sign in to comment.