i#7216: noise generator basic structure #7283

edeiana · 2025-02-18T11:24:12Z

Scaffolding for the noise generator.
Adds noise_generator_t as a subclass of the iterator reader_t.
Currently noise_generator_t only produces TRACE_TYPE_READ records
with address 0xdeadbeef (easier to spot in tools' output) wrapped in
TRACE_TYPE_THREAD, TRACE_TYPE_PID, and
TRACE_TYPE_THREAD_EXIT.
The noise generator is bundled with the scheduler and is enabled with
-enable_noise_generator (boolean flag). To actually generate synthetic
trace records the user needs to pick how many records to generate with
-noise_generator_num_records (uint64_t flag).

Adds a unit test in scheduler_unit_tests.cpp and an end-to-end test leveraging
the schedule_stats tool.

Issue: #7216

Added -noise_generator_num_records to scheduler options info.

edeiana · 2025-02-21T11:37:43Z

clients/drcachesim/common/options.cpp

+                              "with synthetic records.");
+
+droption_t<uint64_t> op_noise_generator_num_records(
+    DROPTION_SCOPE_FRONTEND, "noise_generator_num_records", 0,


The default is 0, but perhaps it should be the number of records of the longest thread of the trace we're adding noise to. Not sure if there is a way to do so at the moment.

To your point about the longest thread: I would think most use cases want an infinite stream that doesn't end until all the other inputs have ended. Not sure how to do that...want a callback from the scheduler when it starts thinking about exiting? If we just make this the longest thread instruction count, that thread might have a bunch of wait syscalls and finish along with all the other regular threads and then the scheduler could sit there just running noise (modulo the exit early flag). Need a new scheduler feature to treat as infinite and not count toward exit calculations and just cut off when real inputs indicate exit?

edeiana · 2025-02-21T11:37:49Z

clients/drcachesim/scheduler/scheduler_impl.cpp

+    // Add noise generator reader to workload_inputs.
+    if (options_.enable_noise_generator) {
+        auto noise_generator = get_noise_generator(options_.noise_generator_num_records);
+        auto noise_generator_end = get_noise_generator(0);


Perhaps we should have a separate get_noise_generator_end()?

edeiana · 2025-02-21T17:16:09Z

clients/drcachesim/tests/schedule_stats_noise_generator.templatex

+Schedule stats tool results:
+Total counts:
+           4 cores
+           2 threads: W.*, W.*


Not really sure why only ci-windows / vs2019-64 does not match this output. The regex seems correct to me. Will investigate.

derekbruening · 2025-02-21T19:32:56Z

api/docs/release.dox

@@ -157,6 +157,9 @@ Further non-compatibility-affecting changes include:
 - Allow v2p.textproto file in a trace directory. This file is present in public traces.
 - Allow v2p.textproto file to have one missing virtual_address field, which indicates
   virtual_address == 0x0. Necessary in case a trace accesses virtual address 0x0.
+ - Added noise_generator_t scaffolding as a reader_t to produce synthetic trace records.


This is the 11.3 list: the next version list is up above.

derekbruening · 2025-02-21T19:34:05Z

api/docs/release.dox

@@ -157,6 +157,9 @@ Further non-compatibility-affecting changes include:
 - Allow v2p.textproto file in a trace directory. This file is present in public traces.
 - Allow v2p.textproto file to have one missing virtual_address field, which indicates
   virtual_address == 0x0. Necessary in case a trace accesses virtual address 0x0.
+ - Added noise_generator_t scaffolding as a reader_t to produce synthetic trace records.


Good to add "in the drmemtrace framework" to give context: remember that this list covers the entire DR release which includes various other tracing sample tools (and calls its internal superblocks "traces").
Ditto below.

derekbruening · 2025-02-21T19:36:02Z

clients/drcachesim/common/options.cpp

@@ -714,6 +714,18 @@ droption_t<std::string>
                   "Cache hierarchy configuration file",
                   "The full path to the cache hierarchy configuration file.");

+droption_t<bool>
+    op_enable_noise_generator(DROPTION_SCOPE_FRONTEND, "enable_noise_generator", false,


But I might want multiple noise streams. These end up looking like software threads and the scheduler will interleave them with real traced threads, right? So to emulate a multi-tenant scenario we might want to add 100 different noise streams, 10 each for 10 different emulated processes and so each with different characteristics? Just trying to plan ahead.

derekbruening · 2025-02-21T19:36:33Z

clients/drcachesim/common/options.cpp

+                              "with synthetic records.");
+
+droption_t<uint64_t> op_noise_generator_num_records(
+    DROPTION_SCOPE_FRONTEND, "noise_generator_num_records", 0,


To your point about the longest thread: I would think most use cases want an infinite stream that doesn't end until all the other inputs have ended. Not sure how to do that...want a callback from the scheduler when it starts thinking about exiting? If we just make this the longest thread instruction count, that thread might have a bunch of wait syscalls and finish along with all the other regular threads and then the scheduler could sit there just running noise (modulo the exit early flag). Need a new scheduler feature to treat as infinite and not count toward exit calculations and just cut off when real inputs indicate exit?

derekbruening · 2025-02-21T19:40:41Z

clients/drcachesim/scheduler/noise_generator.cpp

+        return &entry_;
+    }
+
+    // XXX i#7216: this is a temporary trace record that we use as a placeholder until the


Should be TODO (XXX is too soft: XXX means "nice to have" while this seems required).

derekbruening · 2025-02-21T19:43:00Z

clients/drcachesim/scheduler/scheduler_impl.cpp

+    // Add noise generator reader to workload_inputs.
+    if (options_.enable_noise_generator) {
+        auto noise_generator = get_noise_generator(options_.noise_generator_num_records);
+        auto noise_generator_end = get_noise_generator(0);


derekbruening · 2025-02-21T19:43:40Z

clients/drcachesim/tests/scheduler_unit_tests.cpp

+            return nullptr;
+        }
+
+        // Do not change the order for generating TRACE_TYPE_THREAD and TRACE_TYPE_PID.


Could we put this in its own function to share w/ the parent and avoid duplicating here?

derekbruening · 2025-02-21T19:46:53Z

clients/drcachesim/scheduler/noise_generator.cpp

+    if (!marker_tid_generated_) {
+        entry_ = { TRACE_TYPE_THREAD,
+                   sizeof(int),
+                   { static_cast<addr_t>(IDLE_THREAD_ID) } };


But the idle thread means something special when computing context switch stats: I think this has to have its own unique TID (unique for each instance too). This should not be treated as idle: it should pretend to be a real thread.

derekbruening · 2025-02-21T19:47:58Z

clients/drcachesim/scheduler/noise_generator.cpp

+    if (!marker_pid_generated_) {
+        entry_ = { TRACE_TYPE_PID,
+                   sizeof(int),
+                   { static_cast<addr_t>(INVALID_CPU_MARKER_VALUE) } };


I think this should also have a valid but non-colliding value: wouldn't we want to pretend there's another PID here providing these synthetic records?

edeiana added 12 commits February 17, 2025 23:59

Save work.

d426b35

Save work.

f445f78

Save work.

f70f5f5

Minor improvements.

3814809

Save work.

cbe2176

Cleanup.

9f1a73c

Fixed result aggregation in scheduler issue.

d99c991

Do not count tid, pid, and tid exit when generating synthetic records.

e143e2f

Moved noise_generator into scheduler dir.

db8aa84

Added noise_generator_num_records flag and noise_generator unit test.

03d79a2

Minor improvements.

2e70cd4

Added end-to-end schedule stats test with noise_generator.

ae2d44f

edeiana changed the title ~~i#7216 noise generator basic structure~~ i#7216: noise generator basic structure Feb 21, 2025

edeiana added 2 commits February 21, 2025 03:10

Added changes to doc.

c906bf7

Added -noise_generator_num_records to scheduler options info.

Removed references from doc, as doxygen refuses to generate a link.

d7a4365

edeiana commented Feb 21, 2025

View reviewed changes

edeiana marked this pull request as ready for review February 21, 2025 11:38

edeiana requested a review from derekbruening February 21, 2025 11:38

Cannot rely on thread numbers or order for schedule_stats output.

6e21191

edeiana commented Feb 21, 2025

View reviewed changes

derekbruening reviewed Feb 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

i#7216: noise generator basic structure #7283

i#7216: noise generator basic structure #7283

edeiana commented Feb 18, 2025 •

edited

Loading

edeiana Feb 21, 2025

derekbruening Feb 21, 2025

edeiana Feb 21, 2025

derekbruening Feb 21, 2025

edeiana Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

derekbruening Feb 21, 2025

i#7216: noise generator basic structure #7283

Are you sure you want to change the base?

i#7216: noise generator basic structure #7283

Conversation

edeiana commented Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edeiana commented Feb 18, 2025 •

edited

Loading