Add "delay" and "delayType" options to secnetperf tool #4807

sm-msft · 2025-02-12T01:00:57Z

This not quite the same as the feature request described in issue #1524, but it achieves some of the goals. The stream processing on the server side is already multithreaded and the use of exponential probability distribution to determine the delay for the current stream simulates variable processing time for a request on the server side.

We can introduce background thread processing as suggested in the issue description to simulate further processing delays on the server side to enable a richer simluation.

Description

Add the ability to introduce server-side delays into data streams in the secnetperf tool.

Default/fixed delay introduces specific microseconds of inline busy-wait delay when the stream is established
Variable delay introduces an exponetially distributed random busy-wait delay. This is capped at max(1millisecond, 4x given delay) of inline sleep.
Documentation/help text updated
Minor code reorganization

Testing

This change updates the test tool. Existing tests are all passing with this change.

Documentation

Documentation has been updated.

- Default/fixed delay introduces specific microseconds of inline busy-wait delay when the stream is established - Variable delay introduces an exponetially distributed random busy-wait delay. This is capped at 1millisecond inline sleep. - Documentation/help text updated

codecov · 2025-02-12T01:29:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.91%. Comparing base (7b2fbaf) to head (15a1b04).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4807      +/-   ##
==========================================
- Coverage   87.43%   86.91%   -0.53%     
==========================================
  Files          56       56              
  Lines       17634    17634              
==========================================
- Hits        15419    15326      -93     
- Misses       2215     2308      +93

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/perf/lib/PerfServer.cpp

src/perf/readme.md

src/perf/lib/PerfServer.cpp

src/perf/lib/SecNetPerfMain.cpp

src/perf/lib/PerfServer.h

…and sending responses from the server. - Based on the perflib TCP worker threadpool code - 1 worker thread per processor - Existing code (QUIC/TCP) queues background work on the current processor only when a server response is due. - The existing Quic/TCP connection/stream is assumed to be available till the background operation completes - DelayWork thread adds appropriate amount of delay and then sends the response back on the appropriate stream *Minor code reorganization in PerfServer.h *Addressed review comments from prior iteration

src/perf/lib/SecNetPerfMain.cpp

src/perf/lib/PerfServer.cpp

nibanks · 2025-02-21T13:51:48Z

src/perf/lib/PerfServer.cpp

+    InterlockedFetchAndSetBoolean(&ExecutionContext.Ready);
+    ExecutionContext.NextTimeUs = UINT64_MAX;
+
+    uint16_t ThreadFlags = CXPLAT_THREAD_FLAG_SET_IDEAL_PROC;


Not really necessary to make a separate variable just for this.

nibanks · 2025-02-21T13:52:28Z

src/perf/lib/PerfServer.cpp

+
+    uint16_t ThreadFlags = CXPLAT_THREAD_FLAG_SET_IDEAL_PROC;
+    //
+    // Not using the high priority flag for delay threads


Let's just remove this comment. Most things don't run at high priority, so saying this isn't using that is unnecessary IMO.

This is in contrast with other threadpools we use in this binary and we have to highlight it somehow..

src/perf/lib/PerfServer.cpp

nibanks · 2025-02-21T13:55:09Z

src/perf/lib/PerfServer.cpp

+    if (nullptr == Work) {
+        return false;
+    }


Since this is test/tool code, we generally assume allocation success. Let's remove this check, which simplifies the whole function (it can't fail then) as well as the callers.

src/perf/lib/PerfServer.h

src/perf/lib/SecNetPerf.h

nibanks · 2025-02-21T13:58:13Z

src/perf/lib/SecNetPerfMain.cpp

+        "  -delay:<####>[unit]      Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (def:0)\n"
+        "  -delayType:<fixed|variable>    Optional delay type can be specified in conjunction with the 'delay' argument.\n"


Suggested change

" -delay:<####>[unit] Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (def:0)\n"

" -delayType:<fixed|variable> Optional delay type can be specified in conjunction with the 'delay' argument.\n"

" -delay:<####>[unit] Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (def:0)\n"

" -delayType:<type> Optional delay type can be specified in conjunction with the 'delay' argument.\n"

We generally use this convention for parameters

src/perf/lib/SecNetPerfMain.cpp

nibanks · 2025-02-21T14:01:05Z

src/perf/readme.md

@@ -29,6 +29,8 @@ ecn | `-ecn:<0,1>` | Enables sender-side ECN support.
 exec | `-exec:<lowlat,maxtput,scavenger,realtime>` | The execution profile used for the application.
 pollidle | `-pollidle:<time_us>` | The time, in microseconds, to poll while idle before sleeping (falling back to interrupt-driven IO).
 stats | `-stats:<0,1>` | Prints out statistics at the end of each connection.
+delay | `[-delay:<value>[units]]` | Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (default:0)


Suggested change

delay | `[-delay:<value>[units]]` | Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (default:0)

delay | `[-delay:<value>[units]]` | Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request.

src/perf/lib/PerfServer.cpp

nibanks · 2025-02-21T14:04:05Z

src/perf/lib/PerfServer.cpp

+CXPLAT_THREAD_CALLBACK(DelayWorker::WorkerThread, Context)
+{
+    DelayWorker* This = (DelayWorker*)Context;
+    CXPLAT_EXECUTION_STATE DummyState = {


You're never using CXPLAT_EXECUTION_STATE anywhere. Go ahead and remove all references.

src/perf/lib/PerfServer.cpp

nibanks · 2025-02-21T14:05:44Z

src/perf/lib/PerfServer.cpp

+        This->m_Server->SimulateDelay();
+        This->m_Server->SendResponse(WorkItem->Context, WorkItem->Handle, WorkItem->IsTcp);
+        delete WorkItem;
+        InterlockedFetchAndSetBoolean(&This->ExecutionContext.Ready); // We just did work, let's keep this thread hot.


The usage of ExecutionContext and Ready is overkill here. I'm pretty sure we can just remove it. In general, you just need the queue and the wake event I think.

src/perf/lib/SecNetPerfMain.cpp

src/perf/lib/PerfServer.cpp

sm-msft assigned sm-msft and nibanks and unassigned sm-msft Feb 12, 2025

guhetier reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

sm-msft commented Feb 12, 2025

View reviewed changes

src/perf/readme.md Show resolved Hide resolved

nibanks added the Area: Performance label Feb 12, 2025

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.h Outdated Show resolved Hide resolved

nibanks reviewed Feb 12, 2025

View reviewed changes

src/perf/lib/PerfServer.h Outdated Show resolved Hide resolved

nibanks changed the title ~~- Add "delay" and "delayType" options to secnetperf tool~~ Add "delay" and "delayType" options to secnetperf tool Feb 12, 2025

sm-msft added 5 commits February 18, 2025 17:40

- replace max with explicit code

2d01e19

- Specific template instantiation to address linker issues

dcef37f

- Address build failures (templates/functions not being defined)

dd4b739

- Specific template to address linker errors

c1a4a34

sm-msft commented Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Show resolved Hide resolved

sm-msft commented Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.h Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.h Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerf.h Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerf.h Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/SecNetPerfMain.cpp Show resolved Hide resolved

nibanks reviewed Feb 21, 2025

View reviewed changes

src/perf/lib/PerfServer.cpp Outdated Show resolved Hide resolved

sm-msft added 4 commits February 21, 2025 10:15

- Reorder initialization of members in DelayWorker constructor

3e40376

Most review comments addressed

ca69bb1

Merge branch 'main' into user/sm/perf_with_simulated_delays

18c9065

Test fix for winkernel build break

15a1b04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "delay" and "delayType" options to secnetperf tool #4807

Add "delay" and "delayType" options to secnetperf tool #4807

sm-msft commented Feb 12, 2025 •

edited

Loading

codecov bot commented Feb 12, 2025 •

edited

Loading

nibanks Feb 21, 2025

nibanks Feb 21, 2025

sm-msft Feb 21, 2025

nibanks Feb 21, 2025

nibanks Feb 21, 2025

nibanks Feb 21, 2025

nibanks Feb 21, 2025

nibanks Feb 21, 2025

		" -delay:<####>[unit] Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (def:0)\n"
		" -delayType:<fixed\|variable> Optional delay type can be specified in conjunction with the 'delay' argument.\n"

	delay \| `[-delay:<value>[units]]` \| Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request. (default:0)
	delay \| `[-delay:<value>[units]]` \| Delay, with an optional unit (def unit is us), to be introduced before the server responds to a request.

Add "delay" and "delayType" options to secnetperf tool #4807

Are you sure you want to change the base?

Add "delay" and "delayType" options to secnetperf tool #4807

Conversation

sm-msft commented Feb 12, 2025 • edited Loading

Description

Testing

Documentation

codecov bot commented Feb 12, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sm-msft commented Feb 12, 2025 •

edited

Loading

codecov bot commented Feb 12, 2025 •

edited

Loading