Add Testing Platform Support

[slang25]

Today Stryker has support for VSTest as the test runner, and this is what is commonly used today by various testing frameworks.

However Microsoft have built a new replacement call the Microsoft Testing Platform, which is being adopted by all of the major players (MSTest, NUnit, xUnit).

While many of these frameworks will support both runner engines for a good while, there are also new testing frameworks that are only support on the new testing platform (the excellent TUnit for example).

It would be amazing if Stryker.NET add support for this as a new runner option.

[slang25]

I'll add that it may be possible to get indirect support today by using the VSTest Bridge extension, which I haven't tested but assume would enable TUnit support today.

[rouke-broersma]

We are working on it, but it is a very large feature. We have to reimplement a lot of vstest features ourselves because mstest runner assumes to be self contained in the test project and this doesn't really work for us since stryker is the executing binary. A start was made by an intern at #2988

As far as I know the test framework has to provide support for vstest, and TUnit could do this using the vstest bridge if they choose.

[richardwerkman]

I'm currently working on this. I'll take TUnit into account as well. Would be nice if we could support that.

[slang25]

Awesome, I didn't realise that this was being worked on, and how much work it is. I'm going to do some experimentation and see what I can get working today with TUnit,

[Lorilatschki]

First of all, thanks for the great tool, it really helps a lot qualifying our unit tests. We have currently moved to xunit.v3 and the new Microsoft Testing Platform. We not directly using references to the MTP, we enable running via MSBuild properties.

<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
<UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
<OutputType>Exe</OutputType>

Due to that, stryker does not run anymore when executed like.

dotnet-stryker --output .stryker/ --solution tests/stryker-test/src/all.sln --project Queue.csproj --test-project tests/stryker-test/src/Queue.Test/Queue.Test.csproj --mutation-level Advanced

Version: 4.5.1
[13:10:36 INF] Analysis starting.
[13:10:36 INF] Analyzing 1 test project(s).
[13:10:51 INF] Found project /builds//tests/stryker-test/src/Queue/Queue.csproj to mutate.
[13:10:51 INF] Analysis complete.
[13:10:51 INF] Building test project /builds//tests/stryker-test/src/Queue.Test/Queue.Test.csproj (1/1)
[13:10:51 INF] Building project all.sln using dotnet build all.sln (directory /builds//tests/stryker-test/src.)
[13:10:57 INF] Number of tests found: 2 for project /builds//tests/stryker-test/src/Queue/Queue.csproj. Initial test run started.
[13:11:00 INF] 5 mutants created
[13:11:00 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[13:11:02 ERR] It looks like the test coverage capture failed. Disable coverage based optimisation.
Hint: by passing "--open-report or -o" the report will open automatically and 
update the report in real-time.
[13:11:02 INF] 2     mutants got status Ignored.      Reason: Removed by block already covered filter
[13:11:02 INF] 2     total mutants are skipped for the above mentioned reasons
[13:11:02 INF] 3     total mutants will be tested
Killed:   0
Survived: 3
Timeout:  0

Since you will add support for the Microsoft Testing Platform, is there currently any estimated timeframe, when we can expect stryker to run against the MTP?

[rouke-broersma]

Can't give you an estimation sorry, it's a lot of work.

[richardwerkman]

@Lorilatschki On your project, does stryker run succesful with <TestingPlatformDotnetTestSupport>false</TestingPlatformDotnetTestSupport>? If so, we could introduce a workaround in Stryker that disables this when building the project. Thay way you could use MsTestRunner on your project, but stryker will use VsTest to run the tests.

Obviously we want to support running MsTest natively, but this will take some time to implement.

[richardwerkman]

I'm taking a new approach to implement this. Previously we made a POC that made Stryker the testhost. Basically stryker had to implement the MSTest.SDK and ran the test like this:

            var testApplicationBuilder = await TestApplication.CreateBuilderAsync(args);
            // Register the testing framework
            testApplicationBuilder.AddMSTest(() => new[] { testAssembly });
            var factory = new CompositeExtensionFactory<StrykerExtension>(serviceProvider => new StrykerExtension());
            testApplicationBuilder.TestHost.AddDataConsumer(factory);
            var testApplication = await testApplicationBuilder.BuildAsync();

            await testApplication.RunAsync();

The problem is, that since Stryker is the testhost, the tests don't run in an external process. That means we can't use paralellization. The benefit is that we have access to the process so we can listen to all MsTest events.

Now I'm thinking to inject the above code into the test project, and setting the method as the entry point for the assembly. That way we override the default entrypoint. That way we have full control over the testrunner.

Outstanding issues:

• We will need to figure out how to add the custom extension and make it available in the stryker process. We need this to control the test execution and get the test results.
• We need to figure out how to keep the test process alive and run the tests multiple times for reduced overhead.
• We need to inject the code into the test project, for this we might need to build test projects with Roslyn. Currently we build test projects with dotnet build.

[Lorilatschki]

@Lorilatschki On your project, does stryker run succesful with <TestingPlatformDotnetTestSupport>false</TestingPlatformDotnetTestSupport>? If so, we could introduce a workaround in Stryker that disables this when building the project. Thay way you could use MsTestRunner on your project, but stryker will use VsTest to run the tests.

Obviously we want to support running MsTest natively, but this will take some time to implement.

Unfortunately, setting the property TestingPlatformDotnetTestSupport to false leading to a failing stryker run. It states

Number of tests found: 121 for project xxx.csproj. Initial test run started.

and then it got stucked (those tests are usually running ~5s).

[slang25]

Perhaps @nohwnd or @Evangelink could advise on a good hosting strategy, as MTP is quite flexible in terms of hosting options.

I guess there are multiple approaches available, including shelling out to the test executables directly (now they are self executing), or spinning up dedicated stryker subprocesses that act as the hosts.

To summarize the requirements of stryker we need (correct me if I'm wrong):

• Initial test run
  • With per-test code coverage collection
  • Capturing timings per-test
• Mutation test runs (many times)
  • As efficiently as possible
  • With concurrent test run execution
  • Running different sets of tests per-run
  • Running with different target runtimes
  • Changing an environment variable (active mutants) per-run
  • With custom timeout per-testrun
  • Capturing test results
  • Isolation between runs, i.e. some static state in one test run shouldn't bleed over to another

That last point might rule out a long lived server mode approach, which I think is something that MTP makes possible (assuming this because there is a HotRelead package for MTP).

[rouke-broersma]

Another big requirement is support for different target runtimes, since each test project can have a different runtime requirement. This almost immediately requires external test host orchestration. Before long we are rebuilding vstest 😅

[nohwnd]

I agree on the last point, if you get rid of per-process isolation there is no isolation left, and you cannot guarantee that mutations to static state won't bleed into next iterations. So that is probably the limitation that will need to be acknowledged and addressed.

There is hotreload plugin, details and implementation can be found here: https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.HotReload/HotReloadTestHostTestFrameworkInvoker.cs

In essence, it works by scheduling multiple runs inside of a test run, and that way keeping the process hosting the tests alive, because from the point of view of the process we did not finish running tests yet.

Similar thing could be done by stryker, but it does not guarantee any isolation.

To me the most reasonable way to implement seems to be similar to what we do in VS / dotnet test. A remote client similar to vstestconsolewrapper, that talks with processes over some transport, automating them from the outside, running in separate processes. The speed limitation here is startup of the process and engine, and side by side builds of the different mutations. But the isolation is the best between tests.

Before long we are rebuilding vstest 😅

I don't think that is a bad thing, you'd be re-building the client we have in VS and dotnet test (for MTP those are 2 different clients), and I don't think there is that much work that needs to be done (but I am not stryker expert :) )

[slang25]

Thanks folks, have updated the requirements.

I think my mental model of MTP might be a bit off, I thought we could execute the test project (which is self-executing) and then talk to it via gRPC to tell it what tests to run. If this were true (and I suspect it's not 😅) then we could spin up the processes ahead of time, pooling them, and then grab a running process and start running the tests.

[dupdob]

I am afraid that the overall testing infrastructure gets more and more hostile to Stryker-like use cases. I.e, it optimizes for fast and safe execution of full test base vs the ability to fine control test runs.
One of the first victims of this trend is the actual test isolation. Test isolation is a core tenet of good unit testing and fundamental for Stryker's reliability. Alas, test case generator patterns break isolation. Plus test frameworks do not report this test case accurately.

In an ideal world, Stryker just needs to be able to run any tests list of this choosing and be able to abort it after a first failure (optional). If tests are properly isolated, this can happen within the confine of a single process.

Because test process setup/tear down is costly (around half a second with VsTest), Stryker needs to use as few as possible processes.

Note that thanks to coverage analysis, Stryker is able to test more than one mutation simultaneously, assuming any test only covers one of the active mutations.

Any way, I feel that the best way to integrate MTP with Stryker implies instrumenting the test projects so they behave as needed. But, may be this instrumentation does not need to be automatically injected by Stryker but could be purposely added by the user.

Ideally, he/she would just need to add the proper Stryker package to his/her test project and the this package will provide the needed logic.

[nohwnd]

If tests are properly isolated,

What isolation do you mean here exactly?

I am talking about running the test in an appdomain that will re-initialize statics etc. We don't have that in .NET anymore, same as thread aborts, because process level isolation is preferred, and so we have to work with that in MTP as well. When the runtime does not offer isolation, we cannot offer isolation.

But maybe we are each talking about a different thing.

If this level of isolation is not required then we can work within the limits of the process, and do basically anything, we don't want to be hostile to stryker, the opposite actually :)

[slang25]

I believe Microsoft uses stryker internally, so certainly wants to see it succeed here.

[rouke-broersma]

The speed limitation here is startup of the process and engine, and side by side builds of the different mutations.

This is one thing we were hoping would be improved with MTP since we can have more direct control over execution and shutdown. We may still be able to achieve this, since we know when mutants are in a static context. We can offer different levels of isolation with the least isolated method reusing the testrunner for all mutants except those that have to be isolated. If we can do that, then we can also offer a way to run all mutations in isolation (similar to modes we have now, but then with more direct control).

I think my mental model of MTP might be a bit off, I thought we could execute the test project (which is self-executing) and then talk to it via gRPC to tell it what tests to run. If this were true (and I suspect it's not 😅) then we could spin up the processes ahead of time, pooling them, and then grab a running process and start running the tests.

We can do that, if we inject the communication code and also the code that re-uses the test project. MTP itself is entirely self executing so it doesn't have anything for this, but it does contain extension points where we could inject our own communication and test ordering logic. That is something @richardwerkman is already working on and seems to be entirely possible. The hard part is that we will have to calculate all the different combinations of testrunner, targetframework, source project etc so we know which runners we need to spin up, when they can be reused. Because we will have to build all those different combinations of targets. Basically what vstest currently does by spinning up test hosts per whatever combination of targetframeworks are compatible with each other.

Any way, I feel that the best way to integrate MTP with Stryker implies instrumenting the test projects so they behave as needed. But, may be this instrumentation does not need to be automatically injected by Stryker but could be purposely added by the user.

We could also consider that option, Stryker provides it's own 'test adapter' implementation package with the execution and communication logic, then we don't need to inject it but the user would need to add this package.

[dupdob]

If tests are properly isolated,

What isolation do you mean here exactly?

Properly isolated tests should not depend on any (shared) static. I understand this is an ideal situation which is rarely achieved. Furthermore, tests must also be isolated when they fail, which is even harder to achieve.

[slang25]

I guess it's a requirement for mixed-mutant test runs, which I've never had a problem with.

[nohwnd]

Properly isolated tests should not depend on any (shared) static. I understand this is an ideal situation which is rarely achieved. Furthermore, tests must also be isolated when they fail, which is even harder to achieve.

I do agree with you here. But we cannot give any guarantees. Which was how I approached your comment.

If user does the work themselves, and keeps good practice to make tests independent, not influencing each other in any way even if they fail. Then I don't see a problem with running them in one process and scheduling them in a way that is similar to how hotreload extension does it.

But again I am not an expert in stryker-dotnet. @richardwerkman I assume the code you wrote is internal, and not public, and we would need to have a private call with you and your team to look at it?

[rouke-broersma]

Properly isolated tests should not depend on any (shared) static. I understand this is an ideal situation which is rarely achieved. Furthermore, tests must also be isolated when they fail, which is even harder to achieve.

I do agree with you here. But we cannot give any guarantees. Which was how I approached your comment.

If user does the work themselves, and keeps good practice to make tests independent, not influencing each other in any way even if they fail. Then I don't see a problem with running them in one process and scheduling them in a way that is similar to how hotreload extension does it.

But again I am not an expert in stryker-dotnet. @richardwerkman I assume the code you wrote is internal, and not public, and we would need to have a private call with you and your team to look at it?

So far it's been experiments so I don't think Richard pushed the code yet but our work on stryker is fully open source so we can publish it, no problem. @richardwerkman are you experiments in a state that it would be useful to share with @nohwnd ?

[richardwerkman]

@nohwnd I've prepared a little sample on a branch.

stryker-net/MTP Demo/MsTestRunnerDemo/MsTestRunnerDemo/Program.cs

Line 6 in f655647

// this works, as long as the target framework is the same

I've kinda hit a dead end here I think with the injected code method. It works, as long as the target framework of the executing assembly is the same as in the test project. But then the whole "bennefit" of the code injection is gone... So I see two solutions: running in a new process with the dotnet cli. But this is slow and gives less control. Or making sure the correct runtimes are loaded into stryker. But that seems like a no go to me.

Do you thinks there is another solution for this? Maybe there is another way of starting MTP that I haven't found yet.

Also I'm curious if you have a sample from VS or dotnet test where MTP is used with the hotreload plugin? That looks promising. So far I could only rerun tests by calling TestApplication.BuildAsync() multiple times and then running the build test application.

[nohwnd]

Looking at the sample code it will load a dll into an executable, and run it's main method multiple times. This should work just fine a long as the executing assembly is targeting the same target framework family as the loaded dll, meaning .NET process needs to load .NET, and .NET Framework needs to load .NET Framework. Of course architectures also need to agree, but I am assuming AnyCPU.

The process needs to use appropriate target framework version as well, matching (or higher) than the highest TFM of the loaded assemblies.

But overall I don't see why this should not work. This is pretty much how testhost.exe works in vstest, even though there we build it with the newest SDK, targeting the lowest supported TFM and then provide deps.json and runtime.config.json that is appropriate for the version of the tested assembly, but if we always provided net10 all dlls would load just fine. We also don't re-use testhost because we are afraid of cross-pollution, but we could.

And here I reach the point where I am missing the details of how stryker works :) I am assuming what you are trying to achieve here is that you start 1 process, load an assembly into it, and run it's tests. Then you mutate? And then you run again?

Or do you have a lineup of mutated dlls that you can run, and by the env variables that you set you are able to reach the different mutations without unloading the dll?

For hot reload, I don't think there is any public implementation of the hot-reload part that is done on VS side, the test client on VS side detects that hot reloading happened, and tells the dll to run tests again. Details of how that happens in the assembly can be found here: https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.HotReload/HotReloadTestHostTestFrameworkInvoker.cs

But I don't know if you find details of how the assembly is actually patched, I've asked around if this is tied to VS or also available for others to use.

Demo of how it is used can be seen here, I have no headphones here, but hopefully I am linking close enough to the start of the demo: https://youtu.be/9Jz47Ze9LOI?si=AymLqdCBZapWV1n7&t=1243

[rouke-broersma]

Or do you have a lineup of mutated dlls that you can run, and by the env variables that you set you are able to reach the different mutations without unloading the dll?

Yes, this is what we do in stryker. We mutate once, place all mutations in branches and activate mutations as needed. We then run the unit tests that are relevant for the enabled mutation to limit runtime and we collect the result te determine if mutants are killed/survived.

[nohwnd]

In that case it seems to me like it should work with the approach that is similar to hot reload extension (or maybe even that one). Personally I would stay with the constraints of the same test project (but maybe not necessarily the same tfm, just within the same family). Because then the project has the correct deps.json.

Too bad we don't have an official client yet, that would make doing a protype for you much easier. Will try to whip something up and talk with people in our team.

[richardwerkman]

But overall I don't see why this should not work. This is pretty much how testhost.exe works in vstest, even though there we build it with the newest SDK, targeting the lowest supported TFM and then provide deps.json and runtime.config.json that is appropriate for the version of the tested assembly, but if we always provided net10 all dlls would load just fine. We also don't re-use testhost because we are afraid of cross-pollution, but we could.

So that would mean we (as stryker) have to stay up to date with the latest framework all the time? Currently we only update for LTS releases. But even then that would mean we can't run testprojects that use a preview version.

[rouke-broersma]

But overall I don't see why this should not work. This is pretty much how testhost.exe works in vstest, even though there we build it with the newest SDK, targeting the lowest supported TFM and then provide deps.json and runtime.config.json that is appropriate for the version of the tested assembly, but if we always provided net10 all dlls would load just fine. We also don't re-use testhost because we are afraid of cross-pollution, but we could.

So that would mean we (as stryker) have to stay up to date with the latest framework all the time? Currently we only update for LTS releases. But even then that would mean we can't run testprojects that use a preview version.

Only the test host would need to be up-to-date, right? And we would need to build a test host for dotnet and one for dotnet framework, but not necessarily one for each TFM. Is that right @nohwnd?

[nohwnd]

The test project itself would be your host, you would be just automating it from the outside via an orchestrator. Similarly to what dotnet test does for example.

[nohwnd]

Posted too soon :)

stryker.exe -> testProject1.exe, all test projects need to install your MTP based extension as nuget, that is similar to what our hotReload extension does. Stryker.exe (or some other global tool, that I think you already have) will orchestrate the testproject1.exe and other processes. And will tell them:

load testproject1.dll, and run with those env variables to run mutations,
done?
okay now run testproject1.dll with these other env variables to run other mutations

as Richard shown above in the prototype.

[nohwnd]

Stryker.exe can target any tfm, its purpose would be to mutate and build the different versions of the mutated projects, by calling dotnet build or however you do it. As long as that dotnet build comes from dotnet SDK that supports net10 preview then you can build net10 projects, and because there is air gap between your stryker.exe orachestrator and the testProject1.exe you don't care about the TFM of testProject1.exe.

[nohwnd]

https://github.com/microsoft/testfx/blob/1b5657e065cb412244f2e677dcd30db6eda6b8b2/docs/mstest-runner-protocol/001-protocol-intro.md?plain=1 talking would be done over this protocol, for which we don't have official client package yet.

But the spec is open, and tunit implements the messages you'd need: https://github.com/thomhurst/TUnit/blob/main/TUnit.RpcTests/Clients/TestingPlatformClient.cs

[nohwnd]

This is how you start the exe in server mode https://github.com/thomhurst/TUnit/blob/main/TUnit.RpcTests/Tests.cs#L40

Similar stuff in our own server mode tests, https://github.com/microsoft/testfx/blob/main/test/UnitTests/Microsoft.Testing.Platform.UnitTests/ServerMode/ServerTests.cs

[rouke-broersma]

How would that work with multiple test assemblies? Orchestrate multiple test hosts (test projects) for the same env and aggregate the results ourselves? Because before I was thinking we would create a generic test host that implements MTP -> load all needed test project dll's instead of letting the test project be self-executing.

all test projects need to install your MTP based extension

We considered this, alternatively we might simply inject the MTP code since we already mutate the source project might as well auto-instrumentate the test project as well to lower the barrier to entry.

But thanks for all the information, that is really useful!

[nohwnd]

How would that work with multiple test assemblies? Orchestrate multiple test hosts (test projects) for the same env and aggregate the results ourselves? Because before I was thinking we would create a generic test host that implements MTP -> load all needed test project dll's instead of letting the test project be self-executing.

Yes like that. You can make a project that aggregates multiple dlls test dlls. I think we even natively support it in mstest, because you can provide multiple test dlls to the mstest invocation (addMSTest takes an array, https://grep.app/search?q=addmstest) I am not sure if this path is well tested but should be possible.

Personally I would look into the 1 testhost per test project approach, because that won't force you to "mix" dependencies of test projects, and the tested code, so you are actually testing what you deploy, and it also is easier to run in parallel.

auto instrument

Yeah that is a good idea. Should work either way.

So do you think the orchestration would work for Styker? So I don't waste much time making prototype for something you will not be able to use.

[rouke-broersma]

I think the orchestration would work fine. This is how we currently broadly speaking orchestrate VsTest:

I will try to draw how I imagine it would work with MTP in a minute, you can let me know if that makes sense

Something like this: