From 71b26d71cb6cba545fd5e0d1d4dfcd7c9a89d942 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:07:43 +0100 Subject: [PATCH 01/68] Add docs for Microsoft.Testing.Platform integration Added a new guide for running benchmarks with Microsoft.Testing.Platform (MTP), covering setup, usage, and caveats. Updated the table of contents to include the new page and added a note to the VSTest docs about the MTP adapter option. --- docs/articles/features/testingplatform.md | 177 ++++++++++++++++++++++ docs/articles/features/toc.yml | 4 +- docs/articles/features/vstest.md | 5 + 3 files changed, 185 insertions(+), 1 deletion(-) create mode 100644 docs/articles/features/testingplatform.md diff --git a/docs/articles/features/testingplatform.md b/docs/articles/features/testingplatform.md new file mode 100644 index 0000000000..85fffeeb0b --- /dev/null +++ b/docs/articles/features/testingplatform.md @@ -0,0 +1,177 @@ +--- +uid: docs.testingplatform +name: Running with Microsoft.Testing.Platform +--- + +# Running with Microsoft.Testing.Platform + +BenchmarkDotNet can discover and execute benchmarks through + [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) (MTP), + the test platform that succeeds VSTest. +This gives you the same "benchmarks as tests" experience as [the VSTest adapter](xref:docs.vstest), + but on the platform that `dotnet test` and modern IDE integrations are moving to. + +If you are looking for the VSTest adapter, see [Running with VSTest](xref:docs.vstest) instead. +You only need one of the two. + +## VSTest or Microsoft.Testing.Platform? + +The two adapters solve the same problem on different platforms, and the difference that matters most is *where your + benchmarks run*: + +* With **VSTest**, an external `testhost` process loads your benchmark assembly and the adapter reflects into it. +* With **Microsoft.Testing.Platform**, your benchmark project *is* the test host. + There is no separate host process, so BenchmarkDotNet behaves exactly as it does when you run the app from the CLI. + +The practical consequences of the MTP model are: + +* The adapter no longer needs the child `AppDomain` that the VSTest adapter uses to load your assemblies correctly. +* Your project's entry point is generated by the platform and starts the test application, + so it no longer calls `BenchmarkSwitcher`. See [Keeping a BenchmarkSwitcher entry point](#keeping-a-benchmarkswitcher-entry-point). + +## Caveats and things to know + +* **The benchmark measurements may be affected by the test host and your IDE!** + If you want accurate measurements, + it is still recommended to run benchmarks through the CLI without other processes impacting performance. + The measurements remain useful during development when comparing different approaches. +* **The adapter will not display or execute benchmarks if optimizations are disabled.** + Please ensure you are compiling in Release mode or with `Optimize` set to true. + Using an `InProcess` toolchain will let you run your benchmarks with optimizations disabled + and will let you attach the debugger as well. +* **The adapter will not call your application's entry point.** + If you use the entry point to customize how your benchmarks are run, + you will need to do this through other means such as an assembly-level `IConfigSource`, + as shown in [Setting a default configuration](xref:docs.vstest#setting-a-default-configuration). +* **The adapter will generate an entry point for you automatically.** + Unlike the VSTest adapter, the generated entry point starts the test application rather than `BenchmarkSwitcher`. + +## Getting started + +* **Step 1.** Install the NuGet package. + Only one package is needed; it brings in `Microsoft.Testing.Platform` and the MSBuild integration for you: + +```xml + + + +``` + +* **Step 2.** Make sure the project is an executable and does not define its own entry point. + Microsoft.Testing.Platform applications are executables, and the package generates the entry point for you. + Here is a complete `.csproj` based on the default Console Application template: + +```xml + + + + Exe + net10.0 + enable + enable + + + + + + + +``` + +> [!NOTE] +> The name of your project file must match the name of the produced assembly. +> This is a general BenchmarkDotNet requirement: it rebuilds your project to run benchmarks out of process. + +* **Step 3.** Opt into the Microsoft.Testing.Platform mode of `dotnet test`. + On the .NET 10 SDK and later this is required, because `dotnet test` runs in VSTest mode by default. + Add a `global.json` next to your solution: + +```json +{ + "test": { + "runner": "Microsoft.Testing.Platform" + } +} +``` + + On the .NET 9 SDK and earlier this step is not needed: + the package sets `TestingPlatformDotnetTestSupport` for you, which routes `dotnet test` to the platform. + +* **Step 4.** Switch to the `Release` configuration. + As mentioned above, the adapter does not discover or run benchmarks with optimizations disabled (by design). + +* **Step 5.** Build and run. + +```console +dotnet test -c Release +``` + + You can also run the produced executable directly, which is the same thing without going through MSBuild: + +```console +dotnet run -c Release +``` + +If this doesn't work for you, don't hesitate to file [a new GitHub issue](https://github.com/dotnet/BenchmarkDotNet/issues/new). + +## Listing and filtering benchmarks + +The benchmark project is a normal Microsoft.Testing.Platform application, so it accepts the platform's options. +Run it with `--help` to see all of them; the ones you are most likely to want are: + +```console +# List the benchmarks without running them. +dotnet run -c Release -- --list-tests + +# Run every benchmark of a class. +dotnet run -c Release -- --treenode-filter "/*/*/MyBenchmarks/*" + +# Run every benchmark of a category. +dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" + +# Run one specific benchmark, by the exact id reported by the platform. +dotnet run -c Release -- --filter-uid "MyProject.MyBenchmarks.Add(x: 1) [DefaultJob]" +``` + +The tree node filter path is `////`, + and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. + +## Keeping a BenchmarkSwitcher entry point + +The generated entry point starts the test application, which means it replaces the `BenchmarkSwitcher` entry point that + a benchmark project normally has. +If you want to keep your own entry point, turn off the generated one and register BenchmarkDotNet yourself: + +```xml + + false + +``` + +```csharp +using BenchmarkDotNet.TestAdapter.TestingPlatform; +using Microsoft.Testing.Platform.Builder; + +public static class Program +{ + public static async Task Main(string[] args) + { + var builder = await TestApplication.CreateBuilderAsync(args); + builder.AddBenchmarkDotNet(); + using var app = await builder.BuildAsync(); + return await app.RunAsync(); + } +} +``` + +From there you are free to decide when to start the test application and when to hand over to `BenchmarkSwitcher`, + for example by looking at the arguments your CI passes. + +## Viewing the results + +The full BenchmarkDotNet output, including the summary table that compares benchmarks with each other, + is written to the test run output. + +In addition, each individual benchmark reports its own output, containing a histogram and various statistics for that + single benchmark case. +Depending on your IDE, this is shown when selecting the test after running it. diff --git a/docs/articles/features/toc.yml b/docs/articles/features/toc.yml index b456ed66be..f2fb0728d2 100644 --- a/docs/articles/features/toc.yml +++ b/docs/articles/features/toc.yml @@ -17,4 +17,6 @@ - name: VSProfiler href: vsprofiler.md - name: VSTest - href: vstest.md \ No newline at end of file + href: vstest.md +- name: Microsoft.Testing.Platform + href: testingplatform.md \ No newline at end of file diff --git a/docs/articles/features/vstest.md b/docs/articles/features/vstest.md index 3e94901bdd..e86527211c 100644 --- a/docs/articles/features/vstest.md +++ b/docs/articles/features/vstest.md @@ -5,6 +5,11 @@ name: Running with VSTest # Running with VSTest +> [!NOTE] +> BenchmarkDotNet also ships an adapter for [Microsoft.Testing.Platform](xref:docs.testingplatform), +> the test platform that succeeds VSTest. +> You only need one of the two. + BenchmarkDotNet supports discovering and executing benchmarks through VSTest. This provides an alternative user experience to running benchmarks with the CLI and may be preferable for those who like their IDE's VSTest integrations that they may have used when running unit tests. From 8918420561a72698e012f764bd80472c134874e3 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:08:20 +0100 Subject: [PATCH 02/68] Add InternalsVisibleTo for TestAdapter.TestingPlatform Added InternalsVisibleTo attribute in AssemblyInfo.cs to expose internal members to the BenchmarkDotNet.TestAdapter.TestingPlatform assembly, ensuring it uses the same public key as other related assemblies. --- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 1 + 1 file changed, 1 insertion(+) diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index aa93984520..5a5ff709d3 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -15,3 +15,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 3dd579644cfe191d745fff973a63a2082d410ec5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:08:40 +0100 Subject: [PATCH 03/68] Remove GetUnrandomizedJobDisplayInfo method Deleted the internal static method GetUnrandomizedJobDisplayInfo from BenchmarkCaseExtensions.cs. This method handled normalization of job display info by removing randomness from job IDs for consistent benchmark referencing. No other code changes were made. --- .../BenchmarkCaseExtensions.cs | 21 ------------------- 1 file changed, 21 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs index 06e5782b6a..08ad640e26 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs @@ -1,5 +1,4 @@ using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Characteristics; using BenchmarkDotNet.Exporters; using BenchmarkDotNet.Extensions; using BenchmarkDotNet.Running; @@ -65,26 +64,6 @@ internal static TestCase ToVsTestCase(this BenchmarkCase benchmarkCase, string a return vsTestCase; } - /// - /// If an ID is not provided, a random string is used for the ID. This method will identify if randomness was - /// used for the ID and return the Job's DisplayInfo with that randomness removed so that the same benchmark - /// can be referenced across multiple processes. - /// - /// The benchmark case. - /// The benchmark case' job's DisplayInfo without randomness. - internal static string GetUnrandomizedJobDisplayInfo(this BenchmarkCase benchmarkCase) - { - var jobDisplayInfo = benchmarkCase.Job.DisplayInfo; - if (!benchmarkCase.Job.HasValue(CharacteristicObject.IdCharacteristic) && - benchmarkCase.Job.ResolvedId.StartsWith("Job-", StringComparison.OrdinalIgnoreCase)) - { - // Replace Job-ABCDEF with Job - jobDisplayInfo = "Job" + jobDisplayInfo.Substring(benchmarkCase.Job.ResolvedId.Length); - } - - return jobDisplayInfo; - } - /// /// Gets an ID for a given BenchmarkCase that is uniquely identifiable from discovery to execution phase. /// From 695ee488ec41dee8f7908a4619b1f3fa95c8b516 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:08:56 +0100 Subject: [PATCH 04/68] Add BenchmarkCaseIdentityExtensions for stable job IDs Introduced BenchmarkCaseIdentityExtensions with GetUnrandomizedJobDisplayInfo to normalize Job DisplayInfo by removing random ID components. This ensures consistent benchmark identification across processes for test adapters. --- .../BenchmarkCaseIdentityExtensions.cs | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs new file mode 100644 index 0000000000..78cb64f5ce --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs @@ -0,0 +1,32 @@ +using BenchmarkDotNet.Characteristics; +using BenchmarkDotNet.Running; + +namespace BenchmarkDotNet.TestAdapter +{ + /// + /// Helpers for deriving stable identities for a BenchmarkCase. Shared by the VSTest and the + /// Microsoft.Testing.Platform adapters, because both need identities that survive across processes. + /// + internal static class BenchmarkCaseIdentityExtensions + { + /// + /// If an ID is not provided, a random string is used for the ID. This method will identify if randomness was + /// used for the ID and return the Job's DisplayInfo with that randomness removed so that the same benchmark + /// can be referenced across multiple processes. + /// + /// The benchmark case. + /// The benchmark case' job's DisplayInfo without randomness. + internal static string GetUnrandomizedJobDisplayInfo(this BenchmarkCase benchmarkCase) + { + var jobDisplayInfo = benchmarkCase.Job.DisplayInfo; + if (!benchmarkCase.Job.HasValue(CharacteristicObject.IdCharacteristic) && + benchmarkCase.Job.ResolvedId.StartsWith("Job-", StringComparison.OrdinalIgnoreCase)) + { + // Replace Job-ABCDEF with Job + jobDisplayInfo = "Job" + jobDisplayInfo.Substring(benchmarkCase.Job.ResolvedId.Length); + } + + return jobDisplayInfo; + } + } +} From 770bd0d39ed02216c2525835f1a01d27078dd960 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:09:26 +0100 Subject: [PATCH 05/68] Refactor benchmark extraction into reusable method Introduce GetBenchmarksFromAssembly to extract benchmarks from an already loaded Assembly. Refactor existing logic to use this method, improving code reuse and enabling benchmark retrieval from both loaded assemblies and file paths. --- .../BenchmarkEnumerator.cs | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs index e8cbc4f130..f001ef45f8 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs @@ -48,8 +48,16 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssemblyPath(string assemblyPa }; #endif - var assembly = Assembly.LoadFrom(assemblyPath); + return GetBenchmarksFromAssembly(Assembly.LoadFrom(assemblyPath)); + } + /// + /// Returns all the BenchmarkRunInfo objects from an already loaded assembly. + /// + /// The assembly of the benchmark project. + /// The benchmarks inside the assembly. + public static BenchmarkRunInfo[] GetBenchmarksFromAssembly(Assembly assembly) + { var isDebugAssembly = assembly.IsJitOptimizationDisabled() ?? false; return GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks()) @@ -59,7 +67,7 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssemblyPath(string assemblyPa if (isDebugAssembly) { // If the assembly is a debug assembly, then only display them if they will run in-process - // This will allow people to debug their benchmarks using VSTest if they wish. + // This will allow people to debug their benchmarks from a test runner if they wish. benchmarkRunInfo = new BenchmarkRunInfo( benchmarkRunInfo.BenchmarksCases.Where(c => c.GetToolchain().IsInProcess).ToArray(), benchmarkRunInfo.Type, From 0fa20cd8f7a4525876060389dac9fac10c45ebe1 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:09:57 +0100 Subject: [PATCH 06/68] Integrate BenchmarkDotNet with Microsoft.Testing.Platform Add MSBuild props to enable TestingPlatform integration, set defaults for `dotnet test` compatibility, disable parallel TFM runs, and auto-register BenchmarkDotNet builder hook. --- ...rkDotNet.TestAdapter.TestingPlatform.props | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props new file mode 100644 index 0000000000..7b6f563554 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props @@ -0,0 +1,31 @@ + + + + true + + + true + + + false + + + + + + BenchmarkDotNet + BenchmarkDotNet.TestAdapter.TestingPlatform.TestingPlatformBuilderHook + + + From a97a8f185552576ad4a16c63f43880a463dc2395 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:10:37 +0100 Subject: [PATCH 07/68] Add AsyncWorkQueue for async work item processing Introduced AsyncWorkQueue, an internal sealed class in BenchmarkDotNet.TestAdapter.TestingPlatform. It enables ordered, thread-safe queuing of asynchronous work items, allowing synchronous producers and asynchronous consumers. Utilizes ConcurrentQueue and SemaphoreSlim, supports completion signaling, and implements IDisposable for resource cleanup. --- .../AsyncWorkQueue.cs | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs new file mode 100644 index 0000000000..128eb51b0c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs @@ -0,0 +1,61 @@ +using System.Collections.Concurrent; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. + /// + /// + /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the + /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks + /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks + /// enqueue instead and the caller of DrainAsync does the awaiting. + /// + internal sealed class AsyncWorkQueue : IDisposable + { + private readonly ConcurrentQueue> queue = new(); + private readonly SemaphoreSlim available = new(0); + private volatile bool completed; + + /// + /// Queues a work item. Safe to call from any thread. + /// + /// The work to perform. + public void Enqueue(Func work) + { + queue.Enqueue(work); + available.Release(); + } + + /// + /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued + /// items have been processed. + /// + public void Complete() + { + completed = true; + available.Release(); + } + + /// + /// Processes queued work items in order until has been called and the queue is empty. + /// + /// A task that completes when the queue has been drained. + public async Task DrainAsync() + { + while (true) + { + await available.WaitAsync().ConfigureAwait(false); + + // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. + while (queue.TryDequeue(out var work)) + await work().ConfigureAwait(false); + + if (completed && queue.IsEmpty) + return; + } + } + + public void Dispose() => available.Dispose(); + } +} From ff50c125601df76e6bef44e804e38b1f5efc4c76 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:11:47 +0100 Subject: [PATCH 08/68] Add BenchmarkDotNet.TestAdapter.TestingPlatform project Created a new .csproj targeting netstandard2.0 for the TestingPlatform adapter. Configured project metadata, packaging, and references. Integrated Microsoft.Testing.Platform.MSBuild and BenchmarkDotNet, and linked shared source files for benchmark enumeration. Set IsTestingPlatformApplication to false to avoid test app behavior. --- ...kDotNet.TestAdapter.TestingPlatform.csproj | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj new file mode 100644 index 0000000000..c3dd5c4cb3 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj @@ -0,0 +1,46 @@ + + + + netstandard2.0 + BenchmarkDotNet.TestAdapter.TestingPlatform + BenchmarkDotNet.TestAdapter.TestingPlatform + BenchmarkDotNet.TestAdapter.TestingPlatform + Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform + README.md + True + BenchmarkDotNet.TestAdapter.TestingPlatform + + + false + + + + + + + + + + + + + + + + + + + + + + + + + + From 46e4df0b5fa744a7d6893e294c38001f0b6d08a5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:13:27 +0100 Subject: [PATCH 09/68] Add BenchmarkDotNetExtension for platform integration Introduced BenchmarkDotNetExtension class implementing IExtension to provide extension metadata and enablement status for Microsoft.Testing.Platform integration. --- .../BenchmarkDotNetExtension.cs | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs new file mode 100644 index 0000000000..789fbc611a --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs @@ -0,0 +1,34 @@ +using Microsoft.Testing.Platform.Extensions; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. + /// + /// + /// Several platform services, such as the tree node filter behind --filter, are registered on behalf of an + /// extension rather than of the test framework itself, so the identity lives in its own type. + /// + internal sealed class BenchmarkDotNetExtension : IExtension + { + /// + /// The uid shared by every extension this package registers. + /// + public const string ExtensionUid = "BenchmarkDotNet.TestAdapter.TestingPlatform"; + + /// + public string Uid => ExtensionUid; + + /// + public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; + + /// + public string DisplayName => "BenchmarkDotNet"; + + /// + public string Description => "Runs BenchmarkDotNet benchmarks as tests."; + + /// + public Task IsEnabledAsync() => Task.FromResult(true); + } +} From 2a004ccb4db49c58f442e29b7cf488f28bd62467 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:14:56 +0100 Subject: [PATCH 10/68] Add BenchmarkEventProcessor for test node event handling Introduced BenchmarkEventProcessor to process BenchmarkDotNet events and translate them into test node updates for the testing platform. Handles validation errors, build results, benchmark execution, and ensures all benchmarks have published results. Includes logic for error aggregation, output formatting, and timing information. --- .../BenchmarkEventProcessor.cs | 191 ++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs new file mode 100644 index 0000000000..b48e5cd6e6 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs @@ -0,0 +1,191 @@ +using BenchmarkDotNet.EventProcessors; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Reports; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains.Results; +using BenchmarkDotNet.Validators; +using Microsoft.Testing.Platform.Extensions.Messages; +using Perfolizer.Mathematics.Histograms; +using System.Diagnostics; +using System.Globalization; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Translates the BenchmarkDotNet run into the stream of test node updates the platform expects. + /// + internal sealed class BenchmarkEventProcessor : EventProcessor + { + private readonly IReadOnlyDictionary nodes; + private readonly Action publish; + private readonly Stopwatch runTimerStopwatch = new(); + private readonly Dictionary pendingResults = []; + private readonly HashSet publishedResults = []; + + public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) + { + this.nodes = nodes; + this.publish = publish; + } + + public override void OnValidationError(ValidationError validationError) + { + // If the error is not linked to a benchmark case, then set the error on all benchmarks. + var affected = validationError.BenchmarkCase == null + ? nodes.Values + : [nodes[BenchmarkTestNode.GetUid(validationError.BenchmarkCase)]]; + + foreach (var node in affected) + { + var pending = GetOrCreatePendingResult(node); + + if (validationError.IsCritical) + { + // The result is not published yet, in case there are more validation errors to append. + pending.ErrorMessages.Add(validationError.Message); + } + else + { + pending.Output.AppendLine($"WARNING: {validationError.Message}"); + } + } + } + + public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) + { + // Only build failures need to be reported, successful builds are followed by a run. + if (buildResult.IsBuildSuccess) + return; + + foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) + { + var node = nodes[BenchmarkTestNode.GetUid(benchmarkBuildInfo.BenchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + + if (buildResult.GenerateException != null) + pending.ErrorMessages.Add($"// Generate Exception: {buildResult.GenerateException.Message}"); + else if (buildResult.TryToExplainFailureReason(out string? reason)) + pending.ErrorMessages.Add($"// Build Error: {reason}"); + else if (buildResult.ErrorMessage != null) + pending.ErrorMessages.Add($"// Build Error: {buildResult.ErrorMessage}"); + + // A benchmark that failed to build will never run, so the result can be published immediately. + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark failed to build.")); + } + } + + public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) + { + var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.StartTime = DateTimeOffset.UtcNow; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + runTimerStopwatch.Restart(); + } + + public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) + { + var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.Duration = runTimerStopwatch.Elapsed; + pending.EndTime = DateTimeOffset.UtcNow; + + AppendMeasurementSummary(pending.Output, report); + + TestNodeStateProperty state = report.Success && pending.ErrorMessages.Count == 0 + ? PassedTestNodeStateProperty.CachedInstance + : new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark did not complete successfully."); + + PublishResult(node, pending, state); + } + + /// + /// Publishes a result for every benchmark that was scheduled to run but never reported one, which happens when + /// a critical validation error stopped it or when BenchmarkDotNet never reached it. + /// + public void PublishOutstandingResults() + { + foreach (var node in nodes.Values) + { + if (publishedResults.Contains(node.Uid)) + continue; + + var pending = GetOrCreatePendingResult(node); + var errorMessage = pending.GetErrorMessage(); + + TestNodeStateProperty state = errorMessage != null + ? new FailedTestNodeStateProperty(errorMessage) + : SkippedTestNodeStateProperty.CachedInstance; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, state); + } + } + + private void PublishResult(BenchmarkTestNode node, PendingResult pending, TestNodeStateProperty state) + { + var properties = new List(); + + if (pending.StartTime is { } startTime) + { + var duration = pending.Duration ?? TimeSpan.Zero; + properties.Add(new TimingProperty(new TimingInfo(startTime, pending.EndTime ?? startTime + duration, duration))); + } + + if (pending.Output.Length > 0) + properties.Add(new StandardOutputProperty(pending.Output.ToString())); + + publish(node.ToTestNode(state, properties.ToArray())); + publishedResults.Add(node.Uid); + } + + private PendingResult GetOrCreatePendingResult(BenchmarkTestNode node) + { + if (!pendingResults.TryGetValue(node.Uid, out var pending)) + { + pending = new PendingResult(); + pendingResults[node.Uid] = pending; + } + + return pending; + } + + private static void AppendMeasurementSummary(StringBuilder output, BenchmarkReport report) + { + var resultRuns = report.GetResultRuns(); + if (resultRuns.Count == 0) + return; + + output.AppendLine(report.BenchmarkCase.DisplayInfo); + output.AppendLine($"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}"); + + var statistics = resultRuns.GetStatistics(); + var cultureInfo = CultureInfo.InvariantCulture; + var formatter = statistics.CreateNanosecondFormatter(cultureInfo); + + var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); + output.AppendLine("-------------------- Histogram --------------------"); + output.AppendLine(histogram.ToString(formatter)); + output.AppendLine("---------------------------------------------------"); + output.AppendLine(statistics.ToString(cultureInfo, formatter, calcHistogram: false)); + } + + private sealed class PendingResult + { + public List ErrorMessages { get; } = []; + + public StringBuilder Output { get; } = new(); + + public DateTimeOffset? StartTime { get; set; } + + public DateTimeOffset? EndTime { get; set; } + + public TimeSpan? Duration { get; set; } + + public string? GetErrorMessage() => ErrorMessages.Count == 0 ? null : string.Join("\n", ErrorMessages); + } + } +} From 76fd7da7eb2df1991a7feab6f28dce7499fd26f8 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:15:44 +0100 Subject: [PATCH 11/68] Add BenchmarkTestFramework for test discovery/execution Introduce BenchmarkTestFramework to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmark discovery and execution. Implements session management, filtering, event processing, and output routing. Handles test node updates and cancellation, with support for experimental platform features. --- .../BenchmarkTestFramework.cs | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs new file mode 100644 index 0000000000..569bf063cb --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs @@ -0,0 +1,195 @@ +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Extensions.Messages; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.Extensions.TestFramework; +using Microsoft.Testing.Platform.Requests; +using Microsoft.Testing.Platform.Services; +using System.Reflection; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. + /// + internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IOutputDeviceDataProducer + { + private readonly BenchmarkDotNetExtension extension = new(); + private readonly IServiceProvider serviceProvider; + private readonly Assembly assembly; + + public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) + { + Capabilities = capabilities; + this.serviceProvider = serviceProvider; + this.assembly = assembly; + } + + /// + public string Uid => extension.Uid; + + /// + public string Version => extension.Version; + + /// + public string DisplayName => extension.DisplayName; + + /// + public string Description => extension.Description; + + /// + public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; + + /// + /// Gets the capabilities the framework was registered with. + /// + public ITestFrameworkCapabilities Capabilities { get; } + + /// + public Task IsEnabledAsync() => extension.IsEnabledAsync(); + + /// + public Task CreateTestSessionAsync(CreateTestSessionContext context) + => Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); + + /// + public Task CloseTestSessionAsync(CloseTestSessionContext context) + => Task.FromResult(new CloseTestSessionResult { IsSuccess = true }); + + /// + public async Task ExecuteRequestAsync(ExecuteRequestContext context) + { + try + { + switch (context.Request) + { + case DiscoverTestExecutionRequest discoverRequest: + await DiscoverAsync(discoverRequest, context).ConfigureAwait(false); + break; + case RunTestExecutionRequest runRequest: + await RunAsync(runRequest, context).ConfigureAwait(false); + break; + } + } + finally + { + context.Complete(); + } + } + + private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) + { + foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) + { + context.CancellationToken.ThrowIfCancellationRequested(); + + var message = new TestNodeUpdateMessage( + request.Session.SessionUid, + node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + + await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); + } + } + + private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) + { + var matches = GetMatchingBenchmarks(request.Filter); + if (matches.Count == 0) + return; + + var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); + var sessionUid = request.Session.SessionUid; + var cancellationToken = context.CancellationToken; + + using var workQueue = new AsyncWorkQueue(); + + var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => + { + var message = new TestNodeUpdateMessage(sessionUid, testNode); + workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); + }); + + // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, + // which keeps it in the right place when the platform runs in server mode or inside an IDE. + var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); + + var runInfos = matches + .GroupBy(match => match.RunInfo) + .Select(group => new BenchmarkRunInfo( + group.Select(match => match.Node.BenchmarkCase).ToArray(), + group.Key.Type, + group.Key.Config + .AddEventProcessor(eventProcessor) + .AddLogger(logger) + .RemoveLoggersOfType() + .CreateImmutableConfig(), + group.Key.CompositeInProcessDiagnoser)) + .ToArray(); + + // BenchmarkDotNet blocks the calling thread for the whole run, so it gets a thread of its own and the + // queued messages are published from here as they are produced. + var runTask = Task.Run( + () => + { + try + { + BenchmarkRunner.Run(runInfos, cancellationToken); + } + finally + { + // Benchmarks that never reported a result still need one, unless the run was cancelled: the + // platform expects an OperationCanceledException in that case, and publishing results + // afterwards would contradict it. + if (!cancellationToken.IsCancellationRequested) + eventProcessor.PublishOutstandingResults(); + + logger.Flush(); + workQueue.Complete(); + } + }, + CancellationToken.None); + + await workQueue.DrainAsync().ConfigureAwait(false); + await runTask.ConfigureAwait(false); + } + + /// + /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. + /// + /// The filter of the request. + /// The matching benchmarks, paired with the run info they belong to. + private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) + { + var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); + + foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) + { + // The job only earns a place in the display name when the benchmark actually runs under several jobs. + // This is computed before filtering so that a benchmark keeps the same name however it was selected. + var includeJobInName = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; + + foreach (var benchmarkCase in runInfo.BenchmarksCases) + { + var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); + if (Matches(filter, node)) + matches.Add((runInfo, node)); + } + } + + return matches; + } + +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch + { + TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), + TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), + + // NopFilter, and anything the platform adds later, means "everything". + _ => true + }; +#pragma warning restore TPEXP + } +} From aeba977c2212f6ed5e463b48156265113a9597b4 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:16:04 +0100 Subject: [PATCH 12/68] Add BenchmarkTestNode for test platform integration Introduced the internal sealed class BenchmarkTestNode to encapsulate immutable BenchmarkCase data for Microsoft.Testing.Platform integration. This includes stable UID generation, display name and path construction, property management, and support for test filtering and message bus conversion. --- .../BenchmarkTestNode.cs | 163 ++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs new file mode 100644 index 0000000000..200b22b775 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs @@ -0,0 +1,163 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Extensions.Messages; +using System.Reflection; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// The Microsoft.Testing.Platform view of a single . + /// + /// + /// A carries mutable state (the property bag holds the current outcome), so a fresh node is + /// created for every message published on the bus. This class holds the parts that never change. + /// + internal sealed class BenchmarkTestNode + { + private readonly IProperty[] staticProperties; + + private BenchmarkTestNode(BenchmarkCase benchmarkCase, string uid, string displayName, string path, IProperty[] staticProperties) + { + BenchmarkCase = benchmarkCase; + Uid = uid; + DisplayName = displayName; + Path = path; + this.staticProperties = staticProperties; + } + + /// + /// Gets the benchmark this node represents. + /// + public BenchmarkCase BenchmarkCase { get; } + + /// + /// Gets the stable identifier of the node. It has to be identical in the discovery and the execution phase, + /// which may happen in different processes. + /// + public string Uid { get; } + + /// + /// Gets the name shown by test runners. + /// + public string DisplayName { get; } + + /// + /// Gets the '/' separated path used by . + /// + public string Path { get; } + + /// + /// Gets the stable identifier of a benchmark case, without building the whole node. + /// + /// The benchmark case to identify. + /// The uid of the node representing the benchmark case. + /// + /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would + /// collide. The parameters are already part of the method name. + /// + public static string GetUid(BenchmarkCase benchmarkCase) + { + var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + return $"{fullClassName}.{FullNameProvider.GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; + } + + /// + /// Creates the node for a benchmark case. + /// + /// The benchmark case to describe. + /// + /// Whether the display name should be suffixed with the job, which is only useful when the benchmark runs + /// under more than one job. + /// + /// The created node. + public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool includeJobInName) + { + var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; + var type = benchmarkCase.Descriptor.Type; + var fullClassName = type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); + var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); + + // Unlike the uid, the job is only part of the display name when it actually adds information. + var uid = GetUid(benchmarkCase); + var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); + + var properties = new List + { + new TestMethodIdentifierProperty( + type.Assembly.FullName, + type.Namespace ?? string.Empty, + type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), + benchmarkMethod.Name, + benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, + benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), + benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), + }; + + var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); + if (benchmarkAttribute?.SourceCodeFile != null) + { + // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. + var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); + var position = new LinePosition(line, 0); + properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); + } + + foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) + properties.Add(new TestMetadataProperty("Category", category)); + + var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); + + return new BenchmarkTestNode(benchmarkCase, uid, displayName, path, properties.ToArray()); + } + + /// + /// Creates a message-bus ready node in the given state. + /// + /// The state of the benchmark, e.g. discovered, passed or failed. + /// Any additional properties, such as timing or captured output. + /// The created test node. + public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extraProperties) + { + var properties = new PropertyBag(staticProperties); + properties.Add(state); + foreach (var property in extraProperties) + properties.Add(property); + + return new TestNode + { + Uid = new TestNodeUid(Uid), + DisplayName = DisplayName, + Properties = properties + }; + } + + /// + /// Gets the properties a can match against, + /// which is what makes `--filter "/*/*/*/*[Category=Fast]"` work. + /// + /// The filterable properties. + public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); + + private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) + { + // The convention followed by the other test frameworks is ////. + var className = @namespace == null || !fullClassName.StartsWith(@namespace + ".", StringComparison.Ordinal) + ? fullClassName + : fullClassName.Substring(@namespace.Length + 1); + + return new StringBuilder() + .Append('/').Append(Escape(assembly.GetName().Name)) + .Append('/').Append(Escape(@namespace ?? string.Empty)) + .Append('/').Append(Escape(className)) + .Append('/').Append(Escape($"{methodName} [{jobDisplayInfo}]")) + .ToString(); + } + + // Benchmark parameters are stringified user values, so they can contain the path separator. + private static string Escape(string segment) => segment.Replace("/", "\\/"); + } +} From d0808862a1274c6db5ea865602b144da47f2fe46 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:16:33 +0100 Subject: [PATCH 13/68] Add OutputDeviceLogger to forward logs to output device Introduced OutputDeviceLogger class implementing ILogger to forward BenchmarkDotNet logs to the platform output device. Handles log kinds, buffers lines, and asynchronously displays output to ensure build progress and results are visible in test run output. --- .../OutputDeviceLogger.cs | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs new file mode 100644 index 0000000000..fb17503219 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs @@ -0,0 +1,77 @@ +using BenchmarkDotNet.Loggers; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.OutputDevice; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary + /// show up in the test run output. + /// + internal sealed class OutputDeviceLogger : ILogger + { + private readonly IOutputDevice outputDevice; + private readonly IOutputDeviceDataProducer producer; + private readonly AsyncWorkQueue workQueue; + private readonly CancellationToken cancellationToken; + private readonly StringBuilder currentLine = new(); + private LogKind currentLineKind = LogKind.Default; + + public OutputDeviceLogger( + IOutputDevice outputDevice, + IOutputDeviceDataProducer producer, + AsyncWorkQueue workQueue, + CancellationToken cancellationToken) + { + this.outputDevice = outputDevice; + this.producer = producer; + this.workQueue = workQueue; + this.cancellationToken = cancellationToken; + } + + public string Id => nameof(OutputDeviceLogger); + + public int Priority => 0; + + public void Write(LogKind logKind, string text) + { + currentLine.Append(text); + + // Assume that if any part of the line is an error or a warning, the whole line is. + // The kind is reset when the line is flushed. + if (logKind == LogKind.Error || (logKind == LogKind.Warning && currentLineKind != LogKind.Error)) + currentLineKind = logKind; + } + + public void WriteLine() + { + var text = currentLine.ToString(); + var kind = currentLineKind; + + currentLine.Clear(); + currentLineKind = LogKind.Default; + + IOutputDeviceData data = kind switch + { + LogKind.Error => new ErrorMessageOutputDeviceData(text), + LogKind.Warning => new WarningMessageOutputDeviceData(text), + _ => new TextOutputDeviceData(text) + }; + + workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); + } + + public void WriteLine(LogKind logKind, string text) + { + Write(logKind, text); + WriteLine(); + } + + public void Flush() + { + if (currentLine.Length > 0) + WriteLine(); + } + } +} From d155facb7801334ddd63f52a695c8cf9b12ef52a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:16:52 +0100 Subject: [PATCH 14/68] Add extension methods to register BenchmarkDotNet as test framework Introduce TestApplicationBuilderExtensions with AddBenchmarkDotNet methods for integrating BenchmarkDotNet benchmarks into Microsoft.Testing.Platform. Includes overloads for entry assembly and specific assemblies, null checks, test framework registration, and tree node filter service support. --- .../TestApplicationBuilderExtensions.cs | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs new file mode 100644 index 0000000000..0f72cb90c6 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs @@ -0,0 +1,50 @@ +using Microsoft.Testing.Platform.Builder; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Helpers; +using System.Reflection; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. + /// + public static class TestApplicationBuilderExtensions + { + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the entry assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder) + => builder.AddBenchmarkDotNet( + Assembly.GetEntryAssembly() ?? throw new InvalidOperationException( + "There is no entry assembly to look for benchmarks in. Use the overload that takes an assembly.")); + + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the given assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The assembly to look for benchmarks in. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder, Assembly assembly) + { + if (builder == null) + throw new ArgumentNullException(nameof(builder)); + if (assembly == null) + throw new ArgumentNullException(nameof(assembly)); + + builder.RegisterTestFramework( + _ => new TestFrameworkCapabilities(), + (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); + + // Opts into the tree node filter, which is what backs `--filter "/*/*/MyBenchmarks/*"`. +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + builder.AddTreeNodeFilterService(new BenchmarkDotNetExtension()); +#pragma warning restore TPEXP + + return builder; + } + } +} From 41e20991394917adfe79f392e3430c4432ef65c2 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:17:07 +0100 Subject: [PATCH 15/68] Add TestingPlatformBuilderHook for BenchmarkDotNet integration Introduced a static TestingPlatformBuilderHook class in the BenchmarkDotNet.TestAdapter.TestingPlatform namespace. This class provides an AddExtensions method to register BenchmarkDotNet with the test application builder, intended for use by generated code and hidden from IntelliSense. --- .../TestingPlatformBuilderHook.cs | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs new file mode 100644 index 0000000000..47aa1705a7 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs @@ -0,0 +1,26 @@ +using Microsoft.Testing.Platform.Builder; +using System.ComponentModel; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. + /// + /// + /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestAdapter.TestingPlatform.props. + /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called + /// directly; use + /// instead when writing an entry point by hand. + /// + [EditorBrowsable(EditorBrowsableState.Never)] + public static class TestingPlatformBuilderHook + { + /// + /// Registers BenchmarkDotNet with the test application being built. + /// + /// The builder of the test application. + /// The command line arguments of the process. Unused. + public static void AddExtensions(ITestApplicationBuilder builder, string[] arguments) + => builder.AddBenchmarkDotNet(); + } +} From e7fb11938622c092e320624492b5b36d2197a1d3 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:17:53 +0100 Subject: [PATCH 16/68] Add test runner config to global.json Added a "test" section to global.json to specify "Microsoft.Testing.Platform" as the test runner. This configures the project to use the designated testing platform. --- .../global.json | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json new file mode 100644 index 0000000000..3140116df3 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json @@ -0,0 +1,5 @@ +{ + "test": { + "runner": "Microsoft.Testing.Platform" + } +} From 9838f5788fc1cb7be43e0186053c8c8a743a1e70 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:18:18 +0100 Subject: [PATCH 17/68] Add IntegrationTests.TestingPlatform project for net10.0 Introduce BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj targeting net10.0 as an executable. The project includes assembly metadata, references BenchmarkDotNet.TestAdapter.TestingPlatform, manually imports its build props, and uses shared common.props and common.targets for build configuration. --- ...et.IntegrationTests.TestingPlatform.csproj | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj new file mode 100644 index 0000000000..b89575026a --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -0,0 +1,19 @@ + + + + net10.0 + Exe + BenchmarkDotNet.IntegrationTests.TestingPlatform + BenchmarkDotNet.IntegrationTests.TestingPlatform + BenchmarkDotNet.IntegrationTests.TestingPlatform + + + + + + + + + + + From b84f3b86781266e33b6c8b8dc20f387b59e5b9d1 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:19:08 +0100 Subject: [PATCH 18/68] Add SampleBenchmarks for fast in-process BDN testing Introduced SampleBenchmarks class in BenchmarkDotNet.IntegrationTests.TestingPlatform. Defines Add and Multiply benchmarks with parameterized Size, categorized as "Fast" and "Slow". Uses a custom FastConfig to run benchmarks in-process with a single dry iteration for quick end-to-end testing. --- .../SampleBenchmarks.cs | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs new file mode 100644 index 0000000000..27bce02ef5 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs @@ -0,0 +1,31 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// Benchmarks used to exercise the Microsoft.Testing.Platform adapter end to end. They run in-process with a + /// single iteration so that a full run stays fast. + /// + [Config(typeof(FastConfig))] + public class SampleBenchmarks + { + [Params(1, 2)] + public int Size { get; set; } + + [Benchmark] + [BenchmarkCategory("Fast")] + public int Add() => Size + Size; + + [Benchmark] + [BenchmarkCategory("Slow")] + public int Multiply() => Size * Size; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From d5d3c0b066247c4810f877a5468ceaf68e46278b Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:19:20 +0100 Subject: [PATCH 19/68] Add TestingPlatform projects and update test build configs Added BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform projects to the solution. Updated MonoBenchmarks and SharedDiagnosers integration test projects to only build for the solution in Debug configuration. --- BenchmarkDotNet.slnx | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 2d5a3cf0dd..9b62a63f29 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -20,6 +20,7 @@ + @@ -39,9 +40,14 @@ - - + + + + + + + From f420e45161888411469e72303328aad06fafef35 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 12 Aug 2026 18:33:56 +0100 Subject: [PATCH 20/68] Update slnx to exclude two projects from Debug build Explicitly set BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform to not build in the Debug configuration by adding in BenchmarkDotNet.slnx. No other changes made. --- BenchmarkDotNet.slnx | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 9b62a63f29..119a22f6ca 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -20,7 +20,9 @@ - + + + @@ -47,7 +49,9 @@ - + + + From 61ce3f14609829aad418aad65801374ccafa2a5f Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:12:00 +0100 Subject: [PATCH 21/68] Update docs, namespaces, and add GetBenchmarkUid method - Correct NuGet package and namespace in documentation - Add GetBenchmarkUid for stable benchmark identification - Change namespace in BenchmarkCaseIdentityExtensions - Update InternalsVisibleTo for TestingPlatform assembly --- docs/articles/features/testingplatform.md | 6 +++--- .../Exporters/FullNameProvider.cs | 18 ++++++++++++++++++ .../BenchmarkCaseIdentityExtensions.cs | 6 +++--- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 2 +- 4 files changed, 25 insertions(+), 7 deletions(-) rename src/{BenchmarkDotNet.TestAdapter => BenchmarkDotNet/Extensions}/BenchmarkCaseIdentityExtensions.cs (88%) diff --git a/docs/articles/features/testingplatform.md b/docs/articles/features/testingplatform.md index 85fffeeb0b..a54ccab78e 100644 --- a/docs/articles/features/testingplatform.md +++ b/docs/articles/features/testingplatform.md @@ -53,7 +53,7 @@ The practical consequences of the MTP model are: ```xml - + ``` @@ -72,7 +72,7 @@ The practical consequences of the MTP model are: - + @@ -149,7 +149,7 @@ If you want to keep your own entry point, turn off the generated one and registe ``` ```csharp -using BenchmarkDotNet.TestAdapter.TestingPlatform; +using BenchmarkDotNet.TestingPlatform; using Microsoft.Testing.Platform.Builder; public static class Program diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 19555908d2..922d3590c4 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -63,6 +63,24 @@ public static string GetBenchmarkName(BenchmarkCase benchmarkCase) return name.ToString(); } + /// + /// Gets an identifier of a benchmark case that stays the same across processes, which is what lets a benchmark + /// discovered in one process be selected for execution in another (for example by a test adapter). + /// + /// The benchmark case to identify. + /// The unique identifier of the benchmark case. + /// + /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would + /// collide. The parameters are already part of the method name. + /// + [PublicAPI] + public static string GetBenchmarkUid(BenchmarkCase benchmarkCase) + { + var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + return $"{fullClassName}.{GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; + } + + private static string GetNestedTypes(Type type) { string nestedTypes = ""; diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs b/src/BenchmarkDotNet/Extensions/BenchmarkCaseIdentityExtensions.cs similarity index 88% rename from src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs rename to src/BenchmarkDotNet/Extensions/BenchmarkCaseIdentityExtensions.cs index 78cb64f5ce..948ea1d7fe 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs +++ b/src/BenchmarkDotNet/Extensions/BenchmarkCaseIdentityExtensions.cs @@ -1,11 +1,10 @@ using BenchmarkDotNet.Characteristics; using BenchmarkDotNet.Running; -namespace BenchmarkDotNet.TestAdapter +namespace BenchmarkDotNet.Extensions { /// - /// Helpers for deriving stable identities for a BenchmarkCase. Shared by the VSTest and the - /// Microsoft.Testing.Platform adapters, because both need identities that survive across processes. + /// Helpers for deriving stable identities for a BenchmarkCase. /// internal static class BenchmarkCaseIdentityExtensions { @@ -30,3 +29,4 @@ internal static string GetUnrandomizedJobDisplayInfo(this BenchmarkCase benchmar } } } + diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index 5a5ff709d3..7bd16f9307 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -15,4 +15,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 4a27390fdb7aa93db4710dd3e112dbb7d0caf3b6 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:12:32 +0100 Subject: [PATCH 22/68] Remove BenchmarkDotNet.TestAdapter.TestingPlatform project Deleted all source, project, and props files from BenchmarkDotNet.TestAdapter.TestingPlatform. This removes all implementation and integration for running benchmarks as tests via Microsoft.Testing.Platform, including test discovery, execution, and result processing logic. --- .../AsyncWorkQueue.cs | 61 ------ ...kDotNet.TestAdapter.TestingPlatform.csproj | 46 ----- .../BenchmarkDotNetExtension.cs | 34 --- .../BenchmarkEventProcessor.cs | 191 ----------------- .../BenchmarkTestFramework.cs | 195 ------------------ .../BenchmarkTestNode.cs | 163 --------------- .../OutputDeviceLogger.cs | 77 ------- .../TestApplicationBuilderExtensions.cs | 50 ----- .../TestingPlatformBuilderHook.cs | 26 --- ...rkDotNet.TestAdapter.TestingPlatform.props | 31 --- 10 files changed, 874 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs deleted file mode 100644 index 128eb51b0c..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs +++ /dev/null @@ -1,61 +0,0 @@ -using System.Collections.Concurrent; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. - /// - /// - /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the - /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks - /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks - /// enqueue instead and the caller of DrainAsync does the awaiting. - /// - internal sealed class AsyncWorkQueue : IDisposable - { - private readonly ConcurrentQueue> queue = new(); - private readonly SemaphoreSlim available = new(0); - private volatile bool completed; - - /// - /// Queues a work item. Safe to call from any thread. - /// - /// The work to perform. - public void Enqueue(Func work) - { - queue.Enqueue(work); - available.Release(); - } - - /// - /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued - /// items have been processed. - /// - public void Complete() - { - completed = true; - available.Release(); - } - - /// - /// Processes queued work items in order until has been called and the queue is empty. - /// - /// A task that completes when the queue has been drained. - public async Task DrainAsync() - { - while (true) - { - await available.WaitAsync().ConfigureAwait(false); - - // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. - while (queue.TryDequeue(out var work)) - await work().ConfigureAwait(false); - - if (completed && queue.IsEmpty) - return; - } - } - - public void Dispose() => available.Dispose(); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj deleted file mode 100644 index c3dd5c4cb3..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj +++ /dev/null @@ -1,46 +0,0 @@ - - - - netstandard2.0 - BenchmarkDotNet.TestAdapter.TestingPlatform - BenchmarkDotNet.TestAdapter.TestingPlatform - BenchmarkDotNet.TestAdapter.TestingPlatform - Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform - README.md - True - BenchmarkDotNet.TestAdapter.TestingPlatform - - - false - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs deleted file mode 100644 index 789fbc611a..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs +++ /dev/null @@ -1,34 +0,0 @@ -using Microsoft.Testing.Platform.Extensions; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. - /// - /// - /// Several platform services, such as the tree node filter behind --filter, are registered on behalf of an - /// extension rather than of the test framework itself, so the identity lives in its own type. - /// - internal sealed class BenchmarkDotNetExtension : IExtension - { - /// - /// The uid shared by every extension this package registers. - /// - public const string ExtensionUid = "BenchmarkDotNet.TestAdapter.TestingPlatform"; - - /// - public string Uid => ExtensionUid; - - /// - public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; - - /// - public string DisplayName => "BenchmarkDotNet"; - - /// - public string Description => "Runs BenchmarkDotNet benchmarks as tests."; - - /// - public Task IsEnabledAsync() => Task.FromResult(true); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs deleted file mode 100644 index b48e5cd6e6..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs +++ /dev/null @@ -1,191 +0,0 @@ -using BenchmarkDotNet.EventProcessors; -using BenchmarkDotNet.Extensions; -using BenchmarkDotNet.Reports; -using BenchmarkDotNet.Running; -using BenchmarkDotNet.Toolchains.Results; -using BenchmarkDotNet.Validators; -using Microsoft.Testing.Platform.Extensions.Messages; -using Perfolizer.Mathematics.Histograms; -using System.Diagnostics; -using System.Globalization; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Translates the BenchmarkDotNet run into the stream of test node updates the platform expects. - /// - internal sealed class BenchmarkEventProcessor : EventProcessor - { - private readonly IReadOnlyDictionary nodes; - private readonly Action publish; - private readonly Stopwatch runTimerStopwatch = new(); - private readonly Dictionary pendingResults = []; - private readonly HashSet publishedResults = []; - - public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) - { - this.nodes = nodes; - this.publish = publish; - } - - public override void OnValidationError(ValidationError validationError) - { - // If the error is not linked to a benchmark case, then set the error on all benchmarks. - var affected = validationError.BenchmarkCase == null - ? nodes.Values - : [nodes[BenchmarkTestNode.GetUid(validationError.BenchmarkCase)]]; - - foreach (var node in affected) - { - var pending = GetOrCreatePendingResult(node); - - if (validationError.IsCritical) - { - // The result is not published yet, in case there are more validation errors to append. - pending.ErrorMessages.Add(validationError.Message); - } - else - { - pending.Output.AppendLine($"WARNING: {validationError.Message}"); - } - } - } - - public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) - { - // Only build failures need to be reported, successful builds are followed by a run. - if (buildResult.IsBuildSuccess) - return; - - foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) - { - var node = nodes[BenchmarkTestNode.GetUid(benchmarkBuildInfo.BenchmarkCase)]; - var pending = GetOrCreatePendingResult(node); - - if (buildResult.GenerateException != null) - pending.ErrorMessages.Add($"// Generate Exception: {buildResult.GenerateException.Message}"); - else if (buildResult.TryToExplainFailureReason(out string? reason)) - pending.ErrorMessages.Add($"// Build Error: {reason}"); - else if (buildResult.ErrorMessage != null) - pending.ErrorMessages.Add($"// Build Error: {buildResult.ErrorMessage}"); - - // A benchmark that failed to build will never run, so the result can be published immediately. - publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); - PublishResult(node, pending, new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark failed to build.")); - } - } - - public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) - { - var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; - var pending = GetOrCreatePendingResult(node); - pending.StartTime = DateTimeOffset.UtcNow; - - publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); - runTimerStopwatch.Restart(); - } - - public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) - { - var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; - var pending = GetOrCreatePendingResult(node); - pending.Duration = runTimerStopwatch.Elapsed; - pending.EndTime = DateTimeOffset.UtcNow; - - AppendMeasurementSummary(pending.Output, report); - - TestNodeStateProperty state = report.Success && pending.ErrorMessages.Count == 0 - ? PassedTestNodeStateProperty.CachedInstance - : new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark did not complete successfully."); - - PublishResult(node, pending, state); - } - - /// - /// Publishes a result for every benchmark that was scheduled to run but never reported one, which happens when - /// a critical validation error stopped it or when BenchmarkDotNet never reached it. - /// - public void PublishOutstandingResults() - { - foreach (var node in nodes.Values) - { - if (publishedResults.Contains(node.Uid)) - continue; - - var pending = GetOrCreatePendingResult(node); - var errorMessage = pending.GetErrorMessage(); - - TestNodeStateProperty state = errorMessage != null - ? new FailedTestNodeStateProperty(errorMessage) - : SkippedTestNodeStateProperty.CachedInstance; - - publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); - PublishResult(node, pending, state); - } - } - - private void PublishResult(BenchmarkTestNode node, PendingResult pending, TestNodeStateProperty state) - { - var properties = new List(); - - if (pending.StartTime is { } startTime) - { - var duration = pending.Duration ?? TimeSpan.Zero; - properties.Add(new TimingProperty(new TimingInfo(startTime, pending.EndTime ?? startTime + duration, duration))); - } - - if (pending.Output.Length > 0) - properties.Add(new StandardOutputProperty(pending.Output.ToString())); - - publish(node.ToTestNode(state, properties.ToArray())); - publishedResults.Add(node.Uid); - } - - private PendingResult GetOrCreatePendingResult(BenchmarkTestNode node) - { - if (!pendingResults.TryGetValue(node.Uid, out var pending)) - { - pending = new PendingResult(); - pendingResults[node.Uid] = pending; - } - - return pending; - } - - private static void AppendMeasurementSummary(StringBuilder output, BenchmarkReport report) - { - var resultRuns = report.GetResultRuns(); - if (resultRuns.Count == 0) - return; - - output.AppendLine(report.BenchmarkCase.DisplayInfo); - output.AppendLine($"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}"); - - var statistics = resultRuns.GetStatistics(); - var cultureInfo = CultureInfo.InvariantCulture; - var formatter = statistics.CreateNanosecondFormatter(cultureInfo); - - var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); - output.AppendLine("-------------------- Histogram --------------------"); - output.AppendLine(histogram.ToString(formatter)); - output.AppendLine("---------------------------------------------------"); - output.AppendLine(statistics.ToString(cultureInfo, formatter, calcHistogram: false)); - } - - private sealed class PendingResult - { - public List ErrorMessages { get; } = []; - - public StringBuilder Output { get; } = new(); - - public DateTimeOffset? StartTime { get; set; } - - public DateTimeOffset? EndTime { get; set; } - - public TimeSpan? Duration { get; set; } - - public string? GetErrorMessage() => ErrorMessages.Count == 0 ? null : string.Join("\n", ErrorMessages); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs deleted file mode 100644 index 569bf063cb..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs +++ /dev/null @@ -1,195 +0,0 @@ -using BenchmarkDotNet.Configs; -using BenchmarkDotNet.Loggers; -using BenchmarkDotNet.Running; -using Microsoft.Testing.Platform.Capabilities.TestFramework; -using Microsoft.Testing.Platform.Extensions.Messages; -using Microsoft.Testing.Platform.Extensions.OutputDevice; -using Microsoft.Testing.Platform.Extensions.TestFramework; -using Microsoft.Testing.Platform.Requests; -using Microsoft.Testing.Platform.Services; -using System.Reflection; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. - /// - internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IOutputDeviceDataProducer - { - private readonly BenchmarkDotNetExtension extension = new(); - private readonly IServiceProvider serviceProvider; - private readonly Assembly assembly; - - public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) - { - Capabilities = capabilities; - this.serviceProvider = serviceProvider; - this.assembly = assembly; - } - - /// - public string Uid => extension.Uid; - - /// - public string Version => extension.Version; - - /// - public string DisplayName => extension.DisplayName; - - /// - public string Description => extension.Description; - - /// - public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; - - /// - /// Gets the capabilities the framework was registered with. - /// - public ITestFrameworkCapabilities Capabilities { get; } - - /// - public Task IsEnabledAsync() => extension.IsEnabledAsync(); - - /// - public Task CreateTestSessionAsync(CreateTestSessionContext context) - => Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); - - /// - public Task CloseTestSessionAsync(CloseTestSessionContext context) - => Task.FromResult(new CloseTestSessionResult { IsSuccess = true }); - - /// - public async Task ExecuteRequestAsync(ExecuteRequestContext context) - { - try - { - switch (context.Request) - { - case DiscoverTestExecutionRequest discoverRequest: - await DiscoverAsync(discoverRequest, context).ConfigureAwait(false); - break; - case RunTestExecutionRequest runRequest: - await RunAsync(runRequest, context).ConfigureAwait(false); - break; - } - } - finally - { - context.Complete(); - } - } - - private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) - { - foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) - { - context.CancellationToken.ThrowIfCancellationRequested(); - - var message = new TestNodeUpdateMessage( - request.Session.SessionUid, - node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); - - await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); - } - } - - private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) - { - var matches = GetMatchingBenchmarks(request.Filter); - if (matches.Count == 0) - return; - - var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); - var sessionUid = request.Session.SessionUid; - var cancellationToken = context.CancellationToken; - - using var workQueue = new AsyncWorkQueue(); - - var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => - { - var message = new TestNodeUpdateMessage(sessionUid, testNode); - workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); - }); - - // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, - // which keeps it in the right place when the platform runs in server mode or inside an IDE. - var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); - - var runInfos = matches - .GroupBy(match => match.RunInfo) - .Select(group => new BenchmarkRunInfo( - group.Select(match => match.Node.BenchmarkCase).ToArray(), - group.Key.Type, - group.Key.Config - .AddEventProcessor(eventProcessor) - .AddLogger(logger) - .RemoveLoggersOfType() - .CreateImmutableConfig(), - group.Key.CompositeInProcessDiagnoser)) - .ToArray(); - - // BenchmarkDotNet blocks the calling thread for the whole run, so it gets a thread of its own and the - // queued messages are published from here as they are produced. - var runTask = Task.Run( - () => - { - try - { - BenchmarkRunner.Run(runInfos, cancellationToken); - } - finally - { - // Benchmarks that never reported a result still need one, unless the run was cancelled: the - // platform expects an OperationCanceledException in that case, and publishing results - // afterwards would contradict it. - if (!cancellationToken.IsCancellationRequested) - eventProcessor.PublishOutstandingResults(); - - logger.Flush(); - workQueue.Complete(); - } - }, - CancellationToken.None); - - await workQueue.DrainAsync().ConfigureAwait(false); - await runTask.ConfigureAwait(false); - } - - /// - /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. - /// - /// The filter of the request. - /// The matching benchmarks, paired with the run info they belong to. - private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) - { - var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); - - foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) - { - // The job only earns a place in the display name when the benchmark actually runs under several jobs. - // This is computed before filtering so that a benchmark keeps the same name however it was selected. - var includeJobInName = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; - - foreach (var benchmarkCase in runInfo.BenchmarksCases) - { - var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); - if (Matches(filter, node)) - matches.Add((runInfo, node)); - } - } - - return matches; - } - -#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. - private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch - { - TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), - TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), - - // NopFilter, and anything the platform adds later, means "everything". - _ => true - }; -#pragma warning restore TPEXP - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs deleted file mode 100644 index 200b22b775..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs +++ /dev/null @@ -1,163 +0,0 @@ -using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Exporters; -using BenchmarkDotNet.Extensions; -using BenchmarkDotNet.Running; -using Microsoft.Testing.Platform.Extensions.Messages; -using System.Reflection; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// The Microsoft.Testing.Platform view of a single . - /// - /// - /// A carries mutable state (the property bag holds the current outcome), so a fresh node is - /// created for every message published on the bus. This class holds the parts that never change. - /// - internal sealed class BenchmarkTestNode - { - private readonly IProperty[] staticProperties; - - private BenchmarkTestNode(BenchmarkCase benchmarkCase, string uid, string displayName, string path, IProperty[] staticProperties) - { - BenchmarkCase = benchmarkCase; - Uid = uid; - DisplayName = displayName; - Path = path; - this.staticProperties = staticProperties; - } - - /// - /// Gets the benchmark this node represents. - /// - public BenchmarkCase BenchmarkCase { get; } - - /// - /// Gets the stable identifier of the node. It has to be identical in the discovery and the execution phase, - /// which may happen in different processes. - /// - public string Uid { get; } - - /// - /// Gets the name shown by test runners. - /// - public string DisplayName { get; } - - /// - /// Gets the '/' separated path used by . - /// - public string Path { get; } - - /// - /// Gets the stable identifier of a benchmark case, without building the whole node. - /// - /// The benchmark case to identify. - /// The uid of the node representing the benchmark case. - /// - /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would - /// collide. The parameters are already part of the method name. - /// - public static string GetUid(BenchmarkCase benchmarkCase) - { - var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - return $"{fullClassName}.{FullNameProvider.GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; - } - - /// - /// Creates the node for a benchmark case. - /// - /// The benchmark case to describe. - /// - /// Whether the display name should be suffixed with the job, which is only useful when the benchmark runs - /// under more than one job. - /// - /// The created node. - public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool includeJobInName) - { - var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; - var type = benchmarkCase.Descriptor.Type; - var fullClassName = type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); - var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); - - // Unlike the uid, the job is only part of the display name when it actually adds information. - var uid = GetUid(benchmarkCase); - var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); - - var properties = new List - { - new TestMethodIdentifierProperty( - type.Assembly.FullName, - type.Namespace ?? string.Empty, - type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), - benchmarkMethod.Name, - benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, - benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), - benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), - }; - - var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); - if (benchmarkAttribute?.SourceCodeFile != null) - { - // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. - var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); - var position = new LinePosition(line, 0); - properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); - } - - foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) - properties.Add(new TestMetadataProperty("Category", category)); - - var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); - - return new BenchmarkTestNode(benchmarkCase, uid, displayName, path, properties.ToArray()); - } - - /// - /// Creates a message-bus ready node in the given state. - /// - /// The state of the benchmark, e.g. discovered, passed or failed. - /// Any additional properties, such as timing or captured output. - /// The created test node. - public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extraProperties) - { - var properties = new PropertyBag(staticProperties); - properties.Add(state); - foreach (var property in extraProperties) - properties.Add(property); - - return new TestNode - { - Uid = new TestNodeUid(Uid), - DisplayName = DisplayName, - Properties = properties - }; - } - - /// - /// Gets the properties a can match against, - /// which is what makes `--filter "/*/*/*/*[Category=Fast]"` work. - /// - /// The filterable properties. - public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); - - private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) - { - // The convention followed by the other test frameworks is ////. - var className = @namespace == null || !fullClassName.StartsWith(@namespace + ".", StringComparison.Ordinal) - ? fullClassName - : fullClassName.Substring(@namespace.Length + 1); - - return new StringBuilder() - .Append('/').Append(Escape(assembly.GetName().Name)) - .Append('/').Append(Escape(@namespace ?? string.Empty)) - .Append('/').Append(Escape(className)) - .Append('/').Append(Escape($"{methodName} [{jobDisplayInfo}]")) - .ToString(); - } - - // Benchmark parameters are stringified user values, so they can contain the path separator. - private static string Escape(string segment) => segment.Replace("/", "\\/"); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs deleted file mode 100644 index fb17503219..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs +++ /dev/null @@ -1,77 +0,0 @@ -using BenchmarkDotNet.Loggers; -using Microsoft.Testing.Platform.Extensions.OutputDevice; -using Microsoft.Testing.Platform.OutputDevice; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary - /// show up in the test run output. - /// - internal sealed class OutputDeviceLogger : ILogger - { - private readonly IOutputDevice outputDevice; - private readonly IOutputDeviceDataProducer producer; - private readonly AsyncWorkQueue workQueue; - private readonly CancellationToken cancellationToken; - private readonly StringBuilder currentLine = new(); - private LogKind currentLineKind = LogKind.Default; - - public OutputDeviceLogger( - IOutputDevice outputDevice, - IOutputDeviceDataProducer producer, - AsyncWorkQueue workQueue, - CancellationToken cancellationToken) - { - this.outputDevice = outputDevice; - this.producer = producer; - this.workQueue = workQueue; - this.cancellationToken = cancellationToken; - } - - public string Id => nameof(OutputDeviceLogger); - - public int Priority => 0; - - public void Write(LogKind logKind, string text) - { - currentLine.Append(text); - - // Assume that if any part of the line is an error or a warning, the whole line is. - // The kind is reset when the line is flushed. - if (logKind == LogKind.Error || (logKind == LogKind.Warning && currentLineKind != LogKind.Error)) - currentLineKind = logKind; - } - - public void WriteLine() - { - var text = currentLine.ToString(); - var kind = currentLineKind; - - currentLine.Clear(); - currentLineKind = LogKind.Default; - - IOutputDeviceData data = kind switch - { - LogKind.Error => new ErrorMessageOutputDeviceData(text), - LogKind.Warning => new WarningMessageOutputDeviceData(text), - _ => new TextOutputDeviceData(text) - }; - - workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); - } - - public void WriteLine(LogKind logKind, string text) - { - Write(logKind, text); - WriteLine(); - } - - public void Flush() - { - if (currentLine.Length > 0) - WriteLine(); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs deleted file mode 100644 index 0f72cb90c6..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs +++ /dev/null @@ -1,50 +0,0 @@ -using Microsoft.Testing.Platform.Builder; -using Microsoft.Testing.Platform.Capabilities.TestFramework; -using Microsoft.Testing.Platform.Helpers; -using System.Reflection; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. - /// - public static class TestApplicationBuilderExtensions - { - /// - /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the entry assembly are exposed - /// as tests. - /// - /// The builder of the test application. - /// The same builder, so that calls can be chained. - public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder) - => builder.AddBenchmarkDotNet( - Assembly.GetEntryAssembly() ?? throw new InvalidOperationException( - "There is no entry assembly to look for benchmarks in. Use the overload that takes an assembly.")); - - /// - /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the given assembly are exposed - /// as tests. - /// - /// The builder of the test application. - /// The assembly to look for benchmarks in. - /// The same builder, so that calls can be chained. - public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder, Assembly assembly) - { - if (builder == null) - throw new ArgumentNullException(nameof(builder)); - if (assembly == null) - throw new ArgumentNullException(nameof(assembly)); - - builder.RegisterTestFramework( - _ => new TestFrameworkCapabilities(), - (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); - - // Opts into the tree node filter, which is what backs `--filter "/*/*/MyBenchmarks/*"`. -#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. - builder.AddTreeNodeFilterService(new BenchmarkDotNetExtension()); -#pragma warning restore TPEXP - - return builder; - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs deleted file mode 100644 index 47aa1705a7..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs +++ /dev/null @@ -1,26 +0,0 @@ -using Microsoft.Testing.Platform.Builder; -using System.ComponentModel; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. - /// - /// - /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestAdapter.TestingPlatform.props. - /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called - /// directly; use - /// instead when writing an entry point by hand. - /// - [EditorBrowsable(EditorBrowsableState.Never)] - public static class TestingPlatformBuilderHook - { - /// - /// Registers BenchmarkDotNet with the test application being built. - /// - /// The builder of the test application. - /// The command line arguments of the process. Unused. - public static void AddExtensions(ITestApplicationBuilder builder, string[] arguments) - => builder.AddBenchmarkDotNet(); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props deleted file mode 100644 index 7b6f563554..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props +++ /dev/null @@ -1,31 +0,0 @@ - - - - true - - - true - - - false - - - - - - BenchmarkDotNet - BenchmarkDotNet.TestAdapter.TestingPlatform.TestingPlatformBuilderHook - - - From e54444463aeb8e5acc9cd280dabb5d4f537f5cff Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:13:09 +0100 Subject: [PATCH 23/68] Integrate BenchmarkDotNet with Microsoft.Testing.Platform Add BenchmarkDotNet.TestingPlatform.props to enable seamless integration with Microsoft.Testing.Platform. This includes setting required properties for Testing Platform application behavior, ensuring `dotnet test` compatibility on older SDKs, disabling parallel test execution for multi-targeted projects by default, and registering BenchmarkDotNet as a builder hook. --- .../BenchmarkDotNet.TestingPlatform.props | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props diff --git a/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props b/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props new file mode 100644 index 0000000000..5f1a9412ef --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props @@ -0,0 +1,31 @@ + + + + true + + + true + + + false + + + + + + BenchmarkDotNet + BenchmarkDotNet.TestingPlatform.TestingPlatformBuilderHook + + + From 7da9979d012a49636079956884b442e961f53b5a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:14:45 +0100 Subject: [PATCH 24/68] Add AsyncWorkQueue for ordered async work processing Introduced AsyncWorkQueue in BenchmarkDotNet.TestingPlatform to enable thread-safe, ordered queuing of asynchronous work items. Supports synchronous enqueuing, asynchronous draining, completion signaling, and resource disposal using ConcurrentQueue and SemaphoreSlim. --- .../AsyncWorkQueue.cs | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs new file mode 100644 index 0000000000..89818c96c2 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs @@ -0,0 +1,61 @@ +using System.Collections.Concurrent; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. + /// + /// + /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the + /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks + /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks + /// enqueue instead and the caller of DrainAsync does the awaiting. + /// + internal sealed class AsyncWorkQueue : IDisposable + { + private readonly ConcurrentQueue> queue = new(); + private readonly SemaphoreSlim available = new(0); + private volatile bool completed; + + /// + /// Queues a work item. Safe to call from any thread. + /// + /// The work to perform. + public void Enqueue(Func work) + { + queue.Enqueue(work); + available.Release(); + } + + /// + /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued + /// items have been processed. + /// + public void Complete() + { + completed = true; + available.Release(); + } + + /// + /// Processes queued work items in order until has been called and the queue is empty. + /// + /// A task that completes when the queue has been drained. + public async Task DrainAsync() + { + while (true) + { + await available.WaitAsync().ConfigureAwait(false); + + // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. + while (queue.TryDequeue(out var work)) + await work().ConfigureAwait(false); + + if (completed && queue.IsEmpty) + return; + } + } + + public void Dispose() => available.Dispose(); + } +} From 2d945bf0b6c64ed56f38d8cfcb43c5e6f8b9991c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:18:07 +0100 Subject: [PATCH 25/68] Add BenchmarkDotNet.TestingPlatform integration Introduce a new project to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmarks to be discovered and executed as tests. Implements extension identification, test framework, event processing, test node representation, and output logging. Provides builder extensions for easy registration and an MSBuild hook for automatic integration. Updates project configuration for packaging and dependencies. --- .../BenchmarkDotNet.TestingPlatform.csproj | 46 ++++ .../BenchmarkDotNetExtension.cs | 34 +++ .../BenchmarkEventProcessor.cs | 193 +++++++++++++++++ .../BenchmarkTestFramework.cs | 197 ++++++++++++++++++ .../BenchmarkTestNode.cs | 148 +++++++++++++ .../OutputDeviceLogger.cs | 77 +++++++ .../TestApplicationBuilderExtensions.cs | 50 +++++ .../TestingPlatformBuilderHook.cs | 26 +++ 8 files changed, 771 insertions(+) create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj new file mode 100644 index 0000000000..a97893d407 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj @@ -0,0 +1,46 @@ + + + + netstandard2.0 + BenchmarkDotNet.TestingPlatform + BenchmarkDotNet.TestingPlatform + BenchmarkDotNet.TestingPlatform + Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform + README.md + True + BenchmarkDotNet.TestingPlatform + + + false + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs new file mode 100644 index 0000000000..d2bc8722d8 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs @@ -0,0 +1,34 @@ +using Microsoft.Testing.Platform.Extensions; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. + /// + /// + /// Several platform services, such as the tree node filter behind --filter, are registered on behalf of an + /// extension rather than of the test framework itself, so the identity lives in its own type. + /// + internal sealed class BenchmarkDotNetExtension : IExtension + { + /// + /// The uid shared by every extension this package registers. + /// + public const string ExtensionUid = "BenchmarkDotNet.TestingPlatform"; + + /// + public string Uid => ExtensionUid; + + /// + public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; + + /// + public string DisplayName => "BenchmarkDotNet"; + + /// + public string Description => "Runs BenchmarkDotNet benchmarks as tests."; + + /// + public Task IsEnabledAsync() => Task.FromResult(true); + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs new file mode 100644 index 0000000000..a7d50256da --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs @@ -0,0 +1,193 @@ +using BenchmarkDotNet.EventProcessors; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Reports; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains.Results; +using BenchmarkDotNet.Validators; +using Microsoft.Testing.Platform.Extensions.Messages; +using Perfolizer.Mathematics.Histograms; +using System.Diagnostics; +using System.Globalization; +using System.Text; + +namespace BenchmarkDotNet.TestingPlatform +{ + + /// + /// Translates the BenchmarkDotNet run into the stream of test node updates the platform expects. + /// + internal sealed class BenchmarkEventProcessor : EventProcessor + { + private readonly IReadOnlyDictionary nodes; + private readonly Action publish; + private readonly Stopwatch runTimerStopwatch = new(); + private readonly Dictionary pendingResults = []; + private readonly HashSet publishedResults = []; + + public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) + { + this.nodes = nodes; + this.publish = publish; + } + + public override void OnValidationError(ValidationError validationError) + { + // If the error is not linked to a benchmark case, then set the error on all benchmarks. + var affected = validationError.BenchmarkCase == null + ? nodes.Values + : [nodes[FullNameProvider.GetBenchmarkUid(validationError.BenchmarkCase)]]; + + foreach (var node in affected) + { + var pending = GetOrCreatePendingResult(node); + + if (validationError.IsCritical) + { + // The result is not published yet, in case there are more validation errors to append. + pending.ErrorMessages.Add(validationError.Message); + } + else + { + pending.Output.AppendLine($"WARNING: {validationError.Message}"); + } + } + } + + public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) + { + // Only build failures need to be reported, successful builds are followed by a run. + if (buildResult.IsBuildSuccess) + return; + + foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) + { + var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkBuildInfo.BenchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + + if (buildResult.GenerateException != null) + pending.ErrorMessages.Add($"// Generate Exception: {buildResult.GenerateException.Message}"); + else if (buildResult.TryToExplainFailureReason(buildPartition.GetInProcessDiagnoserHandlerTypes(), out string? reason)) + pending.ErrorMessages.Add($"// Build Error: {reason}"); + else if (buildResult.ErrorMessage != null) + pending.ErrorMessages.Add($"// Build Error: {buildResult.ErrorMessage}"); + + // A benchmark that failed to build will never run, so the result can be published immediately. + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark failed to build.")); + } + } + + public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) + { + var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.StartTime = DateTimeOffset.UtcNow; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + runTimerStopwatch.Restart(); + } + + public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) + { + var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.Duration = runTimerStopwatch.Elapsed; + pending.EndTime = DateTimeOffset.UtcNow; + + AppendMeasurementSummary(pending.Output, report); + + TestNodeStateProperty state = report.Success && pending.ErrorMessages.Count == 0 + ? PassedTestNodeStateProperty.CachedInstance + : new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark did not complete successfully."); + + PublishResult(node, pending, state); + } + + /// + /// Publishes a result for every benchmark that was scheduled to run but never reported one, which happens when + /// a critical validation error stopped it or when BenchmarkDotNet never reached it. + /// + public void PublishOutstandingResults() + { + foreach (var node in nodes.Values) + { + if (publishedResults.Contains(node.Uid)) + continue; + + var pending = GetOrCreatePendingResult(node); + var errorMessage = pending.GetErrorMessage(); + + TestNodeStateProperty state = errorMessage != null + ? new FailedTestNodeStateProperty(errorMessage) + : SkippedTestNodeStateProperty.CachedInstance; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, state); + } + } + + private void PublishResult(BenchmarkTestNode node, PendingResult pending, TestNodeStateProperty state) + { + var properties = new List(); + + if (pending.StartTime is { } startTime) + { + var duration = pending.Duration ?? TimeSpan.Zero; + properties.Add(new TimingProperty(new TimingInfo(startTime, pending.EndTime ?? startTime + duration, duration))); + } + + if (pending.Output.Length > 0) + properties.Add(new StandardOutputProperty(pending.Output.ToString())); + + publish(node.ToTestNode(state, properties.ToArray())); + publishedResults.Add(node.Uid); + } + + private PendingResult GetOrCreatePendingResult(BenchmarkTestNode node) + { + if (!pendingResults.TryGetValue(node.Uid, out var pending)) + { + pending = new PendingResult(); + pendingResults[node.Uid] = pending; + } + + return pending; + } + + private static void AppendMeasurementSummary(StringBuilder output, BenchmarkReport report) + { + var resultRuns = report.GetResultRuns(); + if (resultRuns.Count == 0) + return; + + output.AppendLine(report.BenchmarkCase.DisplayInfo); + output.AppendLine($"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}"); + + var statistics = resultRuns.GetStatistics(); + var cultureInfo = CultureInfo.InvariantCulture; + var formatter = statistics.CreateNanosecondFormatter(cultureInfo); + + var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); + output.AppendLine("-------------------- Histogram --------------------"); + output.AppendLine(histogram.ToString(formatter)); + output.AppendLine("---------------------------------------------------"); + output.AppendLine(statistics.ToString(cultureInfo, formatter, calcHistogram: false)); + } + + private sealed class PendingResult + { + public List ErrorMessages { get; } = []; + + public StringBuilder Output { get; } = new(); + + public DateTimeOffset? StartTime { get; set; } + + public DateTimeOffset? EndTime { get; set; } + + public TimeSpan? Duration { get; set; } + + public string? GetErrorMessage() => ErrorMessages.Count == 0 ? null : string.Join("\n", ErrorMessages); + } + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs new file mode 100644 index 0000000000..48fcfa8d72 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs @@ -0,0 +1,197 @@ +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +// BenchmarkEnumerator is compiled into this assembly from the VSTest adapter, where it keeps its own namespace. +using BenchmarkDotNet.TestAdapter; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Extensions.Messages; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.Extensions.TestFramework; +using Microsoft.Testing.Platform.Requests; +using Microsoft.Testing.Platform.Services; +using System.Reflection; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. + /// + internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IOutputDeviceDataProducer + { + private readonly BenchmarkDotNetExtension extension = new(); + private readonly IServiceProvider serviceProvider; + private readonly Assembly assembly; + + public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) + { + Capabilities = capabilities; + this.serviceProvider = serviceProvider; + this.assembly = assembly; + } + + /// + public string Uid => extension.Uid; + + /// + public string Version => extension.Version; + + /// + public string DisplayName => extension.DisplayName; + + /// + public string Description => extension.Description; + + /// + public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; + + /// + /// Gets the capabilities the framework was registered with. + /// + public ITestFrameworkCapabilities Capabilities { get; } + + /// + public Task IsEnabledAsync() => extension.IsEnabledAsync(); + + /// + public Task CreateTestSessionAsync(CreateTestSessionContext context) + => Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); + + /// + public Task CloseTestSessionAsync(CloseTestSessionContext context) + => Task.FromResult(new CloseTestSessionResult { IsSuccess = true }); + + /// + public async Task ExecuteRequestAsync(ExecuteRequestContext context) + { + try + { + switch (context.Request) + { + case DiscoverTestExecutionRequest discoverRequest: + await DiscoverAsync(discoverRequest, context).ConfigureAwait(false); + break; + case RunTestExecutionRequest runRequest: + await RunAsync(runRequest, context).ConfigureAwait(false); + break; + } + } + finally + { + context.Complete(); + } + } + + private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) + { + foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) + { + context.CancellationToken.ThrowIfCancellationRequested(); + + var message = new TestNodeUpdateMessage( + request.Session.SessionUid, + node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + + await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); + } + } + + private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) + { + var matches = GetMatchingBenchmarks(request.Filter); + if (matches.Count == 0) + return; + + var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); + var sessionUid = request.Session.SessionUid; + var cancellationToken = context.CancellationToken; + + using var workQueue = new AsyncWorkQueue(); + + var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => + { + var message = new TestNodeUpdateMessage(sessionUid, testNode); + workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); + }); + + // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, + // which keeps it in the right place when the platform runs in server mode or inside an IDE. + var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); + + var runInfos = matches + .GroupBy(match => match.RunInfo) + .Select(group => new BenchmarkRunInfo( + group.Select(match => match.Node.BenchmarkCase).ToArray(), + group.Key.Type, + group.Key.Config + .AddEventProcessor(eventProcessor) + .AddLogger(logger) + .RemoveLoggersOfType() + .CreateImmutableConfig(), + group.Key.CompositeInProcessDiagnoser)) + .ToArray(); + + // BenchmarkDotNet blocks the calling thread for the whole run, so it gets a thread of its own and the + // queued messages are published from here as they are produced. + var runTask = Task.Run( + () => + { + try + { + BenchmarkRunner.Run(runInfos, cancellationToken); + } + finally + { + // Benchmarks that never reported a result still need one, unless the run was cancelled: the + // platform expects an OperationCanceledException in that case, and publishing results + // afterwards would contradict it. + if (!cancellationToken.IsCancellationRequested) + eventProcessor.PublishOutstandingResults(); + + logger.Flush(); + workQueue.Complete(); + } + }, + CancellationToken.None); + + await workQueue.DrainAsync().ConfigureAwait(false); + await runTask.ConfigureAwait(false); + } + + /// + /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. + /// + /// The filter of the request. + /// The matching benchmarks, paired with the run info they belong to. + private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) + { + var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); + + foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) + { + // The job only earns a place in the display name when the benchmark actually runs under several jobs. + // This is computed before filtering so that a benchmark keeps the same name however it was selected. + var includeJobInName = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; + + foreach (var benchmarkCase in runInfo.BenchmarksCases) + { + var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); + if (Matches(filter, node)) + matches.Add((runInfo, node)); + } + } + + return matches; + } + +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch + { + TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), + TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), + + // NopFilter, and anything the platform adds later, means "everything". + _ => true + }; +#pragma warning restore TPEXP + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs new file mode 100644 index 0000000000..61d6cfb7a5 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs @@ -0,0 +1,148 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Extensions.Messages; +using System.Reflection; +using System.Text; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// The Microsoft.Testing.Platform view of a single . + /// + /// + /// A carries mutable state (the property bag holds the current outcome), so a fresh node is + /// created for every message published on the bus. This class holds the parts that never change. + /// + internal sealed class BenchmarkTestNode + { + private readonly IProperty[] staticProperties; + + private BenchmarkTestNode(BenchmarkCase benchmarkCase, string uid, string displayName, string path, IProperty[] staticProperties) + { + BenchmarkCase = benchmarkCase; + Uid = uid; + DisplayName = displayName; + Path = path; + this.staticProperties = staticProperties; + } + + /// + /// Gets the benchmark this node represents. + /// + public BenchmarkCase BenchmarkCase { get; } + + /// + /// Gets the stable identifier of the node. It has to be identical in the discovery and the execution phase, + /// which may happen in different processes. + /// + public string Uid { get; } + + /// + /// Gets the name shown by test runners. + /// + public string DisplayName { get; } + + /// + /// Gets the '/' separated path used by . + /// + public string Path { get; } + + /// + /// Creates the node for a benchmark case. + /// + /// The benchmark case to describe. + /// + /// Whether the display name should be suffixed with the job, which is only useful when the benchmark runs + /// under more than one job. + /// + /// The created node. + public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool includeJobInName) + { + var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; + var type = benchmarkCase.Descriptor.Type; + var fullClassName = type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); + var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); + + // Unlike the uid, the job is only part of the display name when it actually adds information. + var uid = FullNameProvider.GetBenchmarkUid(benchmarkCase); + var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); + + var properties = new List + { + new TestMethodIdentifierProperty( + type.Assembly.FullName, + type.Namespace ?? string.Empty, + type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), + benchmarkMethod.Name, + benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, + benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), + benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), + }; + + var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); + if (benchmarkAttribute?.SourceCodeFile != null) + { + // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. + var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); + var position = new LinePosition(line, 0); + properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); + } + + foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) + properties.Add(new TestMetadataProperty("Category", category)); + + var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); + + return new BenchmarkTestNode(benchmarkCase, uid, displayName, path, properties.ToArray()); + } + + /// + /// Creates a message-bus ready node in the given state. + /// + /// The state of the benchmark, e.g. discovered, passed or failed. + /// Any additional properties, such as timing or captured output. + /// The created test node. + public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extraProperties) + { + var properties = new PropertyBag(staticProperties); + properties.Add(state); + foreach (var property in extraProperties) + properties.Add(property); + + return new TestNode + { + Uid = new TestNodeUid(Uid), + DisplayName = DisplayName, + Properties = properties + }; + } + + /// + /// Gets the properties a can match against, + /// which is what makes `--filter "/*/*/*/*[Category=Fast]"` work. + /// + /// The filterable properties. + public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); + + private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) + { + // The convention followed by the other test frameworks is ////. + var className = @namespace == null || !fullClassName.StartsWith(@namespace + ".", StringComparison.Ordinal) + ? fullClassName + : fullClassName.Substring(@namespace.Length + 1); + + return new StringBuilder() + .Append('/').Append(Escape(assembly.GetName().Name)) + .Append('/').Append(Escape(@namespace ?? string.Empty)) + .Append('/').Append(Escape(className)) + .Append('/').Append(Escape($"{methodName} [{jobDisplayInfo}]")) + .ToString(); + } + + // Benchmark parameters are stringified user values, so they can contain the path separator. + private static string Escape(string segment) => segment.Replace("/", "\\/"); + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs new file mode 100644 index 0000000000..917ca5b25e --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs @@ -0,0 +1,77 @@ +using BenchmarkDotNet.Loggers; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.OutputDevice; +using System.Text; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary + /// show up in the test run output. + /// + internal sealed class OutputDeviceLogger : ILogger + { + private readonly IOutputDevice outputDevice; + private readonly IOutputDeviceDataProducer producer; + private readonly AsyncWorkQueue workQueue; + private readonly CancellationToken cancellationToken; + private readonly StringBuilder currentLine = new(); + private LogKind currentLineKind = LogKind.Default; + + public OutputDeviceLogger( + IOutputDevice outputDevice, + IOutputDeviceDataProducer producer, + AsyncWorkQueue workQueue, + CancellationToken cancellationToken) + { + this.outputDevice = outputDevice; + this.producer = producer; + this.workQueue = workQueue; + this.cancellationToken = cancellationToken; + } + + public string Id => nameof(OutputDeviceLogger); + + public int Priority => 0; + + public void Write(LogKind logKind, string text) + { + currentLine.Append(text); + + // Assume that if any part of the line is an error or a warning, the whole line is. + // The kind is reset when the line is flushed. + if (logKind == LogKind.Error || (logKind == LogKind.Warning && currentLineKind != LogKind.Error)) + currentLineKind = logKind; + } + + public void WriteLine() + { + var text = currentLine.ToString(); + var kind = currentLineKind; + + currentLine.Clear(); + currentLineKind = LogKind.Default; + + IOutputDeviceData data = kind switch + { + LogKind.Error => new ErrorMessageOutputDeviceData(text), + LogKind.Warning => new WarningMessageOutputDeviceData(text), + _ => new TextOutputDeviceData(text) + }; + + workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); + } + + public void WriteLine(LogKind logKind, string text) + { + Write(logKind, text); + WriteLine(); + } + + public void Flush() + { + if (currentLine.Length > 0) + WriteLine(); + } + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs new file mode 100644 index 0000000000..9df284d376 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs @@ -0,0 +1,50 @@ +using Microsoft.Testing.Platform.Builder; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Helpers; +using System.Reflection; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. + /// + public static class TestApplicationBuilderExtensions + { + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the entry assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder) + => builder.AddBenchmarkDotNet( + Assembly.GetEntryAssembly() ?? throw new InvalidOperationException( + "There is no entry assembly to look for benchmarks in. Use the overload that takes an assembly.")); + + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the given assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The assembly to look for benchmarks in. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder, Assembly assembly) + { + if (builder == null) + throw new ArgumentNullException(nameof(builder)); + if (assembly == null) + throw new ArgumentNullException(nameof(assembly)); + + builder.RegisterTestFramework( + _ => new TestFrameworkCapabilities(), + (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); + + // Opts into the tree node filter, which is what backs `--filter "/*/*/MyBenchmarks/*"`. +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + builder.AddTreeNodeFilterService(new BenchmarkDotNetExtension()); +#pragma warning restore TPEXP + + return builder; + } + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs new file mode 100644 index 0000000000..5e77ef6b88 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs @@ -0,0 +1,26 @@ +using Microsoft.Testing.Platform.Builder; +using System.ComponentModel; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. + /// + /// + /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestingPlatform.props. + /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called + /// directly; use + /// instead when writing an entry point by hand. + /// + [EditorBrowsable(EditorBrowsableState.Never)] + public static class TestingPlatformBuilderHook + { + /// + /// Registers BenchmarkDotNet with the test application being built. + /// + /// The builder of the test application. + /// The command line arguments of the process. Unused. + public static void AddExtensions(ITestApplicationBuilder builder, string[] arguments) + => builder.AddBenchmarkDotNet(); + } +} From 2158052015ce471adb0fbf1f1cfb705384698609 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:18:20 +0100 Subject: [PATCH 26/68] Update project references to TestingPlatform project Updated BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj to reference BenchmarkDotNet.TestingPlatform instead of BenchmarkDotNet.TestAdapter.TestingPlatform. Adjusted both the ProjectReference and Import paths accordingly. --- .../BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj index b89575026a..59913fd0b3 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -9,11 +9,11 @@ - + - + From 67771650f6dfabfc94640a8939ce0d7e38256bd5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:18:33 +0100 Subject: [PATCH 27/68] Refactor solution file project entries Simplified project definitions in BenchmarkDotNet.slnx by removing custom build entries and updating project paths, including renaming the TestingPlatform project. --- BenchmarkDotNet.slnx | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 119a22f6ca..ca8e06a6dd 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -20,10 +20,8 @@ - - - + @@ -42,16 +40,10 @@ - - - - - - + + - - - + From 83961038d37868cf4f5720b585ad6897527d51f4 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 14:02:44 +0100 Subject: [PATCH 28/68] Update props packaging and cSpell dictionary - Add "testingplatform" to cSpell.json to suppress spelling warnings. - Refactor .props packaging in csproj to use a single entry with multiple paths, ensuring cross-platform compatibility and resolving NU5129. --- build/cSpell.json | 1 + .../BenchmarkDotNet.TestingPlatform.csproj | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/build/cSpell.json b/build/cSpell.json index e90695fa7f..4cb6c27117 100644 --- a/build/cSpell.json +++ b/build/cSpell.json @@ -34,6 +34,7 @@ "vsprofiler", "vstest", "Tailcall", + "testingplatform", "toolchains", "unmanaged" ], diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj index a97893d407..7bcea6e705 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj @@ -38,8 +38,9 @@ - - + + From cd523d53ce876c78fa586c02144602852414c759 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 03:36:56 +0100 Subject: [PATCH 29/68] Add generic benchmark for closed generic type testing Introduced GenericProbe benchmark class in a new BenchmarkDotNet.IntegrationTests.TestingPlatform namespace to evaluate how test runners handle closed generic types. The benchmark tests instance creation for int, char, and List type arguments. Added GenericProbeConfig to configure the benchmark with InProcessEmit toolchain and a dry job. --- .../GenericProbe.cs | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs new file mode 100644 index 0000000000..24ffa27b9c --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs @@ -0,0 +1,25 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A generic benchmark, used to check how closed generic types are named and grouped by test runners. + /// + [Config(typeof(GenericProbeConfig))] + [GenericTypeArguments(typeof(int))] + [GenericTypeArguments(typeof(char))] + [GenericTypeArguments(typeof(System.Collections.Generic.List))] + public class GenericProbe where T : new() + { + [Benchmark] + public T Create() => new T(); + } + + internal class GenericProbeConfig : ManualConfig + { + public GenericProbeConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } +} From 9f2cacb1f7924972fd73b20fb3758aa33933ad67 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 03:38:15 +0100 Subject: [PATCH 30/68] Handle benchmark UID collisions during discovery/run Benchmarks are now grouped by UID to detect collisions. When multiple benchmarks share a UID, `PublishCollisionAsync` reports the issue as a failed test node, allowing other benchmarks to proceed. Only benchmarks with unique UIDs are executed. The refactor introduces a `Match` class, improves cancellation and exception handling, and ensures proper resource cleanup during async operations. --- .../BenchmarkTestFramework.cs | 149 +++++++++++++++--- 1 file changed, 125 insertions(+), 24 deletions(-) diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs index 48fcfa8d72..368c12dfa0 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs @@ -9,7 +9,9 @@ using Microsoft.Testing.Platform.Extensions.TestFramework; using Microsoft.Testing.Platform.Requests; using Microsoft.Testing.Platform.Services; +using Microsoft.Testing.Platform.TestHost; using System.Reflection; +using System.Runtime.ExceptionServices; namespace BenchmarkDotNet.TestingPlatform { @@ -83,13 +85,15 @@ public async Task ExecuteRequestAsync(ExecuteRequestContext context) private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) { - foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) + foreach (var benchmarks in GetMatchingBenchmarks(request.Filter)) { context.CancellationToken.ThrowIfCancellationRequested(); + // Exactly one node per uid: publishing a colliding uid twice would leave the platform with two nodes + // it cannot tell apart. The collision itself is reported when the benchmarks are run. var message = new TestNodeUpdateMessage( request.Session.SessionUid, - node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + benchmarks[0].Node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); } @@ -97,16 +101,29 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) { - var matches = GetMatchingBenchmarks(request.Filter); - if (matches.Count == 0) - return; - - var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); var sessionUid = request.Session.SessionUid; var cancellationToken = context.CancellationToken; + var runnable = new List(); + foreach (var benchmarks in GetMatchingBenchmarks(request.Filter)) + { + if (benchmarks.Count == 1) + runnable.Add(benchmarks[0]); + else + await PublishCollisionAsync(context, sessionUid, benchmarks).ConfigureAwait(false); + } + + if (runnable.Count == 0) + return; + + var nodes = runnable.ToDictionary(match => match.Node.Uid, match => match.Node); + using var workQueue = new AsyncWorkQueue(); + // A failure while publishing has to stop the benchmarks as well, otherwise the run would carry on with + // nobody listening and would keep writing to a queue that is about to be disposed. + using var runCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => { var message = new TestNodeUpdateMessage(sessionUid, testNode); @@ -117,7 +134,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte // which keeps it in the right place when the platform runs in server mode or inside an IDE. var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); - var runInfos = matches + var runInfos = runnable .GroupBy(match => match.RunInfo) .Select(group => new BenchmarkRunInfo( group.Select(match => match.Node.BenchmarkCase).ToArray(), @@ -137,34 +154,93 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte { try { - BenchmarkRunner.Run(runInfos, cancellationToken); + BenchmarkRunner.Run(runInfos, runCancellation.Token); } finally { - // Benchmarks that never reported a result still need one, unless the run was cancelled: the - // platform expects an OperationCanceledException in that case, and publishing results - // afterwards would contradict it. - if (!cancellationToken.IsCancellationRequested) - eventProcessor.PublishOutstandingResults(); - - logger.Flush(); - workQueue.Complete(); + try + { + // Benchmarks that never reported a result still need one, unless the run was cancelled: + // the platform expects an OperationCanceledException in that case, and publishing results + // afterwards would contradict it. + if (!runCancellation.IsCancellationRequested) + eventProcessor.PublishOutstandingResults(); + + logger.Flush(); + } + finally + { + // The drain only ends once the queue is completed, so this has to happen no matter what + // else went wrong. + workQueue.Complete(); + } } }, CancellationToken.None); - await workQueue.DrainAsync().ConfigureAwait(false); - await runTask.ConfigureAwait(false); + ExceptionDispatchInfo? drainFailure = null; + try + { + await workQueue.DrainAsync().ConfigureAwait(false); + } + catch (Exception exception) + { + // Nothing consumes the queue anymore, so the run has to be stopped rather than left orphaned. This is + // also the path a cancelled run takes, since the queued writes are handed the platform's token. + drainFailure = ExceptionDispatchInfo.Capture(exception); + runCancellation.Cancel(); + } + + try + { + // BenchmarkDotNet keeps writing to the queue until its thread returns, so the run always has to be + // over before the queue is disposed. + await runTask.ConfigureAwait(false); + } + catch when (drainFailure != null) + { + // The run was stopped because publishing failed, so that failure is the one worth reporting. + } + + drainFailure?.Throw(); + } + + /// + /// Reports benchmarks that share a uid as a single failed test. + /// + /// + /// The platform identifies test nodes by uid, so benchmarks that produce the same one cannot be reported + /// separately. Failing them keeps the rest of the run going, which is more useful than aborting the request. + /// + private async Task PublishCollisionAsync(ExecuteRequestContext context, SessionUid sessionUid, List collision) + { + var node = collision[0].Node; + var error = + $"{collision.Count} benchmarks are identified as '{node.Uid}', so they cannot be told apart and none " + + "of them were run. Benchmarks are identified by the string representation of their parameters: give " + + "the colliding values distinct ToString() results."; + + await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance))).ConfigureAwait(false); + + await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, node.ToTestNode(new FailedTestNodeStateProperty(error)))).ConfigureAwait(false); } /// /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. /// /// The filter of the request. - /// The matching benchmarks, paired with the run info they belong to. - private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) + /// + /// The matching benchmarks in enumeration order, grouped by uid. A group holding more than one benchmark is a + /// uid collision. + /// + private List> GetMatchingBenchmarks(ITestExecutionFilter filter) { - var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); + var matches = new List>(); + var matchesByUid = new Dictionary>(StringComparer.Ordinal); foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) { @@ -175,8 +251,17 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte foreach (var benchmarkCase in runInfo.BenchmarksCases) { var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); - if (Matches(filter, node)) - matches.Add((runInfo, node)); + if (!Matches(filter, node)) + continue; + + if (!matchesByUid.TryGetValue(node.Uid, out var sameUid)) + { + sameUid = new List(); + matchesByUid.Add(node.Uid, sameUid); + matches.Add(sameUid); + } + + sameUid.Add(new Match(runInfo, node)); } } @@ -193,5 +278,21 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte _ => true }; #pragma warning restore TPEXP + + /// + /// A benchmark that matched the request, together with the run info it belongs to. + /// + private sealed class Match + { + public Match(BenchmarkRunInfo runInfo, BenchmarkTestNode node) + { + RunInfo = runInfo; + Node = node; + } + + public BenchmarkRunInfo RunInfo { get; } + + public BenchmarkTestNode Node { get; } + } } } From 539cc5b7994979121b5e9ba89c8ec83871442c41 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 03:38:29 +0100 Subject: [PATCH 31/68] Adjust whitespace around [PublicAPI] attribute Only whitespace was changed above the GetBenchmarkUid method; no functional or logical modifications were made. --- src/BenchmarkDotNet/Exporters/FullNameProvider.cs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 922d3590c4..79de14dc1c 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -73,14 +73,13 @@ public static string GetBenchmarkName(BenchmarkCase benchmarkCase) /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would /// collide. The parameters are already part of the method name. /// - [PublicAPI] + [PublicAPI] public static string GetBenchmarkUid(BenchmarkCase benchmarkCase) { var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); return $"{fullClassName}.{GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; } - private static string GetNestedTypes(Type type) { string nestedTypes = ""; From f5af28f97efbe07b2db67a1226273d13f8cb3b0a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 04:02:29 +0100 Subject: [PATCH 32/68] Remove AsyncWorkQueue and related async queue logic Deleted AsyncWorkQueue.cs, removing the AsyncWorkQueue class and all associated methods for managing and draining asynchronous work items. This eliminates the custom ordered async work queue implementation. --- .../AsyncWorkQueue.cs | 61 ------------------- 1 file changed, 61 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs deleted file mode 100644 index 89818c96c2..0000000000 --- a/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs +++ /dev/null @@ -1,61 +0,0 @@ -using System.Collections.Concurrent; - -namespace BenchmarkDotNet.TestingPlatform -{ - /// - /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. - /// - /// - /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the - /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks - /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks - /// enqueue instead and the caller of DrainAsync does the awaiting. - /// - internal sealed class AsyncWorkQueue : IDisposable - { - private readonly ConcurrentQueue> queue = new(); - private readonly SemaphoreSlim available = new(0); - private volatile bool completed; - - /// - /// Queues a work item. Safe to call from any thread. - /// - /// The work to perform. - public void Enqueue(Func work) - { - queue.Enqueue(work); - available.Release(); - } - - /// - /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued - /// items have been processed. - /// - public void Complete() - { - completed = true; - available.Release(); - } - - /// - /// Processes queued work items in order until has been called and the queue is empty. - /// - /// A task that completes when the queue has been drained. - public async Task DrainAsync() - { - while (true) - { - await available.WaitAsync().ConfigureAwait(false); - - // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. - while (queue.TryDequeue(out var work)) - await work().ConfigureAwait(false); - - if (completed && queue.IsEmpty) - return; - } - } - - public void Dispose() => available.Dispose(); - } -} From fc1a8da2d4849b41ebf4e0282c4d8a1a3ffea266 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 04:02:42 +0100 Subject: [PATCH 33/68] Refactor to use ChannelWriter for log task queuing Replaces custom AsyncWorkQueue with ChannelWriter> for queuing log display tasks. Updates constructor and field types, and switches from Enqueue to TryWrite for task scheduling. This enhances integration with .NET's built-in concurrency primitives. --- src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs index 917ca5b25e..aad7dc8fae 100644 --- a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs +++ b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs @@ -2,6 +2,7 @@ using Microsoft.Testing.Platform.Extensions.OutputDevice; using Microsoft.Testing.Platform.OutputDevice; using System.Text; +using System.Threading.Channels; namespace BenchmarkDotNet.TestingPlatform { @@ -13,7 +14,7 @@ internal sealed class OutputDeviceLogger : ILogger { private readonly IOutputDevice outputDevice; private readonly IOutputDeviceDataProducer producer; - private readonly AsyncWorkQueue workQueue; + private readonly ChannelWriter> workQueue; private readonly CancellationToken cancellationToken; private readonly StringBuilder currentLine = new(); private LogKind currentLineKind = LogKind.Default; @@ -21,7 +22,7 @@ internal sealed class OutputDeviceLogger : ILogger public OutputDeviceLogger( IOutputDevice outputDevice, IOutputDeviceDataProducer producer, - AsyncWorkQueue workQueue, + ChannelWriter> workQueue, CancellationToken cancellationToken) { this.outputDevice = outputDevice; @@ -59,7 +60,7 @@ public void WriteLine() _ => new TextOutputDeviceData(text) }; - workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); + workQueue.TryWrite(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); } public void WriteLine(LogKind logKind, string text) From 52d46c7a0160bbafef53cb9bcc3bfebf357788b9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 04:02:55 +0100 Subject: [PATCH 34/68] Replace AsyncWorkQueue with Channel for event processing Switch to System.Threading.Channels for the benchmark event work queue to improve thread safety and prevent deadlocks. Update event processor and logger to use the channel writer, and add a DrainAsync method to process queued work items sequentially until completion. --- .../BenchmarkTestFramework.cs | 39 +++++++++++++++---- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs index 368c12dfa0..a08a70a7c4 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs @@ -12,6 +12,7 @@ using Microsoft.Testing.Platform.TestHost; using System.Reflection; using System.Runtime.ExceptionServices; +using System.Threading.Channels; namespace BenchmarkDotNet.TestingPlatform { @@ -118,21 +119,30 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte var nodes = runnable.ToDictionary(match => match.Node.Uid, match => match.Node); - using var workQueue = new AsyncWorkQueue(); + // BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while + // the message bus and the output device are asynchronous. Blocking on those from inside a callback risks + // deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks + // write to this channel and the drain below does the awaiting. Synchronous continuations are left off, so + // that a write can never end up publishing on BenchmarkDotNet's own thread. + var workQueue = Channel.CreateUnbounded>(new UnboundedChannelOptions + { + SingleReader = true, + AllowSynchronousContinuations = false + }); // A failure while publishing has to stop the benchmarks as well, otherwise the run would carry on with - // nobody listening and would keep writing to a queue that is about to be disposed. + // nobody listening to it. using var runCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => { var message = new TestNodeUpdateMessage(sessionUid, testNode); - workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); + workQueue.Writer.TryWrite(() => context.MessageBus.PublishAsync(this, message)); }); // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, // which keeps it in the right place when the platform runs in server mode or inside an IDE. - var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); + var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue.Writer, cancellationToken); var runInfos = runnable .GroupBy(match => match.RunInfo) @@ -172,7 +182,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte { // The drain only ends once the queue is completed, so this has to happen no matter what // else went wrong. - workQueue.Complete(); + workQueue.Writer.TryComplete(); } } }, @@ -181,7 +191,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte ExceptionDispatchInfo? drainFailure = null; try { - await workQueue.DrainAsync().ConfigureAwait(false); + await DrainAsync(workQueue.Reader).ConfigureAwait(false); } catch (Exception exception) { @@ -193,8 +203,8 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte try { - // BenchmarkDotNet keeps writing to the queue until its thread returns, so the run always has to be - // over before the queue is disposed. + // The run has to be over before the request completes, otherwise it would carry on in the background + // and its failure would go unobserved. await runTask.ConfigureAwait(false); } catch when (drainFailure != null) @@ -205,6 +215,19 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte drainFailure?.Throw(); } + /// + /// Runs the queued work items in order, until the queue is completed and empty. + /// + /// The reader of the work queue. + private static async Task DrainAsync(ChannelReader> reader) + { + while (await reader.WaitToReadAsync().ConfigureAwait(false)) + { + while (reader.TryRead(out var work)) + await work().ConfigureAwait(false); + } + } + /// /// Reports benchmarks that share a uid as a single failed test. /// From 8b4492e37d5f0c92ada9ad68ab5cb9c6defa3bf2 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 05:11:19 +0100 Subject: [PATCH 35/68] Switch to GetUniqueId for benchmark case identification Standardized benchmark case UID generation by replacing all usages of FullNameProvider.GetBenchmarkUid with BenchmarkCase.GetUniqueId. Removed the obsolete GetBenchmarkUid method. Updated comments for clarity. Also adjusted .slnx to control build for TestingPlatform projects. --- BenchmarkDotNet.slnx | 8 ++++++-- .../BenchmarkEventProcessor.cs | 8 ++++---- .../BenchmarkTestNode.cs | 6 ++++-- .../Exporters/FullNameProvider.cs | 17 ----------------- 4 files changed, 14 insertions(+), 25 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index ca8e06a6dd..ec83125a62 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -21,7 +21,9 @@ - + + + @@ -43,7 +45,9 @@ - + + + diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs index a7d50256da..9aab6ff3e8 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs @@ -36,7 +36,7 @@ public override void OnValidationError(ValidationError validationError) // If the error is not linked to a benchmark case, then set the error on all benchmarks. var affected = validationError.BenchmarkCase == null ? nodes.Values - : [nodes[FullNameProvider.GetBenchmarkUid(validationError.BenchmarkCase)]]; + : [nodes[validationError.BenchmarkCase.GetUniqueId()]]; foreach (var node in affected) { @@ -62,7 +62,7 @@ public override void OnBuildComplete(BuildPartition buildPartition, BuildResult foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) { - var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkBuildInfo.BenchmarkCase)]; + var node = nodes[benchmarkBuildInfo.BenchmarkCase.GetUniqueId()]; var pending = GetOrCreatePendingResult(node); if (buildResult.GenerateException != null) @@ -80,7 +80,7 @@ public override void OnBuildComplete(BuildPartition buildPartition, BuildResult public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) { - var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var node = nodes[benchmarkCase.GetUniqueId()]; var pending = GetOrCreatePendingResult(node); pending.StartTime = DateTimeOffset.UtcNow; @@ -90,7 +90,7 @@ public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) { - var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var node = nodes[benchmarkCase.GetUniqueId()]; var pending = GetOrCreatePendingResult(node); pending.Duration = runTimerStopwatch.Elapsed; pending.EndTime = DateTimeOffset.UtcNow; diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs index 61d6cfb7a5..f67ed910fd 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs @@ -66,8 +66,10 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); - // Unlike the uid, the job is only part of the display name when it actually adds information. - var uid = FullNameProvider.GetBenchmarkUid(benchmarkCase); + // The uid is the hash BenchmarkDotNet itself uses (and reports through `--list json`), so that a benchmark + // keeps the same identity across processes and across tools. The job is only part of the display name + // when it actually adds information. + var uid = benchmarkCase.GetUniqueId(); var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); var properties = new List diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 79de14dc1c..19555908d2 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -63,23 +63,6 @@ public static string GetBenchmarkName(BenchmarkCase benchmarkCase) return name.ToString(); } - /// - /// Gets an identifier of a benchmark case that stays the same across processes, which is what lets a benchmark - /// discovered in one process be selected for execution in another (for example by a test adapter). - /// - /// The benchmark case to identify. - /// The unique identifier of the benchmark case. - /// - /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would - /// collide. The parameters are already part of the method name. - /// - [PublicAPI] - public static string GetBenchmarkUid(BenchmarkCase benchmarkCase) - { - var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - return $"{fullClassName}.{GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; - } - private static string GetNestedTypes(Type type) { string nestedTypes = ""; From 9f3298023851eb494a362d8e477d00c9965996dc Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:24:02 +0100 Subject: [PATCH 36/68] Move TestingPlatform config to .targets; cleanup props/csproj Refactor BenchmarkDotNet.TestingPlatform integration by moving configuration logic from .props to BenchmarkDotNet.TestAdapter.targets. Remove obsolete .props and .csproj files. Update cSpell dictionary to use "testadapter" instead of "testingplatform". --- build/cSpell.json | 2 +- .../{testingplatform.md => testadapter.md} | 0 .../BenchmarkDotNetExtension.cs | 0 .../BenchmarkEventProcessor.cs | 0 .../BenchmarkTestFramework.cs | 0 .../TestingPlatform}/BenchmarkTestNode.cs | 0 .../TestingPlatform}/OutputDeviceLogger.cs | 0 .../TestApplicationBuilderExtensions.cs | 0 .../TestingPlatformBuilderHook.cs | 0 .../build/BenchmarkDotNet.TestAdapter.targets | 34 ++++++++++++++ .../BenchmarkDotNet.TestingPlatform.csproj | 47 ------------------- .../BenchmarkDotNet.TestingPlatform.props | 31 ------------ 12 files changed, 35 insertions(+), 79 deletions(-) rename docs/articles/features/{testingplatform.md => testadapter.md} (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkDotNetExtension.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkEventProcessor.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkTestFramework.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkTestNode.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/OutputDeviceLogger.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/TestApplicationBuilderExtensions.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/TestingPlatformBuilderHook.cs (100%) create mode 100644 src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets delete mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj delete mode 100644 src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props diff --git a/build/cSpell.json b/build/cSpell.json index 4cb6c27117..db68aadb88 100644 --- a/build/cSpell.json +++ b/build/cSpell.json @@ -34,7 +34,7 @@ "vsprofiler", "vstest", "Tailcall", - "testingplatform", + "testadapter", "toolchains", "unmanaged" ], diff --git a/docs/articles/features/testingplatform.md b/docs/articles/features/testadapter.md similarity index 100% rename from docs/articles/features/testingplatform.md rename to docs/articles/features/testadapter.md diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets new file mode 100644 index 0000000000..339a7fd56b --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets @@ -0,0 +1,34 @@ + + + + + true + false + + + true + + + + + + BenchmarkDotNet + BenchmarkDotNet.TestAdapter.TestingPlatform.TestingPlatformBuilderHook + + + diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj deleted file mode 100644 index 7bcea6e705..0000000000 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj +++ /dev/null @@ -1,47 +0,0 @@ - - - - netstandard2.0 - BenchmarkDotNet.TestingPlatform - BenchmarkDotNet.TestingPlatform - BenchmarkDotNet.TestingPlatform - Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform - README.md - True - BenchmarkDotNet.TestingPlatform - - - false - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props b/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props deleted file mode 100644 index 5f1a9412ef..0000000000 --- a/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props +++ /dev/null @@ -1,31 +0,0 @@ - - - - true - - - true - - - false - - - - - - BenchmarkDotNet - BenchmarkDotNet.TestingPlatform.TestingPlatformBuilderHook - - - From 1f73e5b3979769d7ed253acc1bf321f84dd4b308 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:39:23 +0100 Subject: [PATCH 37/68] Update docs for new TestAdapter and MTP integration Rewrote and reorganized documentation to focus on the new BenchmarkDotNet.TestAdapter package and its integration with Microsoft.Testing.Platform (MTP) and VSTest. Clarified default behaviors, entry point handling, and configuration steps. Updated code samples and project file snippets. Revised table of contents to reflect the new structure and clarified the relationship between MTP and VSTest. Added notes on IDE support and caveats. --- docs/articles/features/testadapter.md | 86 ++++++++++++++++++--------- docs/articles/features/toc.yml | 6 +- docs/articles/features/vstest.md | 26 +++++--- 3 files changed, 79 insertions(+), 39 deletions(-) diff --git a/docs/articles/features/testadapter.md b/docs/articles/features/testadapter.md index a54ccab78e..90e159768b 100644 --- a/docs/articles/features/testadapter.md +++ b/docs/articles/features/testadapter.md @@ -1,33 +1,32 @@ --- -uid: docs.testingplatform -name: Running with Microsoft.Testing.Platform +uid: docs.testadapter +name: Running benchmarks as tests --- -# Running with Microsoft.Testing.Platform +# Running benchmarks as tests -BenchmarkDotNet can discover and execute benchmarks through - [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) (MTP), - the test platform that succeeds VSTest. -This gives you the same "benchmarks as tests" experience as [the VSTest adapter](xref:docs.vstest), - but on the platform that `dotnet test` and modern IDE integrations are moving to. +`BenchmarkDotNet.TestAdapter` lets your IDE and `dotnet test` discover and execute your benchmarks the way they do + unit tests. +This provides an alternative user experience to running benchmarks with the CLI + and may be preferable for those who like their IDE's test integrations that they may have used when running unit tests. -If you are looking for the VSTest adapter, see [Running with VSTest](xref:docs.vstest) instead. -You only need one of the two. +Below is an example of running some benchmarks from the BenchmarkDotNet samples project in Visual Studio's Test Explorer. -## VSTest or Microsoft.Testing.Platform? +![](../../images/vs-testexplorer-demo.png) -The two adapters solve the same problem on different platforms, and the difference that matters most is *where your - benchmarks run*: +The adapter supports two test platforms: -* With **VSTest**, an external `testhost` process loads your benchmark assembly and the adapter reflects into it. -* With **Microsoft.Testing.Platform**, your benchmark project *is* the test host. - There is no separate host process, so BenchmarkDotNet behaves exactly as it does when you run the app from the CLI. +* [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) (MTP), + the platform that `dotnet test` and modern IDE integrations are moving to. **This is the default.** +* [VSTest](xref:docs.vstest), for tooling without Microsoft.Testing.Platform support (such as Visual Studio 2019) + and for solutions that mix benchmark projects with VSTest based test projects. -The practical consequences of the MTP model are: +The difference that matters most is *where your benchmarks run*: -* The adapter no longer needs the child `AppDomain` that the VSTest adapter uses to load your assemblies correctly. -* Your project's entry point is generated by the platform and starts the test application, - so it no longer calls `BenchmarkSwitcher`. See [Keeping a BenchmarkSwitcher entry point](#keeping-a-benchmarkswitcher-entry-point). +* With **VSTest**, an external `testhost` process loads your benchmark assembly and the adapter reflects into it. +* With **Microsoft.Testing.Platform**, your benchmark project *is* the test host. + There is no separate host process, so BenchmarkDotNet behaves exactly as it does when you run the app from the CLI, + and the adapter does not need the child `AppDomain` that the VSTest adapter uses to load your assemblies correctly. ## Caveats and things to know @@ -44,7 +43,8 @@ The practical consequences of the MTP model are: you will need to do this through other means such as an assembly-level `IConfigSource`, as shown in [Setting a default configuration](xref:docs.vstest#setting-a-default-configuration). * **The adapter will generate an entry point for you automatically.** - Unlike the VSTest adapter, the generated entry point starts the test application rather than `BenchmarkSwitcher`. + The generated entry point starts the test application. + See [Keeping your own entry point](#keeping-your-own-entry-point) if your project already has one. ## Getting started @@ -53,7 +53,7 @@ The practical consequences of the MTP model are: ```xml - + ``` @@ -72,7 +72,7 @@ The practical consequences of the MTP model are: - + @@ -128,28 +128,39 @@ dotnet run -c Release -- --treenode-filter "/*/*/MyBenchmarks/*" # Run every benchmark of a category. dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" - -# Run one specific benchmark, by the exact id reported by the platform. -dotnet run -c Release -- --filter-uid "MyProject.MyBenchmarks.Add(x: 1) [DefaultJob]" ``` The tree node filter path is `////`, and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. -## Keeping a BenchmarkSwitcher entry point +## Keeping your own entry point The generated entry point starts the test application, which means it replaces the `BenchmarkSwitcher` entry point that a benchmark project normally has. -If you want to keep your own entry point, turn off the generated one and register BenchmarkDotNet yourself: +There are two ways to keep your own. + +To keep a plain `BenchmarkSwitcher` entry point, tell the adapter that the project generates its own: + +```xml + + + false + +``` + +The project is then a normal console application again, and no test platform integration is set up for it. + +To keep an entry point *and* the test integration, start the test application yourself: ```xml + true false ``` ```csharp -using BenchmarkDotNet.TestingPlatform; +using BenchmarkDotNet.TestAdapter.TestingPlatform; using Microsoft.Testing.Platform.Builder; public static class Program @@ -167,6 +178,23 @@ public static class Program From there you are free to decide when to start the test application and when to hand over to `BenchmarkSwitcher`, for example by looking at the arguments your CI passes. +## Using VSTest instead + +Set `BenchmarkDotNetUseVSTest` and add the VSTest host package: + +```xml + + true + + + + + + +``` + +See [Running with VSTest](xref:docs.vstest) for the details, including the IDE settings that VSTest integration needs. + ## Viewing the results The full BenchmarkDotNet output, including the summary table that compares benchmarks with each other, diff --git a/docs/articles/features/toc.yml b/docs/articles/features/toc.yml index f2fb0728d2..db72e5c3e2 100644 --- a/docs/articles/features/toc.yml +++ b/docs/articles/features/toc.yml @@ -16,7 +16,7 @@ href: event-pipe-profiler.md - name: VSProfiler href: vsprofiler.md +- name: Benchmarks as tests + href: testadapter.md - name: VSTest - href: vstest.md -- name: Microsoft.Testing.Platform - href: testingplatform.md \ No newline at end of file + href: vstest.md \ No newline at end of file diff --git a/docs/articles/features/vstest.md b/docs/articles/features/vstest.md index e86527211c..2e0b3368d3 100644 --- a/docs/articles/features/vstest.md +++ b/docs/articles/features/vstest.md @@ -6,9 +6,9 @@ name: Running with VSTest # Running with VSTest > [!NOTE] -> BenchmarkDotNet also ships an adapter for [Microsoft.Testing.Platform](xref:docs.testingplatform), -> the test platform that succeeds VSTest. -> You only need one of the two. +> `BenchmarkDotNet.TestAdapter` runs your benchmarks through +> [Microsoft.Testing.Platform](xref:docs.testadapter) by default, which is the platform that succeeds VSTest. +> VSTest is opt-in, as described below. BenchmarkDotNet supports discovering and executing benchmarks through VSTest. This provides an alternative user experience to running benchmarks with the CLI @@ -59,7 +59,17 @@ In addition, we can still make use of this boolean output to indicate You need to install two packages into your benchmark project: * `BenchmarkDotNet.TestAdapter`: Implements the VSTest protocol for BenchmarkDotNet * `Microsoft.NET.Test.Sdk`: Includes all the pieces needed for the VSTest host to run and load the VSTest adapter. -* **Step 2.** Make sure that the entry point is configured correctly. +* **Step 2.** Ask the adapter for VSTest. + `BenchmarkDotNet.TestAdapter` uses [Microsoft.Testing.Platform](xref:docs.testadapter) unless you set + `BenchmarkDotNetUseVSTest` in your project file: + +```xml + + true + +``` + +* **Step 3.** Make sure that the entry point is configured correctly. As mentioned in the caveats section, `BenchmarkDotNet.TestAdapter` will generate an entry point for you automatically. So, if you have an entry point already, you will either need to delete it or set `GenerateProgramFile` to `false` in your project file to continue using your existing one. @@ -73,6 +83,8 @@ In addition, we can still make use of this boolean output to indicate net8.0 enable enable + + true false @@ -85,7 +97,7 @@ In addition, we can still make use of this boolean output to indicate ``` -* **Step 3.** Make sure that your IDE supports VSTest integration. +* **Step 4.** Make sure that your IDE supports VSTest integration. In Visual Studio, everything works out of the box. In Rider/R#, the VSTest integration might need to be activated: * Go to the "Unit Testing" settings page. @@ -93,9 +105,9 @@ In addition, we can still make use of this boolean output to indicate * R#: Extensions -> ReSharper -> Options -> Tools -> Unit Testing -> Test Frameworks -> VSTest * Make sure that the "Enable VSTest adapter support" checkbox is checked. In recent versions of Rider, this may be enabled by default. -* **Step 4.** Switch to the `Release` configuration. +* **Step 5.** Switch to the `Release` configuration. As mentioned above, the TestAdapter is not able to discover and run benchmarks with optimizations disabled (by design). -* **Step 5.** Build the project. +* **Step 6.** Build the project. In order to discover the benchmarks, the VSTest adapter needs to be able to find the assembly. Once you build the project, you should observe the discovered benchmarks in your IDE's Unit Test Explorer. From 68be1e28b006c4ed41404c37b2320fa8ffe88533 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:39:52 +0100 Subject: [PATCH 38/68] Update namespaces and extension UID for adapter alignment Refactor namespaces from BenchmarkDotNet.TestingPlatform to BenchmarkDotNet.TestAdapter.TestingPlatform throughout the codebase. Update the extension UID and related comments to match the new adapter naming convention and integration targets. --- .../TestingPlatform/BenchmarkDotNetExtension.cs | 8 ++++---- .../TestingPlatform/BenchmarkEventProcessor.cs | 2 +- .../TestingPlatform/BenchmarkTestFramework.cs | 4 +--- .../TestingPlatform/BenchmarkTestNode.cs | 2 +- .../TestingPlatform/OutputDeviceLogger.cs | 2 +- .../TestingPlatform/TestApplicationBuilderExtensions.cs | 2 +- .../TestingPlatform/TestingPlatformBuilderHook.cs | 4 ++-- 7 files changed, 11 insertions(+), 13 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs index d2bc8722d8..5f6a54403e 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs @@ -1,6 +1,6 @@ using Microsoft.Testing.Platform.Extensions; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. @@ -14,11 +14,11 @@ internal sealed class BenchmarkDotNetExtension : IExtension /// /// The uid shared by every extension this package registers. /// - public const string ExtensionUid = "BenchmarkDotNet.TestingPlatform"; + public const string ExtensionUid = "BenchmarkDotNet.TestAdapter"; /// public string Uid => ExtensionUid; - + /// public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; @@ -26,7 +26,7 @@ internal sealed class BenchmarkDotNetExtension : IExtension public string DisplayName => "BenchmarkDotNet"; /// - public string Description => "Runs BenchmarkDotNet benchmarks as tests."; + public string Description => "Runs BenchmarkDotNet benchmarks as tests."; /// public Task IsEnabledAsync() => Task.FromResult(true); diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs index 9aab6ff3e8..523f7a89a9 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs @@ -11,7 +11,7 @@ using System.Globalization; using System.Text; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index a08a70a7c4..bdce27cd9d 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -1,8 +1,6 @@ using BenchmarkDotNet.Configs; using BenchmarkDotNet.Loggers; using BenchmarkDotNet.Running; -// BenchmarkEnumerator is compiled into this assembly from the VSTest adapter, where it keeps its own namespace. -using BenchmarkDotNet.TestAdapter; using Microsoft.Testing.Platform.Capabilities.TestFramework; using Microsoft.Testing.Platform.Extensions.Messages; using Microsoft.Testing.Platform.Extensions.OutputDevice; @@ -14,7 +12,7 @@ using System.Runtime.ExceptionServices; using System.Threading.Channels; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index f67ed910fd..14f6bd171e 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -6,7 +6,7 @@ using System.Reflection; using System.Text; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// The Microsoft.Testing.Platform view of a single . diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs index aad7dc8fae..a2d1b0d2b4 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs @@ -4,7 +4,7 @@ using System.Text; using System.Threading.Channels; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs index 9df284d376..0f72cb90c6 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs @@ -3,7 +3,7 @@ using Microsoft.Testing.Platform.Helpers; using System.Reflection; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs index 5e77ef6b88..bc7c7db03c 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs @@ -1,13 +1,13 @@ using Microsoft.Testing.Platform.Builder; using System.ComponentModel; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. /// /// - /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestingPlatform.props. + /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestAdapter.targets. /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called /// directly; use /// instead when writing an entry point by hand. From 9deb942cbb07465dc4518bcac6f891b92548fae6 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:40:23 +0100 Subject: [PATCH 39/68] Manually import TestAdapter build files in sample projects Added explicit imports for BenchmarkDotNet.TestAdapter .props and .targets files in both F# and C# sample projects to ensure adapter build logic is applied. Also imported common.targets. This preserves custom entry points and prevents conversion to Microsoft.Testing.Platform applications. --- .../BenchmarkDotNet.Samples.FSharp.fsproj | 8 ++++++++ .../BenchmarkDotNet.Samples.csproj | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj b/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj index eeee8b90f2..fd51c0e8bf 100644 --- a/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj +++ b/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj @@ -25,5 +25,13 @@ + + + + diff --git a/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj b/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj index 4aa0c95e8c..ffa2cd63d3 100644 --- a/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj +++ b/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj @@ -38,5 +38,13 @@ + + + + From 9c81559e3e8f89e9f1f6088a85a98b18cde473ee Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:40:41 +0100 Subject: [PATCH 40/68] Update project to use BenchmarkDotNet.TestAdapter Switched project reference and build file imports from BenchmarkDotNet.TestingPlatform to BenchmarkDotNet.TestAdapter, including .props and .targets files. --- ...BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj index 59913fd0b3..6cea87bd71 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -9,11 +9,12 @@ - + - - + + + From 8b128f593a4b152806169426e39a5e63cbf45e79 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:40:57 +0100 Subject: [PATCH 41/68] Update packaging and entry point logic for test adapter Add NuGet description and set IsTestingPlatformApplication to false to avoid treating the adapter as a test app. Add Microsoft.Testing.Platform.MSBuild as a dependency for downstream projects. Update .props and .targets packaging for cross-platform compatibility. Only generate entry point for VSTest scenarios; clarify comments. --- .../BenchmarkDotNet.TestAdapter.csproj | 20 ++++++++++++++++++- .../build/BenchmarkDotNet.TestAdapter.props | 14 ++++++++----- 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj b/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj index 3deb70b1cd..7cdb59410a 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj @@ -5,8 +5,16 @@ BenchmarkDotNet.TestAdapter BenchmarkDotNet.TestAdapter BenchmarkDotNet.TestAdapter + Runs BenchmarkDotNet benchmarks as tests, through Microsoft.Testing.Platform or VSTest README.md True + + + false @@ -20,6 +28,13 @@ + + + + + @@ -28,7 +43,10 @@ - + + + diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props index 7184842aba..5d1eabd4f6 100644 --- a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props @@ -1,15 +1,19 @@ - + $(MSBuildThisFileDirectory)..\entrypoints\ false - - + From bd91fe3638c6ff970adc151c3366caf306287a4c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:41:19 +0100 Subject: [PATCH 42/68] Remove BenchmarkDotNet.TestingPlatform project Removed BenchmarkDotNet.TestingPlatform from the solution and deleted its InternalsVisibleTo entry from AssemblyInfo.cs, as it no longer requires access to internal members. No other InternalsVisibleTo changes were made. --- BenchmarkDotNet.slnx | 3 --- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 3 +-- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index ec83125a62..f681138fe7 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -21,9 +21,6 @@ - - - diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index 7bd16f9307..4b0041b9e3 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -14,5 +14,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.Diagnostics.dotMemory,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From fe3143e2f8c670050dbb1028331b86b41c6dc91a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 16:55:55 +0100 Subject: [PATCH 43/68] Simplify project entry in BenchmarkDotNet.slnx Replaced the explicit configuration for BenchmarkDotNet.IntegrationTests.TestingPlatform with a standard entry to align with the format used for other projects in the solution. --- BenchmarkDotNet.slnx | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index f681138fe7..c3c733fb82 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -42,9 +42,7 @@ - - - + From 857dc665d3f2213c11466e11dab3eb9987a9d1ea Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 18:32:28 +0100 Subject: [PATCH 44/68] Update assistant introduction message Replaced default introduction with a personalized message identifying as GitHub Copilot and offering software development assistance. --- .../Exporters/FullNameProvider.cs | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 19555908d2..5411f7243b 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -99,6 +99,22 @@ internal static string GetMethodName(BenchmarkCase benchmarkCase) return name.ToString(); } + /// + /// Gets the method name to show to a user, which is the [Benchmark(Description = ...)] when one is set and + /// the method name otherwise, followed by the parameters. + /// + /// The benchmark case. + /// The method name to display. + internal static string GetMethodDisplayName(BenchmarkCase benchmarkCase) + { + var name = new StringBuilder(benchmarkCase.Descriptor.WorkloadMethodDisplayInfo); + + if (benchmarkCase.HasParameters) + name.Append(GetBenchmarkParameters(benchmarkCase.Descriptor.WorkloadMethod, benchmarkCase.Parameters)); + + return name.ToString(); + } + private static string GetBenchmarkParameters(MethodInfo method, ParameterInstances benchmarkParameters) { var methodArguments = method.GetParameters(); From 580eb353eff527e25974ca149c0b03933ebe7732 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 18:33:05 +0100 Subject: [PATCH 45/68] Refactor method display name handling in BenchmarkDotNet Refactored the logic for generating method display names in BenchmarkDotNet. Improved separation of concerns by extracting display name generation to a dedicated provider. Updated the display name formatting to include job information conditionally. Enhanced maintainability and clarity in the test adapter's method identification process. --- .../TestingPlatform/BenchmarkTestNode.cs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index 14f6bd171e..87b9b820a5 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -70,7 +70,13 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include // keeps the same identity across processes and across tools. The job is only part of the display name // when it actually adds information. var uid = benchmarkCase.GetUniqueId(); - var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); + + // Microsoft.Testing.Platform keeps the display name and the identity apart, so the name is free to be the + // [Benchmark(Description = ...)] the author chose. GetMethodDisplayName falls back to the method name when + // no description is set. The path keeps the method name, so that a filter still matches what + // BenchmarkDotNet's own --filter matches. + var displayMethodName = FullNameProvider.GetMethodDisplayName(benchmarkCase); + var displayName = $"{fullClassName}.{displayMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); var properties = new List { From 5311e65ec7c8d89ffc302b19926ddd440e1dab31 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 18:33:26 +0100 Subject: [PATCH 46/68] Add DescribedProbe benchmark class with custom config Introduced DescribedProbe to test benchmarks with and without custom descriptions. Includes FastConfig for quick in-process execution and a configurable Size parameter. --- .../DescribedProbe.cs | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs new file mode 100644 index 0000000000..72922bc2a7 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs @@ -0,0 +1,28 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// Benchmarks whose display name comes from the description rather than the method name. + /// + [Config(typeof(FastConfig))] + public class DescribedProbe + { + [Params(1)] + public int Size { get; set; } + + [Benchmark(Description = "A described benchmark")] + public int Described() => Size; + + [Benchmark] + public int Undescribed() => Size; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From fdeac602c0ee7bb8a49ff3eedff177210c439f74 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 30 Aug 2026 18:50:02 +0100 Subject: [PATCH 47/68] Add BenchmarkDotNet.IntegrationTests.TestingPlatform to tests Added tests/BenchmarkDotNet.IntegrationTests.TestingPlatform to run-tests-selected.yaml. Included a comment clarifying that this project is a Microsoft.Testing.Platform application, relies on global.json for dotnet test routing, and requires the workflow to set the working directory for correct resolution. --- .github/workflows/run-tests-selected.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/run-tests-selected.yaml b/.github/workflows/run-tests-selected.yaml index 0e2f436810..e4e8be56fe 100644 --- a/.github/workflows/run-tests-selected.yaml +++ b/.github/workflows/run-tests-selected.yaml @@ -26,6 +26,10 @@ on: - tests/BenchmarkDotNet.IntegrationTests - tests/BenchmarkDotNet.IntegrationTests.ManualRunning - tests/BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks + # This one is a Microsoft.Testing.Platform application. It carries the `global.json` which routes + # `dotnet test` to the platform, and that file is resolved from the working directory, which this + # workflow sets to the project. + - tests/BenchmarkDotNet.IntegrationTests.TestingPlatform - samples/BenchmarkDotNet.Samples framework: type: choice From b6e386d1d92a9c4bd549cc50dc5a0f8826e981a8 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 30 Aug 2026 18:51:56 +0100 Subject: [PATCH 48/68] Format: Re-add InternalsVisibleTo line without changes No functional changes; removed and immediately re-added the InternalsVisibleTo attribute line for formatting consistency. --- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index 4b0041b9e3..aa93984520 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -14,4 +14,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.Diagnostics.dotMemory,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 32a3793e143cdf01af7f889e7205345b50994694 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 30 Aug 2026 18:52:11 +0100 Subject: [PATCH 49/68] Improve MSBuild property logic in TestAdapter.targets Update logic for IsTestingPlatformApplication and GenerateProgramFile to ensure correct opt-out handling for Microsoft.Testing.Platform. Explicitly set IsTestingPlatformApplication to false when BenchmarkDotNetUseVSTest is true or GenerateProgramFile is false, and default to true otherwise. Set GenerateProgramFile to false when IsTestingPlatformApplication is true to prevent entry point conflicts. Add comments to clarify the changes. --- .../build/BenchmarkDotNet.TestAdapter.targets | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets index 339a7fd56b..09d71557cf 100644 --- a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets @@ -13,11 +13,26 @@ * GenerateProgramFile, which is how a project says it already has an entry point of its own, typically one that calls BenchmarkSwitcher. Generating a second one would not compile. --> - true - false + + false + true true + + + false + + + net10.0 + Exe + enable + enable + + $(MSBuildThisFileDirectory)..\..\..\artifacts + + + + + + + + From f1b10dee9b7162887dd25197c0b6fe017e7210c9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:57:28 +0100 Subject: [PATCH 56/68] Add .csproj for TestingPlatform.Failures integration tests A new BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj was added targeting .NET 10.0 as an executable. The project includes assembly metadata, enforces code optimization for consistent benchmarks, and references BenchmarkDotNet.TestAdapter with manual imports of its .props and .targets files. Common build property and target files are also imported to ensure correct MSBuild behavior. --- ...ationTests.TestingPlatform.Failures.csproj | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj new file mode 100644 index 0000000000..610c70c194 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj @@ -0,0 +1,32 @@ + + + + net10.0 + Exe + BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + + + true + + + + + + + + + + + + From 5ff52b24a947f677eef896482923bb5e88d0a28c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:57:46 +0100 Subject: [PATCH 57/68] Add SeparatorProbe benchmark with FastConfig setup Added SeparatorProbe class in BenchmarkDotNet.IntegrationTests.TestingPlatform. This benchmark uses a parameter with '/' as a tree separator, includes a Length() method, and applies a custom FastConfig with a dry job and InProcessEmitToolchain for faster execution. --- .../SeparatorProbe.cs | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs new file mode 100644 index 0000000000..ab01cae901 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs @@ -0,0 +1,26 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose parameter contains the character Microsoft.Testing.Platform uses to separate the levels of the + /// tree a --treenode-filter walks. It has to stay at the same level of that tree as every other benchmark. + /// + [Config(typeof(FastConfig))] + public class SeparatorProbe + { + [Params("a/b")] + public string Value { get; set; } = ""; + + [Benchmark] + public int Length() => Value.Length; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 53c3d60756c3e617da4a41889cd89cccee50b4a9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:00 +0100 Subject: [PATCH 58/68] Add BuildFailureProbe for build failure testing Introduce BuildFailureProbe in BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures. This class uses a custom toolchain (FailingBuildConfig) with a NoopGenerator, FailingBuilder, and UnreachableExecutor to reliably simulate build failures for adapter testing, without relying on uncompilable code. --- .../BuildFailureProbe.cs | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs new file mode 100644 index 0000000000..3665f8592a --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs @@ -0,0 +1,50 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains; +using BenchmarkDotNet.Toolchains.Parameters; +using BenchmarkDotNet.Toolchains.Results; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures +{ + /// + /// A benchmark whose build always fails, so that the adapter has to turn the build failure into a failed test. + /// The failure is faked by the toolchain rather than by uncompilable code, which keeps it quick and keeps the + /// error message the test asserts on under this file's control. + /// + [Config(typeof(FailingBuildConfig))] + public class BuildFailureProbe + { + internal const string ErrorMessage = "The build of this benchmark always fails, on purpose."; + + [Benchmark] + public int Add() => 1 + 1; + + private class FailingBuildConfig : ManualConfig + { + public FailingBuildConfig() + => AddJob(Job.Dry.WithToolchain(new Toolchain("FailingBuild", new NoopGenerator(), new FailingBuilder(), new UnreachableExecutor()))); + } + + private sealed class NoopGenerator : IGenerator + { + public ValueTask GenerateProjectAsync(BuildPartition buildPartition, ILogger logger, string rootArtifactsFolderPath, CancellationToken cancellationToken) + => new(GenerateResult.Success(ArtifactsPaths.Empty, [])); + } + + private sealed class FailingBuilder : IBuilder + { + public ValueTask BuildAsync(GenerateResult generateResult, BuildPartition buildPartition, ILogger logger, CancellationToken cancellationToken) + => new(BuildResult.Failure(generateResult, ErrorMessage)); + } + + private sealed class UnreachableExecutor : IExecutor + { + // A benchmark that failed to build is never executed. + public ValueTask ExecuteAsync(ExecuteParameters executeParameters, CancellationToken cancellationToken) + => throw new InvalidOperationException("The benchmark should never have been executed."); + } + } +} From c30b2ce0f19bf3dbb8ab053582c6b2f0872d3235 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:17 +0100 Subject: [PATCH 59/68] Add CollisionProbe test for parameter collision handling Added CollisionProbe class to test BenchmarkDotNet's behavior when benchmark parameters have identical string representations, using a custom Ambiguous type. Ensures the adapter reports collisions instead of running ambiguous benchmarks. --- .../CollisionProbe.cs | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs new file mode 100644 index 0000000000..f40a600703 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs @@ -0,0 +1,36 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures +{ + /// + /// Two benchmark cases that BenchmarkDotNet identifies as one, because a benchmark is identified by the string + /// representation of its parameters and both values stringify the same way. The platform cannot tell the two apart + /// either, so the adapter is expected to report the collision instead of running them. + /// + [Config(typeof(FastConfig))] + public class CollisionProbe + { + public IEnumerable Values => [new Ambiguous(1), new Ambiguous(2)]; + + [ParamsSource(nameof(Values))] + public Ambiguous? Value { get; set; } + + [Benchmark] + public int Identity() => Value!.Number; + + public class Ambiguous(int number) + { + public int Number { get; } = number; + + public override string ToString() => "ambiguous"; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 8b4994ede2a0bcb8689a978ecb4d5721e019c271 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:32 +0100 Subject: [PATCH 60/68] Add OutOfProcessProbe for out-of-process benchmark test Added OutOfProcessProbe class in BenchmarkDotNet.IntegrationTests.TestingPlatform with an Add() benchmark method. Configured to run out-of-process using a custom OutOfProcessConfig and Job.Dry to ensure a real build/execute cycle for adapter testing, unlike in-process probes. --- .../OutOfProcessProbe.cs | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs new file mode 100644 index 0000000000..cba4a1748b --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs @@ -0,0 +1,25 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark that runs out of process, on the default toolchain. The other probes stay in process to keep the run + /// fast, which skips the generate/build/execute cycle entirely, so this one is what makes the adapter see a real + /// . + /// + [Config(typeof(OutOfProcessConfig))] + public class OutOfProcessProbe + { + [Benchmark] + public int Add() => 1 + 1; + + private class OutOfProcessConfig : ManualConfig + { + // A dry job on the default toolchain: one iteration, but a separate executable is still generated, built + // and run. + public OutOfProcessConfig() => AddJob(Job.Dry); + } + } +} From 09485e9ee072cc306cd141889cb16baf11247ddc Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:53 +0100 Subject: [PATCH 61/68] Add conditional refs for TestingPlatform probe projects Added conditional project references to BenchmarkDotNet.IntegrationTests.TestingPlatform and .Failures in BenchmarkDotNet.IntegrationTests.csproj. These are included only for .NETCoreApp targets with ReferenceOutputAssembly set to false, ensuring correct build order for probe apps used in TestingPlatformAdapterTests. --- .../BenchmarkDotNet.IntegrationTests.csproj | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj index 768cccaaa7..fbb0355c9c 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj @@ -39,6 +39,15 @@ + + + + + From 22759aa1574f4faea526a820d2c2d22dc8614c45 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:22 +0100 Subject: [PATCH 62/68] Add TestingPlatformAdapterTests for integration testing Added TestingPlatformAdapterTests (under #if NETCOREAPP) using BenchmarkDotNet to perform integration tests on Microsoft.Testing.Platform probe apps. Tests cover benchmark discovery, UID consistency, filtering, build/run behavior, and error reporting by running probe apps as separate processes and asserting on their output. Introduced helper methods for process execution, output parsing, and result summarization. --- .../TestingPlatformAdapterTests.cs | 258 ++++++++++++++++++ 1 file changed, 258 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs new file mode 100644 index 0000000000..2e8520bf77 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -0,0 +1,258 @@ +#if NETCOREAPP +using BenchmarkDotNet.Detectors; +using System.Diagnostics; +using System.Text; +using System.Text.Json; +using System.Text.RegularExpressions; + +namespace BenchmarkDotNet.IntegrationTests +{ + /// + /// Drives the two Microsoft.Testing.Platform probe applications through their command line and asserts on what + /// BenchmarkDotNet.TestAdapter reports back. Everything here goes through a separate process on purpose: the + /// adapter's job is to keep a benchmark identifiable and addressable from the outside, and discovery and execution + /// are two different processes when a test runner drives it. + /// + public class TestingPlatformAdapterTests(ITestOutputHelper output) + { + private const string PassingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform"; + private const string FailingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures"; + + // Both probe projects are single targeted, see their .csproj files. + private const string ProbeTargetFramework = "net10.0"; + + // A run that has to build a benchmark pays for a restore and a build of the generated project. + private static readonly TimeSpan Timeout = TimeSpan.FromMinutes(10); + + [Fact] + public void EveryBenchmarkIsDiscoveredUnderItsOwnName() + { + string[] expected = + [ + // The description of a [Benchmark(Description = ...)] is what a user recognises it by, so it is used + // instead of the method name. Without one the method name is used, and the parameters are appended to + // both. + "DescribedProbe.'A described benchmark'(Size: 1)", + "DescribedProbe.Undescribed(Size: 1)", + + // A generic benchmark is named after the type arguments it was closed over. + "GenericProbe.Create", + "GenericProbe>.Create", + "GenericProbe.Create", + + "OutOfProcessProbe.Add", + "SampleBenchmarks.Add(Size: 1)", + "SampleBenchmarks.Add(Size: 2)", + "SampleBenchmarks.Multiply(Size: 1)", + "SampleBenchmarks.Multiply(Size: 2)", + "SeparatorProbe.Length(Value: \"a/b\")", + ]; + + var discovered = Discover(PassingProbes); + + Assert.Equal( + expected, + discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); + + // The platform identifies a node by its uid, so two benchmarks sharing one cannot be told apart. + Assert.Equal(discovered.Count, discovered.Select(test => test.Uid).Distinct().Count()); + } + + [Fact] + public void TheUidOfABenchmarkIsTheSameInEveryProcess() + { + var first = Discover(PassingProbes).ToDictionary(test => test.Uid, test => test.DisplayName); + var second = Discover(PassingProbes).ToDictionary(test => test.Uid, test => test.DisplayName); + + Assert.Equal(first, second); + } + + [Fact] + public void ABenchmarkCanBeRunByTheUidItWasDiscoveredWith() + { + // This is the contract a test runner relies on: it discovers in one process and asks for a uid in another. + var uid = Discover(PassingProbes) + .Single(test => test.DisplayName.EndsWith("SampleBenchmarks.Add(Size: 2)", StringComparison.Ordinal)) + .Uid; + + var summary = RunAndSummarize(PassingProbes, "--filter-uid", uid); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Succeeded); + Assert.Equal(0, summary.Failed); + } + + [Fact] + public void ATreeNodeFilterMatchesTheCategoriesOfABenchmark() + { + // The categories are published as filterable properties, which is what makes this expression work. + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/*/*[Category=Fast]"); + + Assert.Equal( + new[] { "SampleBenchmarks.Add(Size: 1)", "SampleBenchmarks.Add(Size: 2)" }, + discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); + } + + [Fact] + public void ATreeNodeFilterMatchesTheClassAndTheMethodOfABenchmark() + { + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/SampleBenchmarks/Multiply*"); + + Assert.Equal(2, discovered.Count); + Assert.All(discovered, test => Assert.Contains("SampleBenchmarks.Multiply", test.DisplayName, StringComparison.Ordinal)); + } + + [Fact] + public void ABenchmarkStaysAtTheSameLevelOfTheTreeWhenAParameterContainsTheSeparator() + { + // The platform splits the tree path on every '/' and never unescapes it, so a parameter containing one has + // to be encoded rather than escaped: otherwise the benchmark sits one level deeper and this filter, which + // matches every other benchmark, would miss it. + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/SeparatorProbe/*"); + + Assert.Single(discovered); + Assert.Contains("SeparatorProbe.Length(Value: \"a/b\")", discovered[0].DisplayName, StringComparison.Ordinal); + } + + [Fact] + public void AnOutOfProcessBenchmarkIsBuiltAndRun() + { + // The only probe that is not pinned to an in-process toolchain, so the only one that makes the adapter see + // a real generate/build/execute cycle. + var summary = RunAndSummarize(PassingProbes, "--treenode-filter", "/*/*/OutOfProcessProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Succeeded); + Assert.Equal(0, summary.Failed); + } + + [Fact] + public void ABuildFailureIsReportedAsAFailedTest() + { + var (summary, standardOutput) = Run(FailingProbes, "--treenode-filter", "/*/*/BuildFailureProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Failed); + Assert.Contains("// Build Error: The build of this benchmark always fails, on purpose.", standardOutput, StringComparison.Ordinal); + } + + [Fact] + public void BenchmarksSharingAUidAreReportedAsOneFailedTest() + { + // Two benchmarks the platform cannot tell apart are published as a single node during discovery, and the + // collision is reported when they are asked to run. + Assert.Single(Discover(FailingProbes, "--treenode-filter", "/*/*/CollisionProbe/*")); + + var (summary, standardOutput) = Run(FailingProbes, "--treenode-filter", "/*/*/CollisionProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Failed); + Assert.Contains("2 benchmarks are identified as", standardOutput, StringComparison.Ordinal); + } + + private IReadOnlyList Discover(string project, params string[] arguments) + { + var (exitCode, standardOutput) = Execute(project, ["--list-tests", "json", .. arguments]); + + Assert.Equal(0, exitCode); + + using var document = JsonDocument.Parse(standardOutput); + + return document.RootElement.GetProperty("tests") + .EnumerateArray() + .Select(test => new DiscoveredTest(test.GetProperty("uid").GetString()!, test.GetProperty("displayName").GetString()!)) + .ToArray(); + } + + private TestRunSummary RunAndSummarize(string project, params string[] arguments) => Run(project, arguments).Summary; + + private (TestRunSummary Summary, string StandardOutput) Run(string project, params string[] arguments) + { + var (_, standardOutput) = Execute(project, arguments); + + return (TestRunSummary.Parse(standardOutput), standardOutput); + } + + private (int ExitCode, string StandardOutput) Execute(string project, string[] arguments) + { + var application = GetProbeApplication(project); + var startInfo = new ProcessStartInfo(application) + { + WorkingDirectory = Path.GetDirectoryName(application), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + + // Progress reporting redraws the screen in place, which is noise once the output is redirected. + foreach (var argument in arguments.Concat(["--no-ansi", "--progress", "off"])) + startInfo.ArgumentList.Add(argument); + + var standardOutput = new StringBuilder(); + var standardError = new StringBuilder(); + + using var process = new Process { StartInfo = startInfo }; + process.OutputDataReceived += (_, e) => { if (e.Data != null) lock (standardOutput) standardOutput.AppendLine(e.Data); }; + process.ErrorDataReceived += (_, e) => { if (e.Data != null) lock (standardError) standardError.AppendLine(e.Data); }; + + process.Start(); + process.BeginOutputReadLine(); + process.BeginErrorReadLine(); + + if (!process.WaitForExit((int)Timeout.TotalMilliseconds)) + { + process.Kill(entireProcessTree: true); + throw new TimeoutException($"'{Path.GetFileName(application)} {string.Join(" ", arguments)}' did not finish within {Timeout}."); + } + + // Lets the redirected output be flushed before it is read. + process.WaitForExit(); + + output.WriteLine($"$ {application} {string.Join(" ", startInfo.ArgumentList)}"); + output.WriteLine(standardOutput.ToString()); + + if (standardError.Length > 0) + output.WriteLine($"stderr:{Environment.NewLine}{standardError}"); + + return (process.ExitCode, standardOutput.ToString()); + } + + private static string GetProbeApplication(string project) + { + // The tests run from /tests/BenchmarkDotNet.IntegrationTests/bin///, and + // the probes are built next to them, by the ProjectReferences of this project. + var binaries = new DirectoryInfo(AppContext.BaseDirectory); + var configuration = binaries.Parent!.Name; + var testsFolder = binaries.Parent!.Parent!.Parent!.Parent!.FullName; + + var fileName = OsDetector.IsWindows() ? $"{project}.exe" : project; + var path = Path.Combine(testsFolder, project, "bin", configuration, ProbeTargetFramework, fileName); + + if (!File.Exists(path)) + throw new FileNotFoundException($"The probe application was not built. Expected it at '{path}'.", path); + + return path; + } + + private sealed record DiscoveredTest(string Uid, string DisplayName); + + private sealed record TestRunSummary(int Total, int Failed, int Succeeded, int Skipped) + { + public static TestRunSummary Parse(string standardOutput) + { + // The platform ends a run with a block of " : " lines under "Test run summary:". + int Read(string name) + { + var match = Regex.Match(standardOutput, $@"^\s*{name}:\s*(?\d+)\s*$", RegexOptions.Multiline); + + return match.Success + ? int.Parse(match.Groups["count"].Value) + : throw new InvalidOperationException($"The test run did not report a '{name}' count.{Environment.NewLine}{standardOutput}"); + } + + return new TestRunSummary(Read("total"), Read("failed"), Read("succeeded"), Read("skipped")); + } + } + } +} +#endif From af7c0190ff356e3a589bd6fe01ef214cea0d0384 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:38 +0100 Subject: [PATCH 63/68] Escape '/' as '%2F' in BenchmarkTestNode parameters Updated the Escape method in BenchmarkTestNode.cs to percent-encode '/' as '%2F' and '%' as '%25'. This prevents path segmentation issues in Microsoft.Testing.Platform, ensuring correct tree structure and benchmark addressability. Filters must now use '%2F' instead of '/'. --- .../TestingPlatform/BenchmarkTestNode.cs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index 764067fb67..9d79bd7f35 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -150,7 +150,11 @@ private static string BuildPath(Assembly assembly, string? @namespace, string fu .ToString(); } - // Benchmark parameters are stringified user values, so they can contain the path separator. - private static string Escape(string segment) => segment.Replace("/", "\\/"); + // Benchmark parameters are stringified user values, so they can contain the path separator. A '/' cannot be + // escaped into a segment: Microsoft.Testing.Platform splits the path on every '/' without ever unescaping it, + // and TreeNodeFilter rejects a filter whose segment contains one, so a raw '/' would both deepen the tree and + // leave the benchmark unmatchable. Percent encoding keeps the path four levels deep and the segment + // addressable, at the price of a filter having to spell the separator as '%2F'. + private static string Escape(string segment) => segment.Replace("%", "%25").Replace("/", "%2F"); } } From ba88cc9761b22fc8150e381c7b5a1eaa88edd84e Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:48 +0100 Subject: [PATCH 64/68] Document percent-encoding in filter path Updated documentation to clarify that benchmark parameter values containing slashes (/) or percent signs (%) are percent-encoded in the tree node filter path (e.g., a/b as a%2Fb, % as %25). This encoding applies only to the filter, not to the displayed benchmark name. --- docs/articles/features/testadapter.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/articles/features/testadapter.md b/docs/articles/features/testadapter.md index 90e159768b..8a091886fb 100644 --- a/docs/articles/features/testadapter.md +++ b/docs/articles/features/testadapter.md @@ -132,6 +132,10 @@ dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" The tree node filter path is `////`, and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. +Because the platform separates the levels of that path with `/`, a benchmark parameter whose value contains one is + percent encoded in the path: a parameter value of `a/b` is written `a%2Fb` in a filter, and a literal `%` is + written `%25`. +This affects the filter only; the name the benchmark is displayed under is unchanged. ## Keeping your own entry point From 61fa92eb615dce35bf5b380c3ffc25aedb2378bc Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:58 +0100 Subject: [PATCH 65/68] Add smoke test script for BenchmarkDotNet.TestAdapter Added test-adapter-consumer.ps1 to perform smoke tests on the packed BenchmarkDotNet.TestAdapter NuGet package. The script restores and builds a consumer project, checks MSBuild property resolutions, and verifies benchmark discovery to ensure correct adapter behavior when used as a package. Includes detailed comments, parameter handling, and error checking. --- build/smoke-tests/test-adapter-consumer.ps1 | 115 ++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 build/smoke-tests/test-adapter-consumer.ps1 diff --git a/build/smoke-tests/test-adapter-consumer.ps1 b/build/smoke-tests/test-adapter-consumer.ps1 new file mode 100644 index 0000000000..47c9e586a5 --- /dev/null +++ b/build/smoke-tests/test-adapter-consumer.ps1 @@ -0,0 +1,115 @@ +#!/usr/bin/env pwsh + +<# +.SYNOPSIS + Smoke tests the packed BenchmarkDotNet.TestAdapter against a project that consumes it as a NuGet package. + +.DESCRIPTION + Everything in this repository that uses the adapter's build files imports them by path from the project file, + which MSBuild evaluates before nuget.g.targets. A package consumer gets the opposite order: NuGet imports + Microsoft.Testing.Platform.MSBuild's targets, which default IsTestingPlatformApplication to true, and only then + the adapter's, which have to overwrite that default for the opt-outs to work. Nothing in the solution can + reproduce that order, so this restores the real package and asserts on how the properties resolve. + + Run `build.cmd pack` first, so that the packages exist. + +.PARAMETER ArtifactsDirectory + The directory `build.cmd pack` wrote the packages to. + +.PARAMETER Configuration + The configuration to build the consuming project in. +#> + +[CmdletBinding()] +param( + [string] $ArtifactsDirectory = [System.IO.Path]::Combine($PSScriptRoot, '..', '..', 'artifacts'), + [string] $Configuration = 'Release' +) + +$ErrorActionPreference = 'Stop' +$PSNativeCommandUseErrorActionPreference = $true + +$project = [System.IO.Path]::Combine($PSScriptRoot, 'TestAdapterConsumer', 'TestAdapterConsumer.csproj') +$targetFramework = 'net10.0' + +# $IsWindows only exists on PowerShell Core, where it is the only way to tell; Windows PowerShell is Windows by definition. +$onWindows = ($null -eq $IsWindows) -or $IsWindows + +# build.cmd installs the SDK the repository is pinned to into .dotnet, and only puts it on PATH for its own run. +$dotnet = [System.IO.Path]::Combine($PSScriptRoot, '..', '..', '.dotnet', $(if ($onWindows) { 'dotnet.exe' } else { 'dotnet' })) +if (-not (Test-Path $dotnet)) { + $dotnet = 'dotnet' +} + +$package = Get-ChildItem -Path $ArtifactsDirectory -Filter 'BenchmarkDotNet.TestAdapter.*.nupkg' | + Where-Object { $_.Name -notlike '*.symbols.nupkg' } | + Select-Object -First 1 + +if ($null -eq $package) { + throw "No BenchmarkDotNet.TestAdapter package was found in '$ArtifactsDirectory'. Run 'build.cmd pack' first." +} + +$version = $package.BaseName -replace '^BenchmarkDotNet\.TestAdapter\.', '' +Write-Output "Consuming BenchmarkDotNet.TestAdapter $version from $ArtifactsDirectory" + +function Invoke-Dotnet { + param([Parameter(ValueFromRemainingArguments = $true)] [string[]] $Arguments) + + $output = & $dotnet @Arguments 2>&1 | Out-String + + if ($LASTEXITCODE -ne 0) { + Write-Output $output + throw "'dotnet $($Arguments -join ' ')' failed with exit code $LASTEXITCODE." + } + + return $output +} + +function Assert-Property { + param( + [string] $Name, + [string] $Expected, + [string[]] $With = @() + ) + + $arguments = @($project, '-nologo', '-tl:off', "-p:BenchmarkDotNetVersion=$version", "-p:Configuration=$Configuration") + $With + @("-getProperty:$Name") + $actual = (Invoke-Dotnet msbuild @arguments).Trim() + + $description = if ($With.Count -eq 0) { 'by default' } else { "with $($With -join ' ')" } + + if ($actual -ne $Expected) { + throw "Expected $Name to be '$Expected' $description, but it was '$actual'." + } + + Write-Output " OK: $Name is '$Expected' $description" +} + +Write-Output '##[group]Restoring the consuming project' +Invoke-Dotnet restore $project "-p:BenchmarkDotNetVersion=$version" '-tl:off' | Write-Output +Write-Output '##[endgroup]' + +Write-Output 'Checking how the packaged build files resolve the test platform:' + +# Microsoft.Testing.Platform is the default, and the adapter leaves the entry point to it. +Assert-Property -Name 'IsTestingPlatformApplication' -Expected 'true' +Assert-Property -Name 'GenerateProgramFile' -Expected 'false' + +# The two opt-outs have to win over the default Microsoft.Testing.Platform.MSBuild sets in its own targets, which a +# package consumer imports before the adapter's. +Assert-Property -Name 'IsTestingPlatformApplication' -Expected 'false' -With '-p:BenchmarkDotNetUseVSTest=true' +Assert-Property -Name 'IsTestingPlatformApplication' -Expected 'false' -With '-p:GenerateProgramFile=false' + +Write-Output '##[group]Building the consuming project' +Invoke-Dotnet build $project '--no-restore' '-c' $Configuration "-p:BenchmarkDotNetVersion=$version" '-tl:off' | Write-Output +Write-Output '##[endgroup]' + +Write-Output 'Listing the benchmarks through the entry point Microsoft.Testing.Platform generated:' +$application = [System.IO.Path]::Combine($PSScriptRoot, 'TestAdapterConsumer', 'bin', $Configuration, $targetFramework, 'TestAdapterConsumer.dll') +$listed = Invoke-Dotnet $application '--list-tests' '--no-ansi' +Write-Output $listed + +if ($listed -notmatch 'TestAdapterConsumer\.ConsumedBenchmark\.Add') { + throw 'The packaged adapter did not list the benchmark of the consuming project.' +} + +Write-Output 'The packaged BenchmarkDotNet.TestAdapter behaves as expected.' From 8745d4aa27fc9ad2484e533e08a25a83470224b4 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 12:00:13 +0100 Subject: [PATCH 66/68] Enable optimizations in project file for benchmarks Add true to ensure the assembly is always built with optimizations, preventing BenchmarkEnumerator from hiding out-of-process benchmarks in non-Release builds. Expand comments to clarify manual build file imports and MSBuild processing order. --- ...kDotNet.IntegrationTests.TestingPlatform.csproj | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj index 6cea87bd71..cdbdbc7edc 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -6,13 +6,25 @@ BenchmarkDotNet.IntegrationTests.TestingPlatform BenchmarkDotNet.IntegrationTests.TestingPlatform BenchmarkDotNet.IntegrationTests.TestingPlatform + + + true - + From 3ad898433b620ad3a036a672755355ca21ef8606 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 12:00:26 +0100 Subject: [PATCH 67/68] Document TestingPlatform.Failures project in README Added a section to README.md describing the BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures project. The documentation explains its purpose as a collection of intentionally failing benchmarks for testing BenchmarkDotNet.TestAdapter error handling, including UID collision and build failure mapping. It also clarifies the relationship with TestingPlatformAdapterTests and the location of passing benchmarks. --- .../README.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md new file mode 100644 index 0000000000..1905d04e12 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md @@ -0,0 +1,9 @@ +# BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + +Every benchmark in this Microsoft.Testing.Platform application is expected to be reported as failed, so a plain +`dotnet test` over it fails by design. It exists for the paths of `BenchmarkDotNet.TestAdapter` that only a broken +benchmark reaches: the uid collision report and the mapping of a build failure onto failed tests. + +`TestingPlatformAdapterTests` in `BenchmarkDotNet.IntegrationTests` drives it, one probe at a time, and asserts on +what the platform reports. The benchmarks that are expected to pass live in +`BenchmarkDotNet.IntegrationTests.TestingPlatform` instead. From 67c9996016cf41558a9856c457fd7f0ea81f0e84 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 12:00:37 +0100 Subject: [PATCH 68/68] Add Failures test project to BenchmarkDotNet.slnx Added BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures to the solution file, ensuring it is included with other integration test projects. --- BenchmarkDotNet.slnx | 1 + 1 file changed, 1 insertion(+) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index c3c733fb82..f6801feaba 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -43,6 +43,7 @@ +