BenchmarkDotNet

BenchmarkDotNet: Get Started with Benchmarking Your C# Code

by

Intro

This article will guide you through creating and running benchmarks. Expect to learn how to set up your first benchmarking project, understand the basics of benchmark design, and straightforwardly interpret the results.

BenchmarkDotNet is an open-source library that makes it easy for you as a developer to create benchmarks as a console application. The library became quite popular among .NET developers. That has to do mainly with the ease of use and its ability to provide reliable results out of the box. How you write the benchmark scenarios is very similar to unit tests and, therefore, already feels familiar to most developers. The rest is up to the BenchmarkDotNet. It runs the benchmarks. It figures out the number of executions needed to provide steady results. Then, those results are presented in a human-readable form directly in the console. CSV, Markdown and HTML output are in a separate folder.

The ease of use is highly subjective and should be compared to manually writing reliable benchmarks, which you do not have to do.

Micro Benchmarks

The primary use case of BenchmarkDotNet is to perform micro-benchmarks. You want to test a piece of code instead of a system or whole parts of it. Typically, you want to compare two algorithms or data structures and how they perform in terms of memory consumption and duration of the execution.

Can I use BenchmarkDotNet for REST API Testing?

People are using it to test the performance of their Web APIs (REST). However, using BenchmarkDotNet for this kind of test is a less common approach.

The unpredictable nature of network latency and external dependencies might make a REST API, not an excellent candidate to test with BenchmarkDotNet.

Also, you would have to add more boilerplate code to your BDN project to test an API than you would have to by using a tailored tool for API tests.

Before We Start

In case you encounter issues with slow performance or high memory consumption within your application. Consider identifying first where the weakness in your code is. Once determined, you extract the problem and test different scenarios to solve it.

Suppose you identify a path of code execution in your application that takes too long or uses too much memory. In that case, you can use BenchmarkDotNet to compare different approaches to solving the given problem differently.

So, using a profiler in your IDE or stopwatches in your code is still perfectly fine and hasn’t been replaced by BenchmarkDotNet.

The following sections will focus on how to start with BenchmarkDotNet in a practical step-by-step manner with Visual Studio.

Project Template and NuGet

You may get the latest version of the library from Nuget. That’s all you need. Also, using a template to get started even quicker is helpful.

To install the NuGet package to an existing console application, you can use this command from Visual Studio’s package manager console:

PM> NuGet\Install-Package BenchmarkDotNet

To get started with a simple template, first install the project template from the command line:

dotnet new install BenchmarkDotNet.Templates

If you install the template with dotnet new install, it will be available from the command line and within Visual Studio. Search for “Benchmark” after installing it from the command line. Another template is available from within Visual Studio that you can install via the extensions. It is slightly different. However, I recommend the official template here to get started.

That’s how the official template looks from the new project dialogue within Visual Studio:

Designing the Benchmark

You will find a simple console application after you create the project from the template.

These are the files automatically created by the template:

Program.cs

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;

namespace BenchmarkSuite5;
public class Program
{
  public static void Main(string[] args)
  {
    var config = DefaultConfig.Instance;
    var summary = BenchmarkRunner.Run<Benchmarks>(config, args);

    // Use this to select benchmarks from the console:
    // var summaries = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);
  }
}

Benchmark.cs

Here, you implement your scenarios. Scenario methods are attributed with the [Benchmark] attribute. These are, so to say, your “unit tests”, the different scenarios that BenchmarkDotNet will compare for you.

using BenchmarkDotNet;
using BenchmarkDotNet.Attributes;
using System;

namespace BenchmarkSuite5;
public class Benchmarks
{
  [Benchmark]
  public void Scenario1()
  {
    // Implement your benchmark here
  }

  [Benchmark]
  public void Scenario2()
  {
    // Implement your benchmark here
  }
}

I am sure you will find good examples to create from your imagination. If you need more, see the examples in the examples section at the end of the article.

Running Benchmarks

Please run your benchmarks from the command line with everything closed. Open the console from within your project directory and use this command:

dotnet run -c Release

BenchmarkDotNet will take some time to perform the results. Suppose you do not change the default configuration. In that case, it will determine how often it executes your methods to achieve steady and reliable results.

Understanding Results

The following shows a result table of one of the examples I prepared previously. This article focuses mainly on the basics and how to get started.

Console Output

BenchmarkDotNet=v0.13.4, OS=Windows 11 (10.0.22635.2771)
11th Gen Intel Core i7-1165G7 2.80GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=8.0.100
[Host] : .NET 6.0.25 (6.0.2523.51912), X64 RyuJIT AVX2
.NET 7.0 : .NET 7.0.14 (7.0.1423.51910), X64 RyuJIT AVX2
.NET 8.0 : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

|                           Method |      Job |  Runtime |        Mean |       Error |      StdDev |  Ratio | RatioSD |
|--------------------------------- |--------- |--------- |------------:|------------:|------------:|-------:|--------:|
|      FirstOrDefault_Indexed_Last | .NET 7.0 | .NET 7.0 |    473.5 us |    25.18 us |    73.46 us |   0.81 |    0.18 |
|     FirstOrDefault_Indexed_First | .NET 7.0 | .NET 7.0 |    691.5 us |    44.69 us |   128.21 us |   1.17 |    0.27 |
|     SingleOrDefault_Indexed_Last | .NET 7.0 | .NET 7.0 |    598.8 us |    27.28 us |    80.43 us |   1.00 |    0.00 |
|    SingleOrDefault_Indexed_First | .NET 7.0 | .NET 7.0 |    579.9 us |    37.01 us |   109.13 us |   0.99 |    0.23 |
|   FirstOrDefault_NotIndexed_Last | .NET 7.0 | .NET 7.0 | 53,153.1 us | 2,782.34 us | 7,938.18 us |  90.75 |   20.56 |
|  FirstOrDefault_NotIndexed_First | .NET 7.0 | .NET 7.0 | 19,261.1 us |   812.04 us | 2,381.57 us |  32.81 |    6.14 |
|  SingleOrDefault_NotIndexed_Last | .NET 7.0 | .NET 7.0 | 55,952.9 us | 2,413.56 us | 7,002.18 us |  95.44 |   19.44 |
| SingleOrDefault_NotIndexed_First | .NET 7.0 | .NET 7.0 | 54,349.1 us | 2,026.78 us | 5,912.21 us |  92.26 |   14.75 |
|                                  |          |          |             |             |             |        |         |
|      FirstOrDefault_Indexed_Last | .NET 8.0 | .NET 8.0 |    479.3 us |    30.31 us |    89.36 us |   0.96 |    0.26 |
|     FirstOrDefault_Indexed_First | .NET 8.0 | .NET 8.0 |    493.3 us |    35.22 us |   103.85 us |   0.99 |    0.32 |
|     SingleOrDefault_Indexed_Last | .NET 8.0 | .NET 8.0 |    517.7 us |    30.65 us |    89.41 us |   1.00 |    0.00 |
|    SingleOrDefault_Indexed_First | .NET 8.0 | .NET 8.0 |    486.5 us |    35.18 us |   103.73 us |   0.96 |    0.26 |
|   FirstOrDefault_NotIndexed_Last | .NET 8.0 | .NET 8.0 | 51,649.6 us | 2,765.84 us | 7,980.08 us | 103.08 |   23.39 |
|  FirstOrDefault_NotIndexed_First | .NET 8.0 | .NET 8.0 | 19,363.5 us |   871.89 us | 2,570.80 us |  38.67 |    8.99 |
|  SingleOrDefault_NotIndexed_Last | .NET 8.0 | .NET 8.0 | 52,017.6 us | 2,065.92 us | 5,927.50 us | 104.34 |   24.74 |
| SingleOrDefault_NotIndexed_First | .NET 8.0 | .NET 8.0 | 54,610.0 us | 2,435.62 us | 7,066.18 us | 108.64 |   23.25 |

BDN will show you legends for the columns right in the console:

// * Legends *
  Mean    : Arithmetic mean of all measurements
  Error   : Half of 99.9% confidence interval
  StdDev  : Standard deviation of all measurements
  Ratio   : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD : Standard deviation of the ratio distribution ([Current]/[Baseline])
  1 us    : 1 Microsecond (0.000001 sec)

How to Read the Summary Table

BDN shows some explanation for the summary table directly in the console. Here are some more detailed explanations:

Mean: Arithmetic mean of all measurements. The average execution time in microseconds of your executions. BDN automatically detects the ideal number of executions of your scenarios that provide reliable results.

StdDev (Standard Deviation): Standard deviation of all measurements. It says how much the execution time differs from the mean (average). If this time is high, your execution times vary a lot.

Ratio: Mean of the ratio distribution ([Current]/[Baseline]). Since we do not compare single execution times of different scenarios, but instead, we compare multiple executions of multiple scenarios, this is an average of a distribution of ratios.

RatioSD (not shown always): Here is a good explanation by the official docs: “The ratio of two benchmarks is not a single number; it’s a distribution. In most simple cases, the range of the ratio distribution is narrow, and BenchmarkDotNet displays a column Ratio with the mean value. However, it also adds the RatioSD column (the standard deviation of the ratio distribution) in complex situations.” (Source: BenchmarkDotNet/docs/articles/samples/IntroRatioSD.md at master · dotnet/BenchmarkDotNet (github.com)

1 us (1 Microsecond): One microsecond is very short – just one-millionth of a second.

BDN will produce the needed binaries for the different runtimes and create other files for further analysis. The results table gets exported as markdown, HTML and CSV files by default.

You may also use Charts for BenchmarkDotNet (chartbenchmark.net) to see the different ratios visually. Copy the summary table from your console into that free service, and you will get a nice histogram.

Useful Attributes

BenchmarkDotNet can be highly configured with attributes. Some useful ones are these:

[MemoryDiagnoser]

The memory diagnoser will allow you to see what amount of bytes will be allocated by the Benchmark in the heap. This attribute can be applied to your Benchmark class.

[Benchmark(Baseline = true)]

Defining one of your Benchmark methods as a baseline is especially useful. This means that you will see the ratio in terms of performance in how the other methods are compared. You may use this Baseline parameter for one of your method Benchmarks. All of them should have the Benchmark attribute on them.

[Params(1,4...)

Define a field or property in your Benchmark class that you use in all your benchmark methods. This [Params] attribute will initialise the field with all the different parameters. Doing so lets the Benchmark run multiple times with different values.

[SimpleJob(RuntimeMoniker.Net70)]
[SimpleJob(RuntimeMoniker.Net80)]

It is beneficial to compare the performance of different runtime versions. Please note there is always a Job, even if you do not define one. If you don’t choose a runtime moniker, BenchmarkDotnet selects the version of the host process.

[ReturnValueValidator(failOnError: true)]

This one helps you to ensure you always return the same values from your benchmark methods. However, in my benchmark tests, I encountered some limitations. I recommend writing unit tests to ensure the same results and testing those methods within BenchmarkDotNet afterwards.

Validating Results

Your methods must produce the same results to be comparable. One way to achieve this is to use the following attribute on your benchmark class.

[ReturnValueValidator(failOnError: true)]

In my example Generating The Fibonacci Sequence in C#, the attribute caused an error. If you have the same limitations, I recommend you two write separate unit tests in your solution to ensure you do not compare apples with oranges.

Examples

My BDN blog articles and GitHub examples:

Conclusion

BenchmarkDotNet is extremely powerful when it comes to getting started quickly with benchmarks. Once you are familiar with the basics, you will set up your benchmarks quickly. Please play around with it and see it as another tool in your toolbox.

The library is quite popular and comes with its own statistics engine. Please see on GitHub and on their official homepage/docs to get a first impression on your own!

Links