Recently, I have seen many people posting the “performance” stats of the small code snippet. I started reading about how that is done and that’s how I got to know about a NuGet Package “BenchmarkDotNet“.

In this article, we are going to have a look at what it is and how it can be helpful for benchmarking.

What is BenchmarkDotNet ?

Now a days, for achieving a given outcome, there are many libraries / APIs / packages available. That’s why, sometimes when we are writing any piece of code, we may face situations to compare the available options and perform some kind of benchmarking to ensure how do the libraries behave for different kinds of inputs / on different environments.

Benchmarking is not an easy task. You have to write some code to trigger the things that you actually want to perform benchmarking for. Then you may want to run that code for various inputs.

BenchmarkDotNet is a powerful library which can help to do this job. You do not need to write any additional code to setup the project which calls the code which needs to be tested. It is as easy as writing unit tests. The library is open source and it is available as a NuGet package which can be consumed by .NET application.

Supported Platforms / Features

Before going forward, let’s have a look at where this package is supported.

• This NuGet package can be used with C#, F# and VB.NET.
• This NuGet package is works only with console applications (as of today).
• This package can be used with .NET Framework as well as .NET Core (now .NET) projects.
• It can be used on Windows, Linux or MacOS.

Demo App – Simple

In this section, we are going to use BenchmarkDotNet nuget package and will try to benchmark simple code which computes hash. Note that this example is borrowed from BenchmarkDotNet documentation.

Console Application

Let’s open Visual Studio 2022 and create a .NET 6 based console application. Then add a new class MyBenchmarks. Note that the class is marked as public.

Now add reference to the BenchmarkDotNet nuget package. You can use the command given below to add the package to the console app. This command should be executed from the project directory path.

dotnet add package BenchmarkDotNet

If you are using Visual Studio, you can right click on the project and then select Manage NuGet Packages. It will open the dialog as shown below. Search for BenchmarkDotNet and install the package.

Methods for Benchmarking

MyBenchmarks has two fields – sha256 which is default instance for computing hash using SHA256 and the other md5 which is to compute hash using MD5. Then there are two public methods which uses these instances and compute the hash.

We want to measure the performance of these two methods. Hence let’s add Benchmark attribute attribute on those two methods.

Inputs for Benchmarking

Next question is – how are we going to take inputs for computing hash ? For inputs, the class, MyBenchmarks, has a constructor. The constructor uses Random class to generate a random byte array.

The complete class should be as shown below.

BenchmarkRunner

If you have created a .NET 6 console application and if it has top level statements enabled, then you need to add the below statement in program.cs. If you do not have top level statements enabled, then you can add the below statement in the Main method.

As shown in the code snippet, there are 3 different ways to call to trigger the benchmark run.

• We can invoke benchmarks on specific type
• OR we can configure to invoke benchmarks on specific assembly

The only disadvantage of using below mentioned API calls is – we need to change the source code in case we want to benchmark different type or assembly. The advantage, however, is that this is really easy for quick start.

Running the demo

For benchmarking, it is must to use release configuration while running the application that performs benchmarking. Ideally, you should have optimize option set to true either in csproj or in csc command line while building the project.

Also, we should not have any debugger attached to the process. This means , whenever we want to run the benchmarking app, there are two options

• Run the application from Visual Studio by using Debug -> Start Without Debugging (keyboard shortcut CTRL + F5 ). The configuration should be set to release.
• OR the other option is to run the application using dotnet CLI. While using dotnet run command we should ensure that we add -c release switch to the command.

Running this simple app may take several seconds. This is because BenchmarkDotNet will launch benchmark process several times. This page from documentation provides good insight about how it works.

How to interpret results ?

Once the execution is over, the domo application would show the summary result on the console. Below is the result that I got on my machine.

Let’s try to understand what this result means. There are four columns:

• Method column provides the name of method which was being measured
• Mean column conveys the average time (arithmetic mean) of all the measurements for the given method. The snapshot below shows that Sha256 method execution takes 39.85 nanoseconds on an average.
• Error column (in statistics terms) refers to half of the 99.9% confidence interval.  We can be 99.9% sure that the actual mean execution time is within 0.772 nanoseconds of the sampled mean, for method Sha256.
• StdDev (standard deviation) column provides insights about how much the execution times varied from the mean time. In the below snapshot, execution time of Sha256 method was varied by 0.889 nanoseconds from the corresponding mean time. Lesser the standard deviation, more closer the sample mean execution time is to the ideal mean execution time.

Demo App – Passing Parameters

In the above demo, we created a constructor which was generating random data which was being used as input for computing hash.

If you want to measure some code for some specific inputs, BenchmarkDotNet package provides some ways by which we can specify the inputs.

Let’s say we want to compare three different ways for comparing the strings – which are string.Compare, Equals method and the equality operator. For comparison, we would need two strings.

For that, we will add two public properties of string type to the class. Then we can place ParamsSource attribute on them. ParamsSource attribute takes name of another public property as input. This public property should return IEnumerable<string> which should contain sample inputs that we want to use for benchmarking.

The complete class is shown in the below code snippet.

In this class, we have ValuesForFirst property, which returns sample inputs for property with name First. Similarly, we have ValuesForSecond property which returns sample inputs for property Second.

Then there are three methods which compare the two strings (First and Second) by using three different ways.

For benchmarking, we can use BenchmarkRunner.Run method and specify this type so that our console app benchmarks this type as well. Then we can run the console application in release mode (no debugger should be attached).

It should present us the results in the similar format that we have seen in previous example. For your reference, below is the summary that I received on my machine.

Note that for benchmarking, we will have to take care of turning off all the applications except the benchmark process and the standard OS processes. If you run benchmark and work in the Visual Studio at the same time, it can negatively affect to benchmark results.

Wrapping Up

There are various situations in which BenchmarkDotNet can be used. One of the important usage – is to measure performance of Web APIs or gRPC services in our applications. Theoretically, for that, we just need to add a console app which calls those services.

This article does not fully cover all the features and customization options available. But it should certainly help you to get started with this package. If you want to know more you can refer their documentation here.

I hope you find this content helpful. Have you been using this package already ? How is your experience with it ? I am curious to know. Let me know your thoughts.