BenchmarkDotNet - Open Source Sponsorship

The software development space has creatively coined some memorable statements over the years, from “speed is a feature”, “memory is cheap”, and “always blame the new guy”. All these statements have one thing in common: as developers, we should do our best to baseline our assumptions and verify the truth. In the spirit of building the best software we possibly can by focusing on the fine details, we are happy to announce that this quarter's Duende Open Source Sponsorship goes to BenchmarkDotNet.

In our fourth sponsorship, the team at Duende has chosen BenchmarkDotNet as the next open-source recipient of our ongoing commitment to supporting projects that empower individuals, teams, communities, and organizations.

Now let’s see what BenchmarkDotNet is all about.

What is BenchmarkDotNet?

BenchmarkDotNet is the premier benchmarking library for .NET, helping you transform methods into benchmarks, track their performance, and share reproducible measurement results. While benchmarking may seem challenging, BenchmarkDotNet provides an API that wraps the most difficult aspects of the process and exposes developers to a familiar unit-testing paradigm. 27400+ GitHub projects have adopted BenchmarkDotNet, including .NET Runtime, .NET Compiler, .NET Performance, and many others. The project's grounding philosophy rests on the pillars of simplicity, automation, reliability, and friendliness.

Getting Started

Note: This getting started is adapted from the BenchmarkDotNet documentation.

To get started with BenchmarkDotNet, you’ll need to create a new console application within your solution. After making the new project, you’ll want to add the NuGet package.

dotnet add package BenchmarkDotNet

In the console application, we’ll be measuring the differences between .NET hashing algorithms. Paste the following code into the project.

using System;
using System.Security.Cryptography;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

namespace MyBenchmarks
{
    public class Md5VsSha256
    {
        private const int N = 10000;
        private readonly byte[] data;

        private readonly SHA256 sha256 = SHA256.Create();
        private readonly MD5 md5 = MD5.Create();

        public Md5VsSha256()
        {
            data = new byte[N];
            new Random(42).NextBytes(data);
        }

        [Benchmark]
        public byte[] Sha256() => sha256.ComputeHash(data);

        [Benchmark]
        public byte[] Md5() => md5.ComputeHash(data);
    }

    public class Program
    {
        public static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<Md5VsSha256>();
        }
    }
}

The Program class is the console application’s entry point and invokes the BenchmarkRunner.Run<Md5VsSha256>() method to generate our benchmark results. Running the console application displays the results in the console.

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19045.2251)
Intel Core i7-4770HQ CPU 2.20GHz (Haswell), 1 CPU, 8 logical and 4 physical cores
.NET SDK=7.0.100
  [Host]     : .NET 7.0.0 (7.0.22.51805), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.0 (7.0.22.51805), X64 RyuJIT AVX2


| Method |     Mean |    Error |   StdDev |
|------- |---------:|---------:|---------:|
| Sha256 | 51.57 us | 0.311 us | 0.291 us |
|    Md5 | 21.91 us | 0.138 us | 0.129 us |

While the current output shows only execution deviations between the two hashing algorithms, you can introduce additional facilities to examine other attributes of the code. Some include metrics around garbage collection, code size, threading, and many more. Additionally, you can export a report to multiple output formats, including CSVs, JSON, and HTML.

Here’s an example of the R plot output comparing several important metrics produced by the benchmark.

If you want to know more about BenchmarkDotNet features, check out the Overview page.

BenchmarkDotNet at Duende

At Duende, we utilize BenchmarkDotNet to profile the performance of our Backend-For-Frontend (BFF) product in multiple scenarios. BenchmarkDotNet has been invaluable in helping us verify the different product modalities that customers can configure. The library also helps us track any regressions as we maintain and extend the product.

Here’s an example of our use case, verifying the performance of a single frontend and request throughput while using BFF.

// Copyright (c) Duende Software. All rights reserved.
// See LICENSE in the project root for license information.

using BenchmarkDotNet.Attributes;

namespace Bff.Benchmarks;

public class SingleFrontendBenchmarks : BenchmarkBase
{
    private SingleFrontendFixture _fixture = null!;

    private HttpClient _authenticatedBffClient = null!;
    private HttpClient _anonBffClient = null!;
    private HttpClient _bffServerSideSessionsClient = null!;


    [GlobalSetup]
    public override async Task InitializeAsync()
    {
        _fixture = new SingleFrontendFixture();

        _authenticatedBffClient = _fixture.Internet.BuildHttpClient(_fixture.Bff.Url());
        await _authenticatedBffClient.GetAsync("/bff/login")
            .EnsureStatusCode();

        _anonBffClient = _fixture.Internet.BuildHttpClient(_fixture.Bff.Url());

        _bffServerSideSessionsClient = _fixture.Internet.BuildHttpClient(_fixture.Bff.Url());

        await _bffServerSideSessionsClient.GetAsync("/bff/login")
            .EnsureStatusCode();

    }

    [Benchmark]
    public async Task SingleFrontend_UserToken() => await _authenticatedBffClient
            .GetWithCSRF("/user_token")
            .EnsureStatusCode();

    [Benchmark]
    public async Task SingleFrontend_ClientCredentialsToken() => await _authenticatedBffClient
            .GetWithCSRF("/client_token")
            .EnsureStatusCode();

    [Benchmark]
    public async Task SingleFrontend_AnonLocalEndpoint() => await _anonBffClient
        .GetAsync("/anon")
        .EnsureStatusCode();

    [Benchmark]
    public async Task SingleFrontend_LocalEndpoint() => await _authenticatedBffClient
        .GetAsync("/anon")
        .EnsureStatusCode();

    [Benchmark]
    public async Task SingleFrontend_Login()
    {
        var client = _fixture.Internet.BuildHttpClient(_fixture.Bff.Url());
        await client.GetAsync("/bff/login")
            .EnsureStatusCode();
    }

    [GlobalCleanup]
    public override async Task DisposeAsync() => await _fixture.DisposeAsync();
}

It really is fantastic.

Sponsorship Details

To support the BenchmarkDotNet team and their outstanding work, Duende Software is sponsoring BenchmarkDotNet for the next 12 months at $250 per month (totaling $3000 for the year). This sponsorship will help the maintainers cover expenses and invest in the project’s growth and promotion.

If you benefit from BenchmarkDotNet, consider supporting the project as well. You can contribute in various ways, including financial contributions, code, documentation, or bug reports.