Etimo.Benchmarks – A Lightweight .NET Performance Benchmark Tool

Aug 17, 2014

Etimo.Benchmarks is a pragmatic .NET performance benchmark tool that measures execution time and it is released to the public domain. The key points are:

  • Lightweight and simple to use, for individual component benchmarking or for comparing different components
  • Supports specifying nested groups of delegates to be benchmarked hierarchically and in order
  • Focuses on the benchmark processing and simply returns an object with the results that can easily be consumed in order to write the results to the console/file/other
  • Released to the public domain, so you can do what you want with the source code
There is a NuGet package, a sample NuGet package and the source code is available at GitHub.

Are you in a hurry?

Don’t you have time reading this blog post and just want to get started benchmarking immediately? Then I suggest you get the sample package and simply modify the sample delegates as needed. However, it’s a good idea to read this post later.

The purpose of Etimo.Benchmarks is to provide a lightweight package for developers to quickly compare execution time for different pieces of code in a simple and structured manner. The purpose is not to provide a full profiling tool that can attach to running processes and let you analyze it visually. (For such purposes I have personally found ‘ANTS Performance Profiler’ by Red Gate to be valuable.)

Background

When I have needed to measure execution time of some pieces of code, I have simply started a System.Diagnostics.Stopwatch, executed the code and stopped the stopwatch. That worked fine. Well it seemed to work fine, but the measures were not reliable since I did not consider the garbage collection process, the just-in-time compilation (JIT) etc.

Sometime later I needed to compare several different implementations of a specific interface by comparing execution time for some use-cases. Some use-cases had several steps, and I wanted to compare each step as well as the full group of steps as a whole. I thought ‘that should be simple, I’ll just repeat the process of running stopwatches and running the code, save the results and write it to the console’. Funny enough, it wasn’t at all as simple as I initially thought.

I searched NuGet and I asked google. I found plenty of benchmarking tools, but none that I really liked. That is why I decided to write Etimo.Benchmarks.

Core Concepts

Etimo.Benchmarks is based upon the following core concepts:

  • Benchmark Component: If you want to compare two classes, you write two different benchmark component classes by inheriting IBenchmarkComponent. (I recommend to write an abstract base class called BenchmarkComponentBase that inherits IBenchmarkComponent, and then create two classes that both inherit BenchmarkComponentBase.
  • Operation Group: A group of delegates to benchmark individually.
  • Operation: An object with a name and a delegate to benchmark. There are two classes for operations, OperationWithAction and OperationWithFunc. Use OperationWithFunc if you want the return value to be included in the benchmark results for some reason.
  • Benchmark Processor: It takes benchmark components as argument, executes the benchmark operations and returns the benchmark results.

Etimo.Benchmarks Usage Sample

To explain how to use Etimo.Benchmarks, I will show a full sample that compares two collections, KeyedCollection (included in .NET) and MultiplyIndexedKeyedCollection (published by Etimo).

The below screen shot shows the classes needed, as well as how they relate to the core concepts described earlier.

Etimo.Benchmarks Sample Project

Sample Benchmark Operation Groups

We start by looking at sample code for an operation group, because the root operation in this sample is an operation group that defines the entire benchmark process.

We define a name for the group and four operations.

public class CollectionBenchmarkRootOperationGroup : IOperationGroup<OperationInitialization, OperationGetGdp2010ByCode, OperationGetGdp2010ByName, OperationGroupByIncrease>
{
    public string Name { get { return "Benchmark Root"; } }
    public OperationInitialization Operation1 { get; set; }
    public OperationGetGdp2010ByCode Operation2 { get; set; }
    public OperationGetGdp2010ByName Operation3 { get; set; }
    public OperationGroupByIncrease Operation4 { get; set; }
}

The fourth operation in the above class is actually an entire operation group that contains three operations:

public class OperationGroupByIncrease : IOperationGroup<OperationHasFiveDoubledCount, OperationHasTenDoubledCount, OperationHasTwentyDoubledCount>
{
    public string Name { get { return "ByIncrease"; } }
    public OperationHasFiveDoubledCount Operation1 { get; set; }
    public OperationHasTenDoubledCount Operation2 { get; set; }
    public OperationHasTwentyDoubledCount Operation3 { get; set; }
}

Sample Benchmark Operations

Operations must implement either IOperationWithAction or IOperationWithFunc.

OperationInitialization implements IOperationWithAction:

public class OperationInitialization : IOperationWithAction
{
    public string Name { get { return "Initialization"; } }
    public Action Delegate { get; set; }
}

OperationHasFiveDoubledCount implements IOperationWithFunc:

public class OperationHasFiveDoubledCount : IOperationWithFunc<int>
{
    public string Name { get { return "HasFiveDoubledCount"; } }
    public Func<int> Delegate { get; set; }
}

The other operations are similar, so I will not show them all here (but you can see them in the sample package).

Sample Benchmark Components

Now we have defined the operation groups and the operations, so it’s time to define the benchmark components. BenchmarkComponentBase defines the base class for each of the components that will be benchmarked. Note that the Name-property is abstract.

public abstract class BenchmarkComponentBase : IBenchmarkComponent<CollectionBenchmarkRootOperationGroup>
{
    public abstract string Name { get; }
    public CollectionBenchmarkRootOperationGroup RootOperation { get; set; }
}

The component to benchmark the standard KeyedCollection:

public class BenchmarkComponentKeyedCollection : BenchmarkComponentBase
{
    public override string Name
    {
        get { return "KeyedCollection"; }
    }

    public BenchmarkComponentKeyedCollection(IEnumerable<CountryOrRegionGdpData> listOfCountryOrRegionGdpData)
    {
        StandardKeyedCollectionOfCountryOrRegionGdpData standardKeyedCollectionOfCountryOrRegionGdpData = new StandardKeyedCollectionOfCountryOrRegionGdpData();

        RootOperation = new CollectionBenchmarkRootOperationGroup()
        {
            Operation1 = new OperationInitialization()
            {
                Delegate = () =>
                {
                    foreach (var item in listOfCountryOrRegionGdpData)
                        standardKeyedCollectionOfCountryOrRegionGdpData.Add(item);
                },
            },
            Operation2 = new OperationGetGdp2010ByCode()
            {
                Delegate = () => standardKeyedCollectionOfCountryOrRegionGdpData["SWE500"].GdpYear2010.Value
            },
            Operation3 = new OperationGetGdp2010ByName()
            {
                Delegate = () => standardKeyedCollectionOfCountryOrRegionGdpData.Single(q => q.CountryName == "Sweden500").GdpYear2010.Value,
            },
            Operation4 = new OperationGroupByIncrease()
            {
                Operation1 = new OperationHasFiveDoubledCount()
                {
                    Delegate = () => standardKeyedCollectionOfCountryOrRegionGdpData.Count(countryOrRegionGdpData => countryOrRegionGdpData.GdpYear2010 >= countryOrRegionGdpData.GdpYear1960 * 5),
                },
                Operation2 = new OperationHasTenDoubledCount()
                {
                    Delegate = () => standardKeyedCollectionOfCountryOrRegionGdpData.Count(countryOrRegionGdpData => countryOrRegionGdpData.GdpYear2010 >= countryOrRegionGdpData.GdpYear1960 * 10),
                },
                Operation3 = new OperationHasTwentyDoubledCount()
                {
                    Delegate = () => standardKeyedCollectionOfCountryOrRegionGdpData.Count(countryOrRegionGdpData => countryOrRegionGdpData.GdpYear2010 >= countryOrRegionGdpData.GdpYear1960 * 20),
                },
            },
        };
    }
}

The component to benchmark MultiplyIndexedKeyedCollection:

public class BenchmarkComponentMultiplyIndexedKeyedCollection : BenchmarkComponentBase
{
    public override string Name
    {
        get { return "MultiplyIndexedKeyedCollection"; }
    }

    public BenchmarkComponentMultiplyIndexedKeyedCollection(IEnumerable<CountryOrRegionGdpData> listOfCountryOrRegionGdpData)
    {
        MultiplyIndexedKeyedCollectionOfCountryOrRegionGdpData multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData = new MultiplyIndexedKeyedCollectionOfCountryOrRegionGdpData();

        RootOperation = new CollectionBenchmarkRootOperationGroup()
        {
            Operation1 = new OperationInitialization()
            {
                Delegate = () =>
                {
                    foreach (var item in listOfCountryOrRegionGdpData)
                        multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.Add(item);
                },
            },
            Operation2 = new OperationGetGdp2010ByCode()
            {
                Delegate = () => multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.ByCountryCode["SWE500"].GdpYear2010.Value
            },
            Operation3 = new OperationGetGdp2010ByName()
            {
                Delegate = () => multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.ByCountryName["Sweden500"].GdpYear2010.Value,
            },
            Operation4 = new OperationGroupByIncrease()
            {
                Operation1 = new OperationHasFiveDoubledCount()
                {
                    Delegate = () => multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.ByHasFiveDoubled[true].Count(),
                },
                Operation2 = new OperationHasTenDoubledCount()
                {
                    Delegate = () => multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.ByHasTenDoubled[true].Count(),
                },
                Operation3 = new OperationHasTwentyDoubledCount()
                {
                    Delegate = () => multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.ByHasTwentyDoubled[true].Count(),
                },
            },
        };
    }
}

Benchmark Processor Configuration

BenchmarkProcessorConfiguration allows specifying these configurations:

Executing the Benchmark

To execute the benchmark, use the BenchmarkProcessor.Execute method:

public static void RunSample()
{
    IList<CountryOrRegionGdpData> listData = new DataImporter().Import();

    Func<BenchmarkComponentBase>[] benchmarkComponents =
    {
        () => new BenchmarkComponentKeyedCollection(listData),
        () => new BenchmarkComponentMultiplyIndexedKeyedCollection(listData),
    };

    BenchmarkProcessor benchmarkProcessor = new BenchmarkProcessor();

    BenchmarkProcessorConfiguration benchmarkProcessorConfiguration = new BenchmarkProcessorConfiguration();

    IEnumerable<IBenchmarkComponentResult> benchmarkResults = benchmarkProcessor.Execute(benchmarkProcessorConfiguration, benchmarkComponents);

    foreach (IBenchmarkComponentResult benchmarkComponentResult in benchmarkResults)
        Console.WriteLine("Benchmark Component: {0}{1}{2}", benchmarkComponentResult.Name, Environment.NewLine, FormatBenchmarkResults(benchmarkComponentResult.RootOperationResult, 0));

    Console.ReadLine();
}

In the above code, the call to FormatBenchmarkResults() is where the main part of reading and formatting the results takes place. This will be explained in the next section.

Note that the DataImporter is not part of the benchmarking, it’s just a custom class that provides some data that is used by the collections that are benchmarked.

Reading the Benchmark Results

The result types IOperationGroupResult, IOperationWithActionResult and IOperationWithFuncResult are defined like this:

public interface IOperationResultBase
{
    string Name { get; }
    IDurations Durations { get; }
}

public interface IOperationGroupResult : IOperationResultBase
{
    IEnumerable<IOperationResultBase> ChildOperationResults { get; }
}

public interface IOperationWithActionResult : IOperationResultBase
{
}

public interface IOperationWithFuncResult : IOperationResultBase
{
    object FuncReturnValue { get; }
}

Note in particular that if your benchmark is a deep hierarchy of nested groups, you can easily traverse the hierarchy by recursively access the property IOperationGroupResult.ChildOperationResults.

Since there are so many ways to use the result data, I decided that Etimo.Benchmarks should not include any formatting of result data at all. The sample code formats the result data like this:

private static string FormatBenchmarkResults(IOperationResultBase benchmarkResult, int nestingLevel)
{
    string formattedResults = nestingLevel == 0 ? "" : new string('-', nestingLevel * 3) + '→' + ' ';
    string label = benchmarkResult is IOperationGroupResult ? benchmarkResult.Name + " (Group Total)" : benchmarkResult.Name;

    formattedResults += string.Format("{0}: {1}", label.PadRight(22), benchmarkResult.Durations.Min.TotalMilliseconds.ToString("0.000 ms.").PadLeft(12));

    if (benchmarkResult is IOperationWithFuncResult)
        formattedResults += string.Format("{0}Return Value: {1}", new string(' ', 3), ((IOperationWithFuncResult)benchmarkResult).FuncReturnValue);

    if (benchmarkResult is IOperationGroupResult)
    {
        IOperationGroupResult benchmarkResultTyped = (IOperationGroupResult)benchmarkResult;

        foreach (IOperationResultBase childOperationResult in benchmarkResultTyped.ChildOperationResults)
            formattedResults += Environment.NewLine + FormatBenchmarkResults(childOperationResult, nestingLevel + 1);
    }

    return formattedResults;
}

A screenshot executing the sample project:

Etimo.Benchmarks Sample Results As you can see, our MultiplyIndexedKeyedCollection is sometimes thousands of times faster than the standard .NET-collection! It’s a read-optimized collections, so write operations are slower due to indexing. Read more in our blog-post.

Additional information

In this section I will highlight a few things that might need extra attention.

Benchmark input, state management and initialization

The sample code in this blog post shows how to run a benchmark that needs benchmark input, state management and perform initialization.

  • Benchmark input: The constructors of the two benchmark components both have parameters of type IEnumerable<CountryOrRegionGdpData> for the input. Note that the constructor itself is not benchmarked, so avoid running any code directly in the constructor. Put such code in the benchmark operations instead.
  • State management: The benchmark operation delegates have references to variables outside of of the delegates. The delegates in BenchmarkComponentKeyedCollection have references to listOfCountryOrRegionGdpData and standardKeyedCollectionOfCountryOrRegionGdpData, while the delegates in BenchmarkComponentMultiplyIndexedKeyedCollection have references to listOfCountryOrRegionGdpData and multiplyIndexedKeyedCollectionOfCountryOrRegionGdpData.
  • Initialization: The benchmark operation OperationInitialization performs the needed initialization. Note that technically OperationInitialization is just an ordinary benchmark operation, so it is benchmarked as well.

Names

The benchmark components override the Name property to specify a name for the specific benchmark component like this:

public override string Name
{
    get { return "MultiplyIndexedKeyedCollection"; }
}

However, there is no reason to specify benchmark-component-specific names for the operation groups or the operations.

Hence, if you have 10 benchmark components and each component has a total of 5 operation groups and operations, you will in total specify 15 names (10 names for the benchmark components and 5 names for the operation groups and operations).

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *