[BUG] Benchmark registration order affects reported outcomes

[ccomendant]

Describe the bug
Hello,
I have the following bechmark:

#include <benchmark/benchmark.h>
#include <gtest/gtest.h>

/**
 * defines / pre-allocates a std::vector<std::wstring> examples (within g_dataset)
 */
#include "Helpers.h"

#include <chrono>
#include <cstdint>
#include <ranges>
#include <vector>
#include <execution>
#include <codecvt>

std::string DoWork(const std::wstring& input)
{
    std::string result(input.size(), '\0');
    std::transform(std::execution::par_unseq, std::begin(input), std::end(input), std::begin(result),
                   [](wchar_t inChar) { return (char)(inChar & 0x0ff); });
    return result;
}

std::wstring DoWork1(const std::string& input)
{
    std::wstring result(input.size(), L'\0');
    std::transform(std::execution::par_unseq, std::begin(input), std::end(input), std::begin(result),
                   [](wchar_t inChar) { return (wchar_t)(inChar & 0x0ff); });
    return result;
}

template <typename InStringT,
    typename Fn,
    typename OutStringT = std::invoke_result_t<Fn,InStringT>>
void RunOnExamples(const std::vector<InStringT>& examples, Fn&& fn,
                  std::vector<OutStringT>& results)
{
    for (size_t iterations = 0; iterations < examples.size(); iterations++) {
        results[iterations] = fn(examples[iterations]);
    }
}

static void BM_DoWork(benchmark::State& state)
{
    state.PauseTiming();

    auto expectedResults = std::vector<std::string>(g_dataset.examples.size());

    state.ResumeTiming();

    while (state.KeepRunningBatch(g_dataset.examples.size())) {
        RunOnExamples(g_dataset.examples, DoWork, expectedResults);
    }

    state.SetItemsProcessed(state.iterations() * g_dataset.examples.size());
}

static void BM_DoWork1(benchmark::State& state)
{
    state.PauseTiming();

    auto expectedResults = std::vector<std::wstring>(g_dataset.examples.size());

    std::vector<std::string> examples(g_dataset.examples.size());
    static std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> myconv;
    for (size_t i = 0; i < g_dataset.examples.size(); i++) {
        examples[i] = myconv.to_bytes(g_dataset.examples[i]);
    }

    state.ResumeTiming();

    while (state.KeepRunningBatch(g_dataset.examples.size())) {
        RunOnExamples(examples, DoWork1, expectedResults);
    }

    state.SetItemsProcessed(state.iterations() * g_dataset.examples.size());
}

#pragma endregion

#pragma region Run benchmarks

BENCHMARK(BM_DoWork);
BENCHMARK(BM_DoWork);
BENCHMARK(BM_DoWork);
BENCHMARK(BM_DoWork1);
BENCHMARK(BM_DoWork1);
BENCHMARK(BM_DoWork1);

#pragma endregion

The output looks like this - the items_per_second decreases in rather big jumps:

Run on (20 X 2918 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x10)
  L1 Instruction 32 KiB (x10)
  L2 Unified 1280 KiB (x10)
  L3 Unified 24576 KiB (x1)
------------------------------------------------------------------------------
Benchmark                    Time             CPU   Iterations UserCounters...
------------------------------------------------------------------------------
BM_DoWork                   86347685 ns         6438 ns      1000000 items_per_second=155.34G/s
BM_DoWork                   86352813 ns        10688 ns      1000000 items_per_second=93.5673G/s
BM_DoWork                   86358084 ns        15141 ns      1000000 items_per_second=66.0475G/s
BM_DoWork1                  86363182 ns        19484 ns      1000000 items_per_second=51.3232G/s
BM_DoWork1                  86369707 ns        25062 ns      1000000 items_per_second=39.9002G/s
BM_DoWork1                  86378334 ns        32438 ns      1000000 items_per_second=30.8285G/s

If I reorder the benchmarks, the same pattern is observed.

There has been a similar question on StackOverlow a couple of years ago, but it remained unanswered.

System
Which OS, compiler, and compiler version are you using:

• OS: Microsoft Windows 11
• Compiler and version: MSVC 17.14.15

Expected behavior
Registering benchmarks in different order shouldn't affect the measured performance - a deviation of "a couple" percent is expected, but nothing like the steady decreasing in huge jumps.

[ccomendant]

In fact, the same behavior is observed when the benchmarks are registered to do multiple-runs:

BENCHMARK(BM_DoWork)->Repetitions(5);
BENCHMARK(BM_DoWork)->Repetitions(5);
BENCHMARK(BM_DoWork)->Repetitions(5);
BENCHMARK(BM_DoWork1)->Repetitions(5);
BENCHMARK(BM_DoWork1)->Repetitions(5);
BENCHMARK(BM_DoWork1)->Repetitions(5);

Output - items_per_second decreases in big jumps on the first batch, then only slightly in the following ones:

--------------------------------------------------------------------------------------
Benchmark                            Time             CPU   Iterations UserCounters...
--------------------------------------------------------------------------------------
BM_DoWork/repeats:5           86683819 ns         9234 ns      1000000 items_per_second=108.291G/s
BM_DoWork/repeats:5           86691389 ns        15672 ns      1000000 items_per_second=63.8086G/s
BM_DoWork/repeats:5           86699279 ns        22219 ns      1000000 items_per_second=45.007G/s
BM_DoWork/repeats:5           86706890 ns        28641 ns      1000000 items_per_second=34.9154G/s
BM_DoWork/repeats:5           86714494 ns        34953 ns      1000000 items_per_second=28.6097G/s
BM_DoWork/repeats:5_mean      86699174 ns        22144 ns            5 items_per_second=56.1264G/s
BM_DoWork/repeats:5_median    86699279 ns        22219 ns            5 items_per_second=45.007G/s
BM_DoWork/repeats:5_stddev       12151 ns        10184 ns            5 items_per_second=32.0581G/s
BM_DoWork/repeats:5_cv            0.01 %         45.99 %             5 items_per_second=57.12%
BM_DoWork/repeats:5           86723139 ns        41938 ns      1000000 items_per_second=23.845G/s
BM_DoWork/repeats:5           86730721 ns        48109 ns      1000000 items_per_second=20.786G/s
BM_DoWork/repeats:5           86738141 ns        54469 ns      1000000 items_per_second=18.3592G/s
BM_DoWork/repeats:5           86745756 ns        60859 ns      1000000 items_per_second=16.4313G/s
BM_DoWork/repeats:5           86752562 ns        66734 ns      1000000 items_per_second=14.9848G/s
BM_DoWork/repeats:5_mean      86738064 ns        54422 ns            5 items_per_second=18.8812G/s
BM_DoWork/repeats:5_median    86738141 ns        54469 ns            5 items_per_second=18.3592G/s
BM_DoWork/repeats:5_stddev       11684 ns         9858 ns            5 items_per_second=3.52576G/s
BM_DoWork/repeats:5_cv            0.01 %         18.11 %             5 items_per_second=18.67%
BM_DoWork/repeats:5           86757954 ns        71172 ns      1000000 items_per_second=14.0505G/s
BM_DoWork/repeats:5           86762962 ns        75312 ns      1000000 items_per_second=13.278G/s
BM_DoWork/repeats:5           86768002 ns        79594 ns      1000000 items_per_second=12.5638G/s
BM_DoWork/repeats:5           86773283 ns        84188 ns      1000000 items_per_second=11.8782G/s
BM_DoWork/repeats:5           86778552 ns        88719 ns      1000000 items_per_second=11.2716G/s
BM_DoWork/repeats:5_mean      86768151 ns        79797 ns            5 items_per_second=12.6084G/s
BM_DoWork/repeats:5_median    86768002 ns        79594 ns            5 items_per_second=12.5638G/s
BM_DoWork/repeats:5_stddev        8146 ns         6954 ns            5 items_per_second=1.10116G/s
BM_DoWork/repeats:5_cv            0.01 %          8.71 %             5 items_per_second=8.73%
BM_DoWork1/repeats:5          86784374 ns        93797 ns      1000000 items_per_second=10.6613G/s
BM_DoWork1/repeats:5          86792194 ns       100609 ns      1000000 items_per_second=9.93943G/s
BM_DoWork1/repeats:5          86800669 ns       107797 ns      1000000 items_per_second=9.27671G/s
BM_DoWork1/repeats:5          86809730 ns       115500 ns      1000000 items_per_second=8.65801G/s
BM_DoWork1/repeats:5          86818350 ns       122922 ns      1000000 items_per_second=8.13525G/s
BM_DoWork1/repeats:5_mean     86801063 ns       108125 ns            5 items_per_second=9.33415G/s
BM_DoWork1/repeats:5_median   86800669 ns       107797 ns            5 items_per_second=9.27671G/s
BM_DoWork1/repeats:5_stddev      13521 ns        11567 ns            5 items_per_second=1.0032G/s

[hiSandog]

The positional pattern is the key signal: whichever benchmark runs first comes out fastest and items_per_second decays monotonically down the sequence, and reordering just shifts the decay with position rather than tying it to BM_DoWork vs BM_DoWork1. That argues against a correctness bug in how the library registers or orders benchmarks.

Notably, items_per_second here tracks the CPU column (which climbs ~5×, 6438→32438 ns) while the wall-clock Time column stays nearly flat (~86.3 ms) — so the decay is in the CPU-time measurement, not in the actual work per iteration. That points at the std::execution::par_unseq path's CPU-time accounting (or frequency/affinity effects on how CPU time is charged on Windows/MSVC) rather than registration order.

To confirm, try pinning CPU frequency (high-performance power plan / disable boost states, or Linux cpupower performance), running each benchmark in a separate process, adding a throwaway warmup run at the head of the list, and UseRealTime(). If the CPU column flattens and the decay disappears, registration order is exonerated.