Writing benchmarks with pytest benchmark and memray

pytest-benchmark is a tool that allows you to write benchmarks to be run with pytest.

The pytest-benchmark creates a benchmark fixture that can be passed to a test. For example consider this test:

def test_support_enumeration_on_two_by_two_game(benchmark):
    A = np.array(((1, -1), (-1, 1)))
    eqs = support_enumeration(A, -A)
    benchmark(tuple, eqs)

The benchmark fixture is a function with signature:

benchmark(<function>, *args, **kwargs)

Running benchmarks

To run the benchmarks no extra commands are necessary. For example if the benchmarks are located in benchmarks/ then the following command will run them:

python -m pytest benchmarks

Comparing benchmarks

To save the results of a benchmark run you can run:

python -m pytest benchmarks --benchmark-autosave

To compare the results with a saved set of benchmarks:

python -m pytest benchmarks --benchmark-compare

Profiling memory with memray

pytest-memray is a plugin that profiles the memory use when running a specific set of tests.