Benchmark Runner output

[maximusunc]

The Benchmark Runner has two main outputs, (1) a text output of the overall results metrics and (2) screenshot images of those metrics plotted.

Sample text output:

Benchmark: ameliorates
Results Directory: /var/folders/lr/65kxj04n3vz3ttm_vw0_tq8r0000gn/T/tmpzkjrkmhe/ameliorates/aragorn/2023-11-17_12-42-58

                        k=1     k=5     k=10    k=20
Precision @ k           0.0909  0.0818  0.0636  0.0545
Recall @ k              0.0476  0.2143  0.3333  0.5714
mAP @ k                 0.0909  0.1697  0.1877  0.2092
Top-k Accuracy          0.0476  0.2143  0.3333  0.5714

Mean Reciprocal Rank    0.23669455544455545

The screenshots are a dictionary where the keys are the plot types (i.e. precision @ k) and the values are raw image bytes.

[maximusunc]

I've also been playing around with how these would look in the Information Radiator and so far I've gotten this up and running.

[sierra-moxon]

@maximusunc - is there any high-level notion of "passing a benchmark" or "failing a benchmark" in the current output?

[maximusunc]

Nope. If it runs successfully, then it passes.

[sierra-moxon]

runs successfully == gives output such as that above?

[maximusunc]

Yes

[sierra-moxon]

What is the hook in the output that allows a user to go back to the "run" of the test. E.g. see what the benchmark was, which component it ran on, etc.?

[maximusunc]

Good point. I will add that info in. It will most likely be included in the results metrics.

[sierra-moxon]

e.g. #10 (comment)