CI: Performance regression tests update

jenkins/L0_Test.groovy

@@ -1230,11 +1230,21 @@ def runLLMTestlistOnPlatformImpl(pipeline, platform, testList, config=VANILLA_CO
         }
 
         if (perfMode) {
+            basePerfFilename = stageName.contains("PyTorch") ? "base_perf_pytorch.csv" : "base_perf.csv"
+            basePerfPath = "${llmSrc}/tests/integration/defs/perf/${basePerfFilename}"
             stage("Check perf result") {
                 sh """
                     python3 ${llmSrc}/tests/integration/defs/perf/sanity_perf_check.py \
                     ${stageName}/perf_script_test_results.csv \
-                    ${llmSrc}/tests/integration/defs/perf/base_perf.csv
+                    ${basePerfPath}
+                """
+            }
+            stage("Create perf report") {
+                sh """
+                    python3 ${llmSrc}/tests/integration/defs/perf/create_perf_comparison_report.py \
+                    --output_path ${stageName}/report.pdf \
+                    --files ${stageName}/perf_script_test_results.csv \
+                    ${basePerfPath}
                 """
             }
         }
@@ -1572,8 +1582,9 @@ def launchTestJobs(pipeline, testFilter, dockerNode=null)
         "H100_PCIe-TensorRT-[Post-Merge]-2": ["h100-cr", "l0_h100", 2, 2],
         "B200_PCIe-Triton-Python-[Post-Merge]-1": ["b100-ts2", "l0_b200", 1, 1],
         "DGX_H100-4_GPUs-TensorRT-[Post-Merge]-1": ["dgx-h100-x4", "l0_dgx_h100", 1, 1, 4],
-        "A100_80GB_PCIE-TensorRT-Perf-1": ["a100-80gb-pcie", "l0_perf", 1, 1],
+        // "A100_80GB_PCIE-TensorRT-Perf-1": ["a100-80gb-pcie", "l0_perf", 1, 1],
         "H100_PCIe-TensorRT-Perf-1": ["h100-cr", "l0_perf", 1, 1],
+        "H100_PCIe-PyTorch-Perf-1": ["h100-cr", "l0_perf", 1, 1],
         "DGX_H200-8_GPUs-PyTorch-[Post-Merge]-1": ["dgx-h200-x8", "l0_dgx_h200", 1, 1, 8],
         "DGX_H200-4_GPUs-PyTorch-[Post-Merge]-1": ["dgx-h200-x4", "l0_dgx_h200", 1, 2, 4],
         "DGX_H200-4_GPUs-PyTorch-[Post-Merge]-2": ["dgx-h200-x4", "l0_dgx_h200", 2, 2, 4],

tests/integration/defs/perf/README.md

@@ -1,11 +1,66 @@
-Sanity Perf Check Introduction
+# Sanity Perf Check Introduction
 
-# Background
-The sanity perf check mechanism is the way of perf regression detection for L0 testing. We create the base_perf.csv which consists of the several models' perf baseline and use the sanity_perf_check.py to detect the perf regression.
-# Usage
-There're four typical scenarios for sanity perf check feature.
+## Background
+"Sanity perf check" is a mechanism to detect performance regressions in the L0 pipeline.
+The tests defined in `l0_perf.yml` are the ones that are required to pass for every PR before merge.
 
-1. The newly added MR doesn't impact the models' perf, the perf check will pass w/o exception.
-2. The newly added MR introduces the new model into perf model list. The sanity check will trigger the exception and the author of this MR needs to add the perf into base_perf.csv.
-3. The newly added MR improves the existed models' perf and the MR author need to refresh the base_perf.csv data w/ new baseline.
-4. The newly added MR introduces the perf regression and the MR author needs to fix the issue and rerun the pipeline.
+### `base_perf.csv`
+The baseline for performance benchmarking is defined at `base_perf.csv` - this file contains the metrics that we verify regression on between CI runs.
+
+This file contains records in the following format:
+```
+perf_case_name metric_type	perf_metric	threshold absolute_threshold
+```
+
+To allow for some machine dependent variance in performance benchmarking we also define a `threshold` and an `absolute_threshold`. This ensures we do not fail on results that reside within legitimate variance thresholds.
+
+`threshold` is relative.
+
+## CI
+As part of our CI, the `test_perf.py` collects performance metrics for configurations defined in `l0_perf.yml`. This step outputs a `perf_script_test_results.csv` containing the metrics collected for all configurations.
+
+After this step completes, the CI will run `sanity_perf_check.py`. This script will make sure that all differences in metrics from the run on this branch is within a designated threshold of the baseline (`base_perf.csv`).
+
+There're 4 possible results for this:
+1. The current HEAD impact on the performance for our setups is within accepted threshold - the perf check will **pass** w/o exception.
+2. The current HEAD introduces a new setup/metric in `l0_perf.yml` or removes some of them. This will result in new metrics collected by `test_perf.py` which will **fail** `sanity_perf_check.py`. This requires an update for `base_perf.csv`.
+3. The current HEAD improves performance for at least one metric by more than the accepted threshold, which will **fail** `sanity_perf_check.py`. This requires an update for `base_perf.csv`
+4. The current HEAD introduces a regression to one of the metrics that is over the accepted threshold, which will **fail** `sanity_perf_check.py`. This will require to fix the current branch and rerun the pipeline.
+
+### Updating `base_perf.csv`
+If a CI run fails `sanity_perf_check.py`, it will upload a patch file as an artifact. This file can be applied to current branch using `git apply <patch_file>`.
+
+This patch will only update the metrics that had a difference which was over the accepted threshold. The patch will also remove/add metrics according to the removed or added tests.
+
+## Running locally
+Given a `target_perf_csv_path` you can compare it to another perf csv file.
+First make sure you install the dependencies:
+```
+pip install -r tests/integration/defs/perf/requirements.txt
+```
+Then, you can run it with:
+```
+sanity_perf_check.py <target_perf_csv_path> <base_perf_csv_path>
+```
+** In the CI, `<base_perf_csv_path>` is the `base_perf.csv` file path mentioned above.
+
+Running this print the diffs between both performance results. It presents only:
+1. Metrics that have a diff bigger than the accepted threshold.
+2. Metrics missing in `base_perf_csv`.
+3. Metrics missing in `target_perf_csv`.
+
+If any diffs were found it will also generate a patch file to change `base_perf_csv`  with the new metrics, it will be written to the same directory as <target_perf_csv_path> resides in.
+
+
+## Generating diff report
+To view the difference between performance reports, it is possible to generate a pdf report containing Bar graphs comparing the perf metric value per-metric.
+Each metric will contain comparison bars per configuration.
+
+For example: If we run the script with 3 files and test 2 configurations per metric, we will have 2 groups of 3 bars - A group per-configuration, each group containing the 3 performance metrics reported in the 3 files.
+
+To generate this report:
+```
+python tests/integration/defs/perf/create_perf_comparison_report.py --output_path=<output_path_for_report> --files <csv file paths separated by spaces>
+```
+
+This will create a pdf file at <output_path_for_report>.

tests/integration/defs/perf/base_perf.csv

@@ -0,0 +1,5 @@
+network_name,perf_case_name,test_name,threshold,absolute_threshold,metric_type,perf_metric
+"llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-TensorRT-Perf-1/perf/test_perf.py::test_perf_metric_build_time[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_build_time[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",0.1,30,BUILD_TIME,143.5976
+"llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-TensorRT-Perf-1/perf/test_perf.py::test_perf_metric_inference_time[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_inference_time[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",0.1,50,INFERENCE_TIME,106778.60992
+"llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-TensorRT-Perf-1/perf/test_perf.py::test_perf_metric_seq_throughput[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_seq_throughput[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",-0.1,10,SEQ_THROUGHPUT,76.72174
+"llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-TensorRT-Perf-1/perf/test_perf.py::test_perf_metric_token_throughput[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_token_throughput[llama_v3.1_8b_instruct-bench-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",-0.1,10,TOKEN_THROUGHPUT,9820.38162

tests/integration/defs/perf/base_perf_pytorch.csv

@@ -0,0 +1,4 @@
+network_name,perf_case_name,test_name,threshold,absolute_threshold,metric_type,perf_metric
+"llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-PyTorch-Perf-1/perf/test_perf.py::test_perf_metric_inference_time[llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_inference_time[llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",0.1,50,INFERENCE_TIME,99133.65406
+"llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-PyTorch-Perf-1/perf/test_perf.py::test_perf_metric_seq_throughput[llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_seq_throughput[llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",-0.1,10,SEQ_THROUGHPUT,82.63618
+"llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192","H100_PCIe-PyTorch-Perf-1/perf/test_perf.py::test_perf_metric_token_throughput[llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]","test_perf_metric_token_throughput[llama_v3.1_8b_instruct-bench-pytorch-float16-maxbs:512-maxnt:2048-input_output_len:128,128-reqs:8192]",-0.1,10,TOKEN_THROUGHPUT,10577.431520000002

tests/integration/defs/perf/create_perf_comparison_report.py

@@ -0,0 +1,155 @@
+import argparse
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from matplotlib.backends.backend_pdf import PdfPages
+from matplotlib.figure import Figure
+
+TEST_NAME = 'test_name'
+METRIC_VALUE = 'perf_metric'
+METRIC_TYPE = 'metric_type'
+MEAN_COL = 'mean'
+
+
+def shorten_names(merged: pd.DataFrame) -> tuple[dict[str, str], pd.DataFrame]:
+    name_mapping = {
+        k: f'configuration_{i+1}'
+        for i, k in enumerate(set(config for config in merged[TEST_NAME]))
+    }
+    merged[TEST_NAME] = merged[TEST_NAME].apply(lambda name: name_mapping[name])
+    return merged, name_mapping
+
+
+def write_name_mapping_table(name_mapping: dict[str, str],
+                             pdf: PdfPages) -> None:
+    fig, ax = plt.subplots(figsize=(max(len(n)
+                                        for n in name_mapping.keys()) * 0.3,
+                                    len(name_mapping) *
+                                    0.4))  # height depends on number of entries
+    ax.axis('off')
+
+    table_data = [["Original Name", "Short Name"]]
+    for original, short in name_mapping.items():
+        table_data.append([original, short])
+
+    plt.title("Long name to short name mapping")
+    table = ax.table(cellText=table_data, cellLoc='left', loc='center')
+    table.auto_set_font_size(False)
+    table.set_fontsize(10)
+    table.scale(1, 1.5)
+    pdf.savefig(fig)
+    plt.close(fig)
+
+
+def plot_metric(merged: pd.DataFrame, metric: str,
+                suffixes: set[str]) -> Figure:
+    metric_data = merged[merged[METRIC_TYPE] == metric]
+    relevant_metrics = {
+        MEAN_COL: metric_data[MEAN_COL]
+    } | {
+        suffix: metric_data[f"{METRIC_VALUE}_{suffix}"]
+        for suffix in suffixes
+    }
+
+    # Prepare the data: extract only the needed columns
+    plot_data = pd.DataFrame({
+        TEST_NAME: metric_data[TEST_NAME],
+    } | relevant_metrics)
+
+    plot_data = plot_data.set_index(TEST_NAME)
+
+    x = np.arange(len(plot_data))
+    width = 0.8 / len(relevant_metrics.keys())
+
+    fig, ax = plt.subplots(figsize=(10, 6))
+
+    for i, suffix in enumerate(relevant_metrics.keys()):
+        values = plot_data[suffix]
+        bar_positions = x + i * width
+        bars = ax.bar(bar_positions, values, width, label=suffix)
+
+        for bar in bars:
+            height = bar.get_height()
+            ax.text(bar.get_x() + bar.get_width() / 2,
+                    height + 0.01,
+                    f'{height:.2f}',
+                    ha='center',
+                    va='bottom',
+                    fontsize=8)
+
+    ax.set_title(f"Comparison for {metric}")
+    ax.set_ylabel("Metric Value")
+    ax.set_xlabel("Model Name")
+    ax.set_xticks(x + width * (len(suffixes) - 1) / 2)
+    ax.set_xticklabels(plot_data.index, rotation=45, ha='right')
+    ax.legend(title='Suffix')
+
+    return fig
+
+
+def generate_plots(output_path: Path, name_mapping: dict[str, str],
+                   merged: pd.DataFrame, suffixes: set[str]) -> None:
+    metric_types = merged[METRIC_TYPE].unique()
+    with PdfPages(output_path.as_posix()) as pdf:
+        write_name_mapping_table(name_mapping, pdf)
+        for metric in metric_types:
+            fig = plot_metric(merged, metric, suffixes)
+            plt.tight_layout()
+            pdf.savefig(fig)
+            plt.close(fig)
+
+
+def parse_perf_data(
+        perf_files: list[str]) -> tuple[dict[str, str], pd.DataFrame, set[str]]:
+    perfs = {
+        Path(file_path).name: pd.read_csv(file_path)
+        for file_path in perf_files
+    }
+
+    merged = pd.DataFrame(columns=[TEST_NAME, METRIC_TYPE])
+    suffixes: set[str] = set()
+    for file_path, df in perfs.items():
+        df = df.rename(
+            columns={
+                column: f'{column}_{file_path}'
+                for column in df.columns
+                if column not in (TEST_NAME, METRIC_TYPE)
+            })
+        merged = merged.merge(df, on=[TEST_NAME, METRIC_TYPE], how='outer')
+        suffixes.add(file_path)
+
+    merged[MEAN_COL] = merged[[
+        f'{METRIC_VALUE}_{suffix}' for suffix in suffixes
+    ]].mean(axis=1)
+    merged, name_mapping = shorten_names(merged)
+
+    return name_mapping, merged, suffixes
+
+
+def generate_perf_compare_report(perf_files: list[str],
+                                 output_path: str) -> None:
+    name_mapping, merged, suffixes = parse_perf_data(perf_files)
+    generate_plots(Path(output_path), name_mapping, merged, suffixes)
+
+
+def parse_arguments() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Create a report comparing multiple performance csvs")
+    parser.add_argument('--files',
+                        nargs='*',
+                        help="A list of csv files to compare")
+    parser.add_argument("--output_path",
+                        type=str,
+                        help="Output path for report (pdf file)")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_arguments()
+    generate_perf_compare_report(args.files, args.output_path)
+
+
+if __name__ == '__main__':
+    main()

tests/integration/defs/perf/diff_tools.py

@@ -0,0 +1,86 @@
+from io import StringIO
+
+import numpy as np
+import pandas as pd
+
+PERF_CASE_NAME = 'perf_case_name'
+PERF_METRIC = 'perf_metric'
+THRESHOLD = 'threshold'
+ABSOLUTE_THRESHOLD = 'absolute_threshold'
+METRIC_TYPE = 'metric_type'
+IGNORED_METRICS = {'BUILD_TIME'}
+
+
+def load_file(csv_file: str) -> pd.DataFrame:
+    return pd.read_csv(csv_file)
+
+
+def get_intersecting_metrics(
+    base: pd.DataFrame, target: pd.DataFrame
+) -> tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]:
+    missing_from_target = base.index.difference(target.index)
+    missing_from_base = target.index.difference(base.index)
+
+    cleaned_base = base.drop(missing_from_target).sort_index()
+    cleaned_target = target.drop(missing_from_base).sort_index()
+    return cleaned_base, cleaned_target, base.loc[
+        missing_from_target], target.loc[missing_from_base]
+
+
+def get_diff_exceeding_threshold(
+        base: pd.DataFrame,
+        target: pd.DataFrame) -> tuple[np.array, pd.DataFrame]:
+    diff_exceeding_threshold = ~np.isclose(base[PERF_METRIC],
+                                           target[PERF_METRIC],
+                                           rtol=abs(base[THRESHOLD]),
+                                           atol=abs(base[ABSOLUTE_THRESHOLD]))
+    diff_exceeding_threshold = np.array([
+        diff and base[METRIC_TYPE][i] not in IGNORED_METRICS
+        for i, diff in enumerate(diff_exceeding_threshold)
+    ])
+    diff_mask = np.tile(diff_exceeding_threshold[:, None],
+                        (1, target.shape[-1]))
+    return diff_exceeding_threshold, target.where(diff_mask, base)
+
+
+def get_full_diff(base: pd.DataFrame, target: pd.DataFrame,
+                  missing_from_base: pd.Series, missing_from_target: pd.Series,
+                  diff_over_threshold: np.array) -> pd.DataFrame:
+    PERF_METRIC_BASE = f'{PERF_METRIC}_base'
+    PERF_METRIC_TARGET = f'{PERF_METRIC}_target'
+    thershold_diff = pd.merge(base,
+                              target,
+                              on=PERF_CASE_NAME,
+                              how='outer',
+                              suffixes=['_base', '_target'])
+    if not thershold_diff.empty:
+        thershold_diff = thershold_diff[diff_over_threshold][[
+            PERF_METRIC_BASE, PERF_METRIC_TARGET
+        ]]
+    missing_from_base = missing_from_base.rename(
+        columns={PERF_METRIC: PERF_METRIC_TARGET})[[PERF_METRIC_TARGET]]
+    missing_from_target = missing_from_target.rename(
+        columns={PERF_METRIC: PERF_METRIC_BASE})[[PERF_METRIC_BASE]]
+    return pd.concat([thershold_diff, missing_from_base, missing_from_target])
+
+
+def get_diff(base: pd.DataFrame,
+             target: pd.DataFrame) -> tuple[pd.DataFrame, pd.DataFrame]:
+    relevant_columns = base.columns
+    base = base.set_index(PERF_CASE_NAME)
+    target = target.set_index(PERF_CASE_NAME)
+    cleaned_base, cleaned_target, missing_from_target, missing_from_base = get_intersecting_metrics(
+        base, target)
+    diff_over_threshold, new_df = get_diff_exceeding_threshold(
+        cleaned_base, cleaned_target)
+    full_diff = get_full_diff(cleaned_base, cleaned_target, missing_from_base,
+                              missing_from_target, diff_over_threshold)
+    return full_diff, pd.concat([new_df, missing_from_base
+                                 ]).reset_index()[relevant_columns]
+
+
+def get_csv_lines(df: pd.DataFrame) -> list[str]:
+    string_buffer = StringIO()
+    df.to_csv(string_buffer, index=False)
+    string_buffer.seek(0)
+    return string_buffer.readlines()

tests/integration/defs/perf/requirements.txt

@@ -0,0 +1,3 @@
+pandas
+numpy<2
+matplotlib