CI: Performance regression tests update

[amirkl94]

This PR updates the performance regression tests running in CI.

The changes include:

1. l0_perf tests update.
2. Update sanity_perf_check.py script to generate patch file for differing metrics.
3. A tool for generating comparison reports between 2 performance test results.
4. Updating base_perf.csv.

Example plot of report (pytorch inference time):

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2160 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2160 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1567 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2181 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2181 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1574 completed with status: 'SUCCESS'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2298 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2298 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1653 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2327 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2327 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1674 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2443 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2443 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #1754 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2462 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2462 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1766 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2630 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2630 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1872 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2646 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2646 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1886 completed with status: 'SUCCESS'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #2939 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2939 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2060 completed with status: 'FAILURE'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #3044 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #3044 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2128 completed with status: 'SUCCESS'

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #6917 [ run ] triggered by Bot

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #6917 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5020 completed with status: 'FAILURE'

[amirkl94]

/bot run

[amirkl94]

/bot run

[amirkl94]

/bot run

[tensorrt-cicd]

PR_Github #6942 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #6942 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5040 completed with status: 'FAILURE'

[kaiyux]

/bot run --only-multi-gpu-test

[tensorrt-cicd]

PR_Github #7002 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #7002 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5082 (Partly Tested) completed with status: 'FAILURE'

[kaiyux]

/bot run

[tensorrt-cicd]

PR_Github #7115 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #7115 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5140 completed with status: 'SUCCESS'