[HOLD for payment 2024-06-20] [HOLD for payment 2024-06-18] [LOW] Performance - E2E warmup results are included in the final measurements

[mountiny]

Problem

The warmup results are included in the final measurements. The warmup stage is needed to prepare a network cache for subsequent runs.

Typically, the warmup stage takes much more time than subsequent runs (because it sends real requests and waits for real results). Keeping such records in final measurements leads to unpredictable results (because network request depends on many variables: server responsiveness, speed of the network, etc.). From my observations, I've seen anomaly variables with values ~6-8s (while the average run with cache takes 400ms) -> obviously 8s will significantly increase the average and thus will bias real measurements.

Solution

We don't need to include warmup measurements in the final report and we need to add a mechanism to exclude it.

Issue Owner

Current Issue Owner: @laurenreidexpensify

[melvin-bot]

Triggered auto assignment to @laurenreidexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[kirillzyusko]

Problem

The warmup results are included in final measurements. The warmup stage is needed to prepare a network cache for subsequent runs.
Typically warmup stage takes much more time than subsequent runs (because it sends real request and wait for real results). Keeping such records in final measurements leads to unpredictable results (because network request depends on many variables: server responsiveness, speed of the network, etc.). From my observations I've seen anomaly variables with values ~6-8s (while average run with cache takes 400ms) -> obviously that 8s will significantly increase the average and thus will bias a real measurements.

Solution

We don't need to include warmup measurements in the final report and we need to add a mechanism to exclude it.