Skip to content

Rewrite App Profiling Insights docs as a dashboard reference; add the Comparison tab, p90 views and threshold alerts - #3198

Closed
hardikshah197 wants to merge 5 commits into
LambdaTest:mainfrom
hardikshah197:te-12492-app-profiling-metrics-update
Closed

Rewrite App Profiling Insights docs as a dashboard reference; add the Comparison tab, p90 views and threshold alerts#3198
hardikshah197 wants to merge 5 commits into
LambdaTest:mainfrom
hardikshah197:te-12492-app-profiling-metrics-update

Conversation

@hardikshah197

@hardikshah197 hardikshah197 commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Rewrites the App Profiling Insights page as a reference for what the dashboard actually does, and documents the new Comparison tab.

The previous version was advisory rather than descriptive — sections like Advanced Analysis Techniques, Correlation Analysis, Performance Budgeting and Continuous Monitoring explained performance practice in general but did not tell a reader what any control on the page does. It is replaced with a walkthrough of the real UI: how to reach it, every filter, every widget, the SLA model, and the alerts.

What the page now covers

  • How to access and the two tabs — Trends and Comparison.
  • Filter bar — every filter with its values and default, and the fact that dropdowns only list values present in the selected tests.
  • Saved filters — filters, date range and widget layout auto-save at org level; no Save button.
  • Average and p90 views — what p90 means here and why it matters for regression analysis.
  • Performance Overview, Performance Trends, Device Performance Matrix, Label Page Load Time, and the eight per-metric trend widgets, including Disk Read / Disk Write.
  • Compare mode — the per-widget dimension overlay.
  • SLA thresholds — the default bands, where they render, and that they are org-level and admin-owned.
  • Threshold breach alerts — the email and Slack notifications, with screenshots of both.

Comparison tab

A new section documenting the session-level overlay view, placed after Compare mode so the two "compare" concepts sit together and can be told apart — one overlays dimensions on a single widget, the other overlays whole test sessions.

  • Complete flow — six steps from opening the tab to zooming the timeline.
  • Test Sessions rail — a table of every control: search, Test Name, date range, the More menu (OS, App Build Version, Device, Status), Clear Filters, the selection checkbox and Set as Baseline.
  • How the graphs are drawn — one series per session with colours stable across every card, the shared elapsed-time axis and drag-to-zoom, legend toggles, per-metric variant toggles, SLA bands, and the Avg / Min / Max / P90 stats table. Startup Time is called out separately as a grouped bar chart with its Load Time variant.
  • Baseline and Baseline Diff — how the deltas read, and that the direction counting as "better" follows the metric.
  • Saved comparisons — the rail filters, selected sessions, baseline, Baseline Diff toggle and hidden series all persist per test at org level. The date range deliberately does not: it is owned by the shared page picker, so both tabs always report over the same window.

The framing the section leads with: Trends charts calendar time, Comparison charts elapsed time within each run — every session starts at 0:00. That is what lets runs of different lengths and start times be read against one another, and it is the thing a reader needs before the charts make sense.

Screenshots

Five images. Two for the alert formats, and three new ones captured from a live Android profiling test with three sessions overlaid:

Image Shows
app-profiling-comparison.png The overlaid view — CPU chart, SLA bands, per-session stats table with baseline deltas
app-profiling-comparison-filters.png The Test Sessions rail with the More filter menu open
app-profiling-comparison-startup.png Startup Time — Cold/Hot bars against their SLA lines, with baseline deltas

All three were reviewed before committing: they show test names, truncated Test IDs and device models only — no email, org name or account details.

Notes for review

  • I kept Docusaurus's auto-generated heading slug for the #comparison cross-link rather than pinning an explicit {#comparison} id, because no other doc in this repo uses that syntax and onBrokenAnchors is not configured (it defaults to warn, not throw). Worth a second opinion if you'd prefer the explicit form.
  • Two {/* TODO screenshot: ... */} placeholders remain, for the Average/p90 toggle and the Performance Trends KPI selector. They are comments, not broken image references, but they are still open.
  • I have not run a local Docusaurus build to confirm the MDX compiles — relying on CI for that.

@hardikshah197
Add App Profiling email and Slack alert images; update documentation …
764783d 
…for App Profiling Insights
@hardikshah197
Document the App Profiling Comparison tab
85bc2cd 
Adds a Comparison section to the App Profiling Insights page covering the
session-level overlay view: what it is for, the end-to-end flow, the Test
Sessions rail and its filters, how the charts are drawn, the baseline and
Baseline Diff behaviour, and what gets saved between visits.

- Notes the two tabs under "How to access" and cross-links Comparison.
- Explains the elapsed-time X-axis, which is what distinguishes these charts
  from the calendar-time charts on Trends.
- Documents the five-session cap and the at-least-one-selected rule.
- Records that the rail filters, selected sessions, baseline, Baseline Diff and
  hidden series are auto-saved per test at org level, while the date range is
  owned by the shared page picker and is not part of the saved comparison.
- Updates "Sharing a dashboard": the URL now round-trips the active tab.
- Adds session comparison / baseline diff keywords and mentions Comparison in
  the page description.

Three screenshots captured from a live Android profiling test: the overlaid
view with the stats table, the rail with the More filter menu open, and the
Startup Time card showing Cold/Hot bars, SLA lines and baseline deltas.
@hardikshah197 hardikshah197 changed the title Add App Profiling email and Slack alert images; update documentation for App Profiling Insights Rewrite App Profiling Insights docs as a dashboard reference; add the Comparison tab, p90 views and threshold alerts Aug 4, 2026
@hardikshah197
Extend the Average/p90 guidance and convert the Comparison screenshot…
156350c 
…s to WebP

Addresses the two self-review findings.

Average / p90:
- The toggle lives in the Trends top bar, not "every widget on the dashboard".
  It is rendered only when the Comparison tab is inactive, so the old wording
  became wrong the moment this page documented a second tab.
- Adds "Which one to read" — what each aggregation answers, a table of when to
  reach for which, and the point that Average and p90 sitting close together is
  itself a signal the metric is stable.
- Records how p90 is actually derived: the mean of each session's own
  pre-computed 90th percentile, not the percentile of the whole window. Notes
  which metrics have no p90 and render as an em dash (cold/hot startup, crash
  and ANR counts) and why frozen/janky frames stay on their average.
- Adds "p90 on the Comparison tab": there is no toggle there and the Trends
  toggle does not carry across. Every Comparison stats table reports Avg, Min,
  Max and P90 together for each selected session, which is the more useful
  shape for individual runs. Cross-linked from the Comparison section.

Screenshots:
- Convert the three Comparison PNGs to WebP: 1.2 MB -> 220 KB. WebP is the
  dominant format in this directory (72 files vs 40 PNG) and the comparable
  app-profiling-screenshot.webp is 120 KB, so the PNGs were ~10x heavier than
  convention. Text remains legible at quality 82.
@hardikshah197
Correct two Comparison behaviours the doc described wrongly
25d5a6c 
Both were caught re-reading the doc against the reducer.

Auto-selection: the doc said "the run you opened is selected automatically".
pickAutoSelection sorts the filtered sessions byNewest and takes the first, and
the URL's testId is never passed down to the Comparison component — so opening
the dashboard from an older run and switching to Comparison selects a different
session than the doc promised. Reworded to "the most recent run of that test",
with the caveat spelled out.

Rail ordering: the doc said "newest first". buildOrderedSessions returns
[...picked, ...rest], so sessions selected for comparison are pinned to the top
and only the remainder is newest-first. A reader selecting an older run and
watching it jump to the top would otherwise think the sort was broken.

Also expands the Clear Filters row while in the same table: it resets the
comparison to the test's most recent run as well as resetting the filters, and
the date range is preserved because it belongs to the shared picker.
@hardikshah197
Shrink every image under 100 KB, tighten the meta description, rebran…
722ef05 
…d site_name

Images — all six the page references are now WebP and under 100 KB:
- Convert the email and Slack alert screenshots from PNG to WebP at quality 88;
  264 KB -> 80 KB and 192 KB -> 60 KB. Checked at full size first: the threshold
  values, per-metric deltas and the footer URL all stay legible.
- app-profiling-screenshot.webp was 120 KB. Re-encoding an already-lossy WebP
  made it larger, so the lever was its size: it was 1825 px wide but rendered at
  768. Downscaled to 1200 px (still ~1.6x for retina) -> 84 KB.
- That image's height attribute said 373 for a 1.29 aspect ratio, which never
  matched and would shift the layout as it loads. Corrected to 594.

Meta description trimmed from 297 to 159 characters. Note the first attempt
started "App Profiling Insights: CPU, ..." — the colon-space made the front
matter invalid YAML and would have broken the page build, so the wording avoids
the colon rather than quoting around it.

site_name: LambdaTest -> TestMu AI, matching the url and canonical fields that
already point at testmuai.com.

Removes both {/* TODO screenshot */} placeholders. The p90 one is obsolete now
that the section documents the toggle in prose; the KPI-selector one was the
same class of leftover.
@hardikshah197

hardikshah197 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Superseded — reopening against the correct release branches.

This PR targeted main, which is well behind and is not the branch this content ships from. stage and testmuCom already carried a newer, TestMu-AI-rebranded copy of the page, so merging here would not have reached either.

Replaced by, with identical content:

Closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hardikshah197