Skip to content

feat: Massive performance overhaul for editor playback#1447

Merged
richiemcilroy merged 31 commits intomainfrom
video-playback
Dec 16, 2025
Merged

feat: Massive performance overhaul for editor playback#1447
richiemcilroy merged 31 commits intomainfrom
video-playback

Conversation

@richiemcilroy
Copy link
Copy Markdown
Member

@richiemcilroy richiemcilroy commented Dec 10, 2025

Significant improvements to the frame streaming pipeline for the desktop app, focusing on WebSocket frame transmission, compression, and performance monitoring. The main changes include adding frame compression and latency tracking on both the backend (Rust) and frontend (TypeScript), updating dependencies to support LZ4 compression, and enhancing performance logging for debugging and optimization.


Note

Massive playback/rendering perf revamp: WebGPU worker + SharedArrayBuffer frame path, pipelined GPU readback, GPU YUV→RGBA, smarter prefetch/cache; plus mode-cycling hotkey/tray UI and build/deps updates.

  • Rendering/Frames:
    • WebSocket frames now pack stride/height/width; switch to watch-based streaming with per-frame created_at.
    • New worker-driven pipeline (frame-worker.ts) with OffscreenCanvas + WebGPU renderer and optional SharedArrayBuffer fast path.
    • GPU YUV→RGBA conversion (NV12/YUV420P), IOSurface zero‑copy on macOS, and pipelined GPU readback.
  • Playback:
    • Smarter prefetching (ahead/behind), LRU frame cache, reduced stalls; non-blocking renderer and tuned timing.
  • Editor/Screenshot:
    • Canvas handoff to worker, debounced sizing, ImageBitmap fallback; strip padded rows when sending frames.
  • App UX:
    • New recording-mode cycle hotkey and tray submenu with dynamic icons; added set_recording_mode command.
    • Mode Select window/UI redesigned (smaller, images), fixed window sizing.
  • Backend/Recovery:
    • Mark unrecoverable studio recordings; numerous log/format fixes; Windows/macOS capture/encoder tweaks.
  • Build/CI/Deps:
    • Add lz4_flex, lru, wgpu-core/hal, Metal/ObjC deps; frontend adds lz4-wasm, WebGPU types, WASM/top-level-await plugins.
    • CI runs cargo clippy directly with -D warnings.

Written by Cursor Bugbot for commit 33625f0. This will update automatically on new commits. Configure here.

Summary by CodeRabbit

  • New Features

    • Added hotkey support to cycle through recording modes (Studio, Instant, Screenshot).
    • Added mode selection submenu in system tray.
    • Redesigned recording mode selection interface with improved visuals.
  • Performance Improvements

    • Implemented multi-threaded frame processing with worker optimization.
    • Added advanced frame buffering and prefetching strategies.
    • Enhanced GPU rendering with dual-buffer and streamlined readback pipeline.
  • Bug Fixes & Refinements

    • Improved logging consistency and reduced per-frame overhead.
    • Enhanced platform-specific rendering support.

✏️ Tip: You can customize this high-level summary in your review settings.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant