Flutter embedder step 3a: live texture bridge into the GUI - #31
Merged
Conversation
Out-of-process guest engine renders an animated, interactive scene; the flutterware GUI displays it live in a Texture via a shared IOSurface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SCM_RIGHTS transfers file descriptors, not mach ports. IOSurfaces are shared across processes by their global IOSurfaceID via IOSurfaceLookup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The guest dumps its first frame as a step-2-format raw file via --capture-raw, so run.dart can encode a PNG without reading an IOSurface from Dart. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two plans: Plan 1 builds the headless out-of-process guest and wire protocol; Plan 2 adds the macOS texture plugin and dev-harness GUI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The field has no zero enumerator (kFlutterKeyEventDeviceTypeKeyboard = 1); leaving it zero-initialized makes the engine reject key events. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The StreamController is closed via the socket's onDone handler. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Side effect of running `flutter build macos`: deployment target bump to 10.15, @main + applicationSupportsSecureRestorableState, scheme/Podfile.lock refresh. Committed separately to keep the embedder change focused. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A macOS app launched by `flutter run` has a stripped environment and a `/` working directory, so FlutterSdkPath.findSdks() and Directory.current both fail. Take FLUTTERWARE_APP_ROOT and FLUTTER_SDK_ROOT as dart-defines, with SDK discovery as a fallback and a clear error screen when unset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Runtime fixes found by running the GUI harness: - surface.c: mark IOSurfaces global so the GUI process can resolve them with IOSurfaceLookup (cross-process lookup fails otherwise). - EmbedderTexturePlugin: create the CVPixelBuffer with the Metal compatibility key, so the Flutter engine's Metal texture cache accepts it (was failing per-frame with kCVReturnPixelBufferNotMetalCompatible). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- FrameReady now carries the surface-ring generation; EmbeddedEngine discards frames composited against superseded surfaces, honoring the spec's resize-race contract. - EmbeddedEngine.start races the socket accept against guest exit, so a guest that dies before connecting surfaces an error instead of hanging. - Fix a stale build_guest.dart doc comment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resizing the harness window left it white at some sizes and flashed at others. Root causes and fixes: - surface.c: stop forcing kIOSurfaceBytesPerRow to width*4. That stride is not always validly aligned, so IOSurfaceCreate failed for some widths and surface_ring_init tore down the ring with no replacement. Let IOSurface choose an aligned stride (downstream reads IOSurfaceGetBytesPerRow). - surface.c: build the new ring into temporaries and swap atomically, so a failed re-allocation keeps the existing ring presentable; reject zero-size. - embedded_engine.dart: handle socket messages strictly in arrival order via a future chain instead of unawaited concurrent handlers, so the generation gate sees an up-to-date generation and stops discarding valid frames. - embedder_harness_screen.dart: skip degenerate zero-size resize events. Transient flicker during an active resize drag remains (a full software relayout per resize); frame-perfect resize sync is out of scope for 3a. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 3a of the experimental Flutter engine embedder: the embedded engine's live, animated, interactive output is now displayed inside the flutterware desktop GUI via an external
Texture.app/native/) became a long-lived guest process: runs the engine with the software renderer, copies each composited frame into sharedIOSurfaces, and speaks a framed wire protocol over a Unix domain socket (SurfacesAllocated/FrameReady/Resize/PointerEvent/KeyEvent/Shutdown). Split intohost.c+surface.c+ipc.c+input.c.EmbeddedEnginebuilds/spawns the guest and bridges the protocol; a macOS plugin (EmbedderTexturePlugin.swift) registers an externalFlutterTextureand wraps theIOSurfaces asCVPixelBuffers; a dev-harness screen (main_embedder_dev.dart) hosts the liveTexturewith resize and pointer/keyboard forwarding.Built per the design + plans under
docs/superpowers/(2026-05-15-flutter-embedder-step3a-*). Two later steps remain: 3b (Metal/zero-copy) and 4 (hot reload).Test plan
cd app && dart test test/embedder integration_test/embedder— 18 tests pass (protocol_test,raw_frame_test,flutter_cache_test,compiler_test,live_bridge_test)flutter analyze— clean workspace-widedart run app/tool/embedder/run.dart— headless smoke renders the scene to a PNGFlutterEmbedder.framework, ~31 MB)Known limitations (scoped out of 3a)
🤖 Generated with Claude Code