Viser Manipulation Module

[TomCC7]

Problem

Going to replace the drake meshcat and provide a moveit style console.

Closes #2412

Solution

1. Viser as a dimos module
2. most of communication through rpc
3. moveit style dragging and plan-review-execute
4. small issue: ik kind of slow (currently through rpc to manipulation module), not sure it's rpc or ik impl issue

How to Test

uv run --extra manipulation-viser dimos run xarm7-viser-panel-mock

Contributor License Agreement

• I have read and approved the CLA.

[greptile-apps]

Greptile Summary

This PR introduces ViserManipulationPanelModule, a browser-based manipulation operator panel built on viser that replaces the Drake meshcat visualizer and provides a MoveIt-style drag-to-target plan-preview-execute workflow. Communication with ManipulationModule flows over RPC, new endpoints are added, and a mock xArm7 demo blueprint is wired up.

• New viser_panel package: poll-based session state machine, ghost-robot URDF preview, path line visualization using real FK poses, and a PreviewWorker queue for debounced IK evaluation on drag.
• ManipulationModule additions: evaluate_pose_target, evaluate_joint_target, get_planned_path, and get_planned_path_poses RPC endpoints, plus enriched get_robot_info response.
• xarm7_viser_panel_mock blueprint registered in all_blueprints.py and __all__; manipulation-viser optional dependency group added to pyproject.toml.

Confidence Score: 4/5

Safe to merge after addressing the concurrent planning defect in module.py.

The Plan button never transitions action_status away from IDLE, so rapid double-clicks can dispatch two concurrent plan_to_joints RPC calls that race to write session.plan_state. The rest of the module is well-structured.

dimos/manipulation/viser_panel/module.py — specifically the _run_operation action-status map and _can_plan_for_operation guard.

Important Files Changed

Filename	Overview
dimos/manipulation/viser_panel/module.py	Core panel module: concurrent planning defect — Plan maps to ActionStatus.IDLE so multiple planning threads can run simultaneously
dimos/manipulation/viser_panel/backend.py	RPC client wrapper: preview-busy guard correct; reset_client silently leaks old remote connection when preview thread is alive
dimos/manipulation/viser_panel/state.py	State dataclasses and PreviewWorker: queue-based debounce clean, can_execute guards well-designed; no issues found
dimos/manipulation/viser_panel/scene.py	Viser scene rendering: path visualization uses real FK poses with None filtering; URDF ghost/current robot handling correct
dimos/manipulation/manipulation_module.py	New RPC endpoints have proper None guards and error returns; no issues found
dimos/manipulation/blueprints.py	xarm7_viser_panel_mock blueprint added and correctly included in all
dimos/manipulation/viser_panel/animation.py	Linear interpolation for joint-path preview animation; logic sound for normal inputs

Sequence Diagram

sequenceDiagram
    participant Browser
    participant ViserServer
    participant PanelModule
    participant PreviewWorker
    participant PollThread
    participant ManipulationModule

    PollThread->>ManipulationModule: list_robots() / get_robot_info()
    ManipulationModule-->>PollThread: robot list + info
    PollThread->>PanelModule: update session state

    Browser->>ViserServer: drag ee_control handle
    ViserServer->>PanelModule: _target_pose_changed(target)
    PanelModule->>PreviewWorker: submit(PreviewRequest)
    PreviewWorker->>ManipulationModule: evaluate_pose_target(pose)
    ManipulationModule-->>PreviewWorker: result
    PreviewWorker->>PanelModule: _apply_preview_result()
    PanelModule->>ViserServer: update ghost robot + feasibility color

    Browser->>ViserServer: click Plan
    ViserServer->>PanelModule: _run_operation [new thread]
    PanelModule->>ManipulationModule: plan_to_joints(target)
    ManipulationModule-->>PanelModule: ok
    PanelModule->>ViserServer: render_plan_path

    Browser->>ViserServer: click Execute
    ViserServer->>PanelModule: _run_operation [new thread]
    PanelModule->>ManipulationModule: execute(robot)
    ManipulationModule-->>PanelModule: ok

Loading

_{Reviews (7): Last reviewed commit: "fix: skip failed fk poses in viser path ..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 77.04113% with 374 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/manipulation/viser_panel/module.py	55.90%	191 Missing and 22 partials ⚠️
dimos/manipulation/viser_panel/gui.py	43.47%	48 Missing and 4 partials ⚠️
dimos/manipulation/viser_panel/scene.py	68.46%	26 Missing and 15 partials ⚠️
dimos/manipulation/manipulation_module.py	65.62%	11 Missing and 11 partials ⚠️
dimos/manipulation/viser_panel/backend.py	69.44%	16 Missing and 6 partials ⚠️
dimos/manipulation/viser_panel/state.py	94.33%	5 Missing and 4 partials ⚠️
dimos/manipulation/viser_panel/animation.py	79.48%	4 Missing and 4 partials ⚠️
dimos/manipulation/viser_panel/controller.py	76.92%	6 Missing ⚠️
dimos/manipulation/test_viser_panel.py	99.77%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

None poses crash render_plan_path

get_planned_path_poses in manipulation_module.py returns [world_monitor.get_ee_pose(robot_id, joint_state=waypoint) for waypoint in path], where get_ee_pose is typed as -> Pose | None. When FK fails for any waypoint (e.g., malformed joint count or world-monitor state change), that slot is None. The list comprehension on line 106 then raises AttributeError: 'NoneType' object has no attribute 'position' at float(pose.position.x). Both _plan_impl and _preview_impl hit this path and pass the unchecked list directly to render_plan_path.