feature: add real-time Rerun URDF visualizer for robot arms (#845)

[KrishnaH96]

Problem

Robot arm visualization is currently tied to Drake Meshcat, which lives in a separate browser window and sits outside the core dimOS observability workflow.

This creates a fragmented developer experience when monitoring manipulation together with navigation maps, sensor streams, and point clouds in one place.

This PR delivers a unified visualization path by adding a Rerun-native URDF visualizer for robot arms.

Closes #845

---

Solution

Implemented a real-time URDF-to-Rerun pipeline so robot arms render and update directly in the Rerun viewer.

Highlights

• Pinocchio-based FK: Uses the existing pinocchio dependency to compute forward kinematics on each JointState update.
• Automatic mesh compatibility: Reuses the existing mesh conversion path to transform .stl / .dae assets into .obj for Rerun Asset3D.
• Robust joint name mapping: Handles namespace differences between runtime joint names and URDF joint names.
• Config-level toggle: Adds enable_rerun_viz to ManipulationModuleConfig.
• Blueprint defaults: Enables Rerun visualization by default in the relevant keyboard teleop blueprints.

---

Breaking Changes

• None. Existing Drake/Meshcat visualization remains supported via enable_viz.

---

Files Updated

dimos/visualization/rerun/robot.py (new)

• Added RobotVisualizer to load URDF visuals and publish geometry and transforms to Rerun.
• Uses prepare_urdf_for_drake(..., convert_meshes=True) so .stl / .dae assets are converted to .obj for Rerun Asset3D.
• Uses Pinocchio forward kinematics to compute per-link transforms from incoming JointState messages.
• Handles joint namespace mismatches by attempting exact match first, then prefix-stripped matching.

dimos/manipulation/manipulation_module.py (modified)

• Added enable_rerun_viz: bool = False to ManipulationModuleConfig.
• Initializes RobotVisualizer instances when Rerun visualization is enabled.
• Updates planning state and Rerun visualization from the same joint-state stream.

dimos/robot/manipulators/xarm/blueprints.py (modified)

• Enabled enable_rerun_viz=True in xArm keyboard teleop blueprints.

dimos/robot/manipulators/piper/blueprints.py (modified)

• Enabled enable_rerun_viz=True in Piper keyboard teleop blueprints.

dimos/visualization/rerun/__init__.py (new)

• Added package initialization for the rerun visualization module.

---

Demonstration

Watch the real-time arm movement in Rerun here

---

How To Test

Use a 3-terminal validation flow.

Terminal 1 (Viewer)

rerun

Terminal 2 (Bridge)

dimos rerun-bridge --viewer-mode connect

Terminal 3 (Simulation)

CI=1 dimos run keyboard-teleop-xarm7

Verification Steps

1. Confirm the xArm7 model appears in the Rerun 3D scene.
2. Click the Keyboard Teleop (Pygame) window to focus it.
3. Press and hold W or S.
4. Verify that the robot arm in Rerun moves in sync with the teleop updates.

---

Contributor License Agreement

• I have read and approved the CLA.

[greptile-apps]

Greptile Summary

This PR adds a real-time Rerun URDF visualizer for robot arms using Pinocchio for forward kinematics. It introduces dimos/visualization/rerun/robot.py with a RobotVisualizer class, wires it into ManipulationModule behind an enable_rerun_viz config flag, and enables it by default in the xarm7 and piper keyboard-teleop blueprints.

• P1: robot_config.model_path in manipulation_module.py will raise AttributeError at runtime since RobotModelConfig exposes urdf_path, not model_path. The error is caught silently, so Rerun visualization will always be silently disabled when enable_rerun_viz=True, making the feature non-functional as shipped.

Confidence Score: 4/5

Not safe to merge as-is — the Rerun visualizer is non-functional due to a wrong attribute name that is silently swallowed.

One P1 defect (model_path vs urdf_path) prevents the feature from working at all. The error is caught and logged but never surfaced as a hard failure, so users see no visualization with no clear indication why. Fixing it is a one-line change. Remaining findings are P2 style/correctness suggestions.

dimos/manipulation/manipulation_module.py (line 203) and dimos/visualization/rerun/robot.py (sphere radii call)

Important Files Changed

Filename	Overview
dimos/visualization/rerun/robot.py	New file: Pinocchio-based URDF visualizer for Rerun. Contains unused imports (numpy, Any), hppfcl imported inside a per-geometry loop, and a potentially incorrect sphere primitive call.
dimos/manipulation/manipulation_module.py	P1 bug: robot_config.model_path does not exist on RobotModelConfig (field is urdf_path); silently swallowed by except Exception, disabling the feature entirely.
dimos/robot/manipulators/piper/blueprints.py	Refactored to use catalog helpers; enables enable_rerun_viz=True; no issues found.
dimos/robot/manipulators/xarm/blueprints.py	Refactored to use catalog helpers; enables enable_rerun_viz=True; no issues found.
dimos/visualization/rerun/init.py	Empty __init__.py to establish the subpackage; fine.

Sequence Diagram

sequenceDiagram
    participant Coord as ControlCoordinator
    participant MM as ManipulationModule
    participant RV as RobotVisualizer
    participant Pin as Pinocchio FK
    participant Rerun as Rerun Viewer

    Note over MM: start() → _initialize_planning()
    MM->>RV: RobotVisualizer(urdf_path, entity_path, ...)
    RV->>Pin: buildModelFromUrdf()
    RV->>Rerun: rr.log(visuals, static=True)

    loop Every JointState tick
        Coord->>MM: _on_joint_state(msg)
        MM->>MM: split joints per robot
        MM->>RV: update_joints(sub_msg)
        RV->>Pin: forwardKinematics(model, data, q)
        RV->>Rerun: rr.log(transforms per joint)
    end

Loading

_{Reviews (1): Last reviewed commit: "feature: add real-time Rerun URDF visual..." | Re-trigger Greptile}