VRScout Agent Orchestration Unity Project

This repository contains a Unity project with a runtime UI and IPC components intended to orchestrate a VR Testing & Gameplay agent inference loop (connect/disconnect, apply actions, trigger presets, and send/receive agent-facing events). It also extends the CLOVR Logging to capture synchronized image, poses, and audio in OpenVR applications.

This project is based on the MIT-licensed work by Esteban Segarra Martinez (2024).

https://github.com/xrtlab/clovr

This is one of the three repos that together make up the VR Agent.

You can find the agent policy and inference server code here: https://github.com/wrainw/VRScout_Agent

And you can find the virtual devices driver code here: https://github.com/wrainw/VRScout_openvr_virtualdriver

You can also find the WASAPI audio capture wrapper code here: https://github.com/wrainw/WASAPI_AudioCaptureWrapper

Quick Start (Unity)

1. Open the Unity project under:

   Unity Project/VR 1.0

2. Open a scene that contains the orchestration UI and IPC components. Common entry points:

   • Assets/Scenes/test-scene-5-for-bs-pose-reset.unity (example scene for orchestration UI and IPC components)
   • Assets/Packages/CLOVR_Scene_26_01.unity (modified scene for synchronised VR gameplay capture)

3. Press Play.

VR-Agent / Inference Orchestration

Inference client (Unity -> Python)

Assets/Scripts/IPCClient/InferencePoseClient.cs

• Sends state to an inference server and receives actions.
• Default inference host/port is configured in the Inspector.
• Includes optional audio capture/streaming integration.

Orchestration controls (keyboard + TCP)

Assets/Scripts/IPCClient/InferenceClientKeyboardController.cs

This component provides:

• Keyboard shortcuts to control connection and runtime behaviors.
• A TCP server (default port 12345) to receive commands from an external agent/controller process.

Default keyboard shortcuts

• F1: Start inference connection (StartConnection)
• F2: Stop inference connection (StopConnection)
• F3: Toggle automatic OpenVR pose updates
• F4: Reset transforms

OpenVR device control (Unity -> OpenVR virtual devices)

Assets/Scripts/IPCClient/OpenVRDeviceController.cs

This component is the Unity-side bridge to the OpenVR virtual devices driver.

• Sends HMD / controller / tracker poses over a named pipe (VirtualDevicesControlPipe)
• Can inject controller inputs (button and analog states)
• Supports saving initial transforms as presets and resetting devices back to preset transforms (ResetTransformsToPreset)

Action Presets (repeatable “macro” actions)

Presets let you capture device transforms + a control event, then replay them.

• Config file:

  StreamingAssets/ActionPresets/action_presets.json

• Usage guide:

  Assets/Scripts/IPCClient/README_PresetActions.md

Typical workflow:

• Configure “Next Preset …” fields in the Inspector
• Press F6 to record a preset
• Use a preset key binding (or call ExecutePreset(...)) to replay

Game Events (Unity -> Agent notifications)

Assets/Scripts/IPCClient/GameEventManager.cs

The game event system monitors runtime status (e.g., connection state) and triggers actions such as:

• Execute a preset
• Send a TCP message to an agent
• Log messages

Defaults:

• Config file:

  StreamingAssets/GameEvents/game_events_config.json

• Agent notification TCP port:

  12346

More details:

Assets/Scripts/IPCClient/README_GameEventManager.md

Notes

• The orchestration pieces are primarily:

  • InferencePoseClient
  • InferenceClientKeyboardController
  • GameEventManager
  • Preset + GameEvent JSON configs under StreamingAssets

• Please refer to the upstream README for CLOVR logging setup instructions:

  https://github.com/xrtlab/clovr