This work is sponsored by the NSF Future of Work at the Human-Technology Frontier award #2128743. Any opinions, findings, or conclusions expressed in this material are those of the authors and do not reflect the views of the NSF.
We introduce psiUnity, an open-source C# integration that bridges \psi’s .NET libraries with Unity 2022.3 and MRTK3 for HoloLens 2. psiUnity enables bidirectional, real-time streaming of head pose, hand tracking, gaze, IMU, audio, and depth sensor data (AHAT and long-throw) with microsecond-level temporal precision—allowing Unity applications to both consume and produce synchronized multimodal data streams. By embedding \psi’s native serialization, logging, and temporal coordination directly within Unity’s architecture, psiUnity extends \psi beyond its previous StereoKit limitations and empowers the HRI, HCI, and embodied-AI communities to develop reproducible, data-driven XR interactions and experiments within the familiar Unity environment.
This repository contains the following components:
- UnityPackage - The main source code for the Psi-Unity Integration package.
- SampleProject - A sample Unity project preconfigured for MRTK3, the HoloLens 2 headset, and our sample scene demonstrating how to use the integration.
- psi - A subset of the \psi codebase containing modifications to its core libraries that enable it to run within a Unity environment.
The Platform for Situated Intelligence (\psi) is a cross-platform framework designed to facilitate research on multimodal AI systems by providing infrastructure for processing multimodal sensor data streams and tools for visualizing, synchronizing, and analyzing that data. \psi includes support for developing mixed reality applications for the HoloLens 2 headset and capturing data streams from its vast array of sensors. However, this existing support is designed for StereoKit applications, which constrains its adoption given that Unity and MRTK—the most popular development stack for HoloLens mixed reality applications—are not currently supported. We have addressed this limitation by enabling Unity-based HoloLens 2 applications to integrate with the broader \psi ecosystem. \psi provides a sample end-to-end HoloLens Capture pipeline that includes both a HoloLens app to capture and stream the sensor data and a remote server to log these data streams for future analysis. Our integration in particular is based on the HoloLens Capture App.
This integration assumes you already have a development environment set up for building HoloLens applications with Unity. Specifically, that you:
- have installed the tools required for general HoloLens development, as outlined here.
- have set up an MRTK3 project with Unity 2022.3 LTS, as outlined here.
- are familiar with testing and deploying your application to a HoloLens device, as outlined here.
Your HoloLens 2 device should be configured for Research Mode in order to provide access to an extended set of sensor streams. For more information on HoloLens Research Mode and instructions on how to set this up via the Device Portal, please see this documentation.
You should ensure your Windows system is configured to support long paths, as outlined here.
In order to run the HoloLens Capture Server that your app will connect to, please follow the steps for building the \psi codebase, as outlined on this wiki page. Then verify you are able to start the HoloLens Capture Server console app.
We provide a sample scene that recreates \psi's HoloLens Capture App in Unity. A sample Unity project is included in this repo that is ready to be deployed and tested on a HoloLens 2 headset. Please note, you must be signed in to the Unity Hub in order for Unity to be able to download certain Asset Store package dependencies upon first launching the project. Additionally, please make sure to configure the server endpoint to connect to, as outlined here. Alternatively, if you have already completed the steps below to set up the Psi-Unity Integration, you can import the sample scene directly into your project by using the Unity Package Manager.
Given that this integration relies on the \psi libraries along with its dependencies, you will need to add these dependencies into your Unity project prior to installing the integration in order to avoid compilation errors.
Since \psi doesn't provide direct support for Unity, there isn't a Unity-native way to import it into your project. However, since it is based on the .NET platform and designed for use in .NET applications, it can be imported using the NuGet package manager.
-
Setting Up the Client: Since Unity doesn't provide native NuGet support, you will need to add a third-party NuGet client that can import the packages and configure them for Unity. We recommend using NuGetForUnity. Instructions for its installation and usage can be found in the tool's documentation.
-
Package List: The table below lists the packages that should be installed with the NuGet client. Please note that NuGetForUnity does not support all platforms, and in particular certain packages that contain ARM-specific Windows libraries or UWP-specific libraries may require manual resolution later. You may also see a warning prompt informing you as such when attempting to install such a package - please click the "OK" button to acknowledge the warning and proceed.
| Package Name | Package Version | Is Prerelease |
|---|---|---|
Microsoft.Azure.SpatialAnchors.WinRT* |
2.14.1 (latest) | |
Microsoft.MixedReality.SceneUnderstanding |
0.5.2069 | |
System.ServiceModel.Syndication |
9.0.10 (latest) | |
Microsoft.Psi.Runtime^ |
0.19.100.1-beta (latest) | ✓ |
Microsoft.Psi.Interop^ |
0.19.100.1-beta (latest) | ✓ |
Microsoft.Psi.MixedReality |
0.19.100.1-beta (latest) | ✓ |
Microsoft.Psi.MixedReality.UniversalWindows* |
0.19.100.1-beta (latest) | ✓ |
- * Contains unsupported libaries that need to be manually added and configured.
- ^ Needs to be replaced with custom modified libraries for Unity support.
Some of the packages installed above require manual intervention to prepare them to a state where they are ready to be utilized by the Psi-Unity Integration.
A few of the packages don't seem to get configured properly and remain in a state where they contains duplicate libraries within its contents, resulting in compilation errors in Unity. To remove the duplicate libraries, simply delete the "runtimes" folder for each of the following packages:
Microsoft.MixedReality.SceneUnderstandingMicrosoft.VCRTForwarders(installed automatically as a dependency)
Given the intricate nature of this section, a preconfigured .unitypackage file (linked here) that addresses the steps below has been provided for your convenience. Simply follow these steps to import it into your project and continue on to the next section.
As mentioned earlier, the NuGetForUnity client does not currently support importing Universal Windows Platform (UWP) libaries nor Windows libraries compiled for ARM architectures. Since the HoloLens 2 runs a Windows Holographic operating system (OS) on an ARM-based processor, these libraries are necessary to build and deploy an app to the headset. Below are the specific libraries that contain such libraries that will need to be imported manually, along with the links to their NuGet package listings:
For each of the above packages, download the raw .nupkg file using the "Download package" button found in the "About" section of the right sidebar. Once downloaded, rename the files to .zip files and extract them to your computer.
The next step will be to copy over the relevant files into the corresponding package's folder within your Unity project. Please note that some of the path lengths can be long, which may cause Unity to fail to import all the file contents. If this occurs, you may have to import files into a location that is closer to the root of the Unity Assets folder and subsequently move it to the appropriate location after it has imported. For each of the extracted folders:
- Copy its lib folder into the corresponding package's folder within your Unity project, maintaining the folder structure. For each
.dlllibrary contained within the lib folder's contents, use the inspector Window to restrict it to the WSAPlayer platform. Make sure to click the "Apply" button to save these settings. - Within the runtimes folder, copy the win10-arm and win10-arm64 folders into the matching location of the respective package's folder within your Unity project. You may have to create this runtimes folder if it does not already exist. Similar to the previous bullet, for each
.dllfolder contained in each folder's contents, use the inspector Window to restrict it to the WSAPlayer platform. Additionally, these folders are compiled for specific processor architectures, you’ll also need to set the CPU platform settings to either ARM or ARM64, as appropriate for each folder. Once again, please remember to click the "Apply" button to save these settings.
As mentioned earlier, \psi doesn't natively support Unity. To enable compatibility, we've made modifications to the core Microsoft.Psi.Runtime and Microsoft.Psi.Interop libraries. These modified versions need to be built and imported into your Unity project to replace the packages installed via NuGet. Take the following steps:
- Open the
psi/Psi.slnsolution file (located in this repository) using Visual Studio 2022 or later. - In the Solution Explorer, locate the following projects:
Microsoft.Psi(underRuntime)Microsoft.Psi.Interop(underRuntime)
- Build each project for the Release configuration and ARM64 platform. This ensures the libraries are compatible with the HoloLens 2 device.
- Once built, navigate to the output directories for each project (typically
psi/Sources/Runtime/[ProjectName]/bin/ARM64/Release/net6.0/). - For each project, copy the resulting
.dllfiles into the corresponding NuGet package folder in your Unity project'sAssets/Packages/directory:- Copy
Microsoft.Psi.dllto replace the one inMicrosoft.Psi.Runtime.0.19.100.1-beta/lib/net6.0/ - Copy
Microsoft.Psi.Interop.dllto replace the one inMicrosoft.Psi.Interop.0.19.100.1-beta/lib/net6.0/
- Copy
- In Unity's Inspector Window, configure each replaced
.dllfile:- Set the platform to WSAPlayer only
- Set the CPU to ARM64
- Click "Apply" to save the settings
Once all the dependencies are set up, this integration can be installed using the Unity Package Manager. Please follow these steps for installing a package via a Git URL. The URL to use for this integration is as follows:
https://github.com/sailgt/psiUnity.git?path=/UnityPackage
Create an instance of the PsiCaptureController in your scene. The prefab can be found in the /Runtime/Prefabs/ folder Within the Psi-Unity Integration package.
While most of the references used by the PsiCaptureController are pre-configured and self-contained in the prefab, the EyesStream component requires you to manually set the following scene references:
- Eye Calibration Checker - This is not in an MRTK scene by default but can easily be added onto any GameObject. This component is included as part of the MRTK Input package.
- Gaze Interactor - If using the default MRTK XR Rig, this would be the
FuzzyGazeInteractorcomponent on the GameObject found atMRTK XR Rig / Camera Offset / MRTK Gaze Controller / GazeInteractor.
The Psi-Unity Integration follows a modular architecture designed to seamlessly bridge Unity-based HoloLens 2 applications with the \psi ecosystem. The architecture consists of three main layers:
The PsiCaptureController is the central orchestrator for data capture. It:
- Manages the \psi pipeline lifecycle (creation, running, and disposal)
- Handles connection to the remote HoloLens Capture Server using the Rendezvous protocol
- Coordinates state transitions throughout the capture session
- Synchronizes clocks between the headset and server using
RemoteClockExporter - Monitors server connectivity via heartbeat messages
The controller follows a finite state machine pattern with states including:
WaitingToStart- Initial state before connectionConstructPipeline- Beginning pipeline constructionCalibrateCameras- Calibrating depth cameras (if applicable)ConnectToCaptureServer- Establishing connection to serverWaitingForCaptureServer- Awaiting server acknowledgmentCapturing- Active data capture in progressStoppingPipeline- Gracefully shutting down
Data streams are modular components that capture specific sensor data and publish it to the \psi pipeline. Each stream:
- Inherits from either
DataStream<T>for event-driven data orPeriodicDataStream<T>for time-sampled data - Manages its own TCP connection to the server via
TcpWriter<T> - Handles data serialization using \psi's interop format serializers
- Publishes data with precise originating timestamps
The architecture supports two data capture patterns:
- Periodic Streams: Sample data at regular intervals (e.g., IMU, head pose, eye tracking). These use Unity coroutines to maintain consistent sampling rates.
- Event-Driven Streams: Publish data when new samples become available (e.g., audio). These use Unity's
Update()method to poll and publish data.
The DefaultPsiCaptureController is a concrete implementation that demonstrates how to configure and activate specific data streams. It provides:
- Boolean toggles for enabling/disabling individual streams
- Automatic port assignment for each stream (starting at port 30000)
- Depth camera calibration management
- Integration with Unity's component system for modular stream attachment
- User calls
Connect()on thePsiCaptureController - Pipeline is constructed and depth cameras are calibrated (if needed)
- Controller establishes connection with the Capture Server via Rendezvous
- Each enabled stream is set up with its own TCP endpoint
- Streams begin publishing data to their respective endpoints
- Server receives, logs, and optionally processes the data
- User calls
Disconnect()to gracefully shut down the pipeline
This architecture ensures extensibility (new streams can be easily added), modularity (streams can be independently enabled/disabled), and maintainability (clear separation of concerns between controller, streams, and \psi pipeline).
The integration provides out-of-the-box support for a comprehensive set of HoloLens 2 sensor streams. Each stream can be independently enabled or disabled in the DefaultPsiCaptureController inspector.
The HoloLens 2's Inertial Measurement Unit (IMU) provides motion and orientation data through three sensor streams. Accelerometer (AccelerometerStream) captures linear acceleration in 3D space; Gyroscope (GyroscopeStream) captures rotational velocity; and Magnetometer (MagnetometerStream) captures magnetic field data for compass functionality
- Data Type:
Vector3(x, y, z components) - Sampling Pattern: Periodic
- Default Frequency: 60 Hz
Head (HeadStream) captures the 6-DOF pose (position and orientation) of the HoloLens headset in the application's world coordinate system.
- Data Type:
CoordinateSystem(position + rotation matrix) - Sampling Pattern: Periodic
- Default Frequency: 45 Hz
Eyes (EyesStream) captures eye gaze data including gaze ray origin, direction, and calibration status
- Data Type:
Eyes(contains gaze information per eye, combined gaze, and calibration status) - Sampling Pattern: Periodic
- Default Frequency: 30 Hz
- Requirements: Eye calibration must be completed on the device
Hands (HandsStream) captures articulated hand tracking data for both hands, including joint positions and orientations for all 26 hand joints per hand
- Data Type:
Hand[](array of hand data with joint poses) - Sampling Pattern: Periodic
- Default Frequency: 30 Hz
Audio (AudioStream) captures microphone audio data from the HoloLens microphone array
- Data Type:
AudioBuffer(16-bit PCM audio at 48 kHz) - Sampling Pattern: Event-driven (continuous buffering)
- Default Frame Size: 16,384 bytes
Video (VideoStream) captures H.264-encoded video frames from the HoloLens RGB camera
- Data Type:
EncodedImageCameraView(encoded image with camera intrinsics and extrinsics) - Resolution: Configurable (typically 1280x720 or 1920x1080)
- Sampling Pattern: Event-driven
The HoloLens 2 Research Mode provides access to two depth sensing cameras. Long Throw Depth (LongThrowDepthStream) is long-range depth camera optimized for environmental mapping (range: ~0.5m to ~3m). AHAT Depth (AhatDepthStream) is the Articulated Hand Tracking depth camera optimized for close-range interaction (range: ~0.2m to ~1m).
- Data Type:
DepthImageCameraView(depth image with camera intrinsics and extrinsics) - Resolution:
- Long Throw: 320x288 pixels
- AHAT: 512x512 pixels
- Sampling Pattern: Event-driven
- Requirements: Depth camera calibration files must be present (automatically handled during pipeline setup)
All periodic streams allow configuration of their publishing interval via the Unity Inspector. Lower intervals result in higher data rates but increased computational overhead and network bandwidth usage. The default frequencies are chosen to balance data quality with system performance.
Before connecting to the HoloLens Capture Server, you will need to provide the server's IP address to connect to. We provide two methods below to configure this. If you are testing the Unity application via holographic remoting and running the Capture Server on the same machine, this will typically be 127.0.0.1 (i.e. loopback). Otherwise, this can be found in your network settings or via the ipconfig/ifconfig command in your terminal depending on the capture server's host operating system (OS).
- Option 1 - Static Configuration: The Capture Server's address can be manually specified via the inspector window for the
PsiCaptureController. - Option 2 - Dynamic Configuration: To maintain feature parity with \psi's HoloLens Capture App, we also provide the ability to load an endpoint via a txt file in the User's Documents folder. Please refer to the HoloLens Capture App's configuration steps for details on how to set this up.
Please note, dynamically loading the endpoint from a file:
- is only supported on builds, not via holographic remoting in the Unity Editor, due to its usage of WinRT APIs.
- is only performed if a static configuration is not already provided in the built application.
To start and end a data capture session, simply call the Connect() and Disconnect() methods of the PsiCaptureController.
If you use psiUnity in your research, please consider citing "psiUnity: A Platform for Multimodal Data-Driven XR":
@misc{ajikumar2025psiunity,
title={psiUnity: A Platform for Multimodal Data-Driven XR},
author={Akhil Ajikumar and Sahil Mayenkar and Steven Yoo and Sakib Reza and Mohsen Moghaddam},
year={2025},
eprint={2511.05304},
archivePrefix={arXiv},
primaryClass={cs.HC},
url={https://arxiv.org/abs/2511.05304}
}