XR Data Capture Pipeline

Capture

Learning Outcomes

  • Explain a multimodal XR data pipeline. Distinguish continuous sensor streams from discrete interaction events and understand why timestamps and coordinate frames matter.
  • Record headset and controller data. Capture HMD/controller pose and common controller inputs into a timestamped CSV file.
  • Record hand tracking. Capture tracked hand joints from Unity XR Hands into a synchronized CSV file.
  • Record XRI interactions. Log hover, select, activate, and UI events using the XR Interaction Toolkit.
  • Capture egocentric RGB. Save timestamped Quest 3/3S passthrough-camera frames together with camera pose and intrinsics.
  • Test quickly through Meta Horizon Link. Run the pipeline from the Unity Editor for fast iteration and immediate access to PC-side data files.
  • Inspect and visualize a session. Open the CSV/JPEG data directly and optionally build a simple Streamlit dashboard.

Why Capture XR Data?

An XR headset is not only a display. It is also a multimodal sensing platform. For engineering applications, these data streams can support human-factors analysis, task-performance evaluation, robotics studies, training assessment, digital twins, interaction research, and computer vision.

Data Types and Synchronization

During a single VR or AR session, the headset can provide information about:

  • Head movement and orientation
  • Controller position, orientation, triggers, grips, and thumbsticks
  • Hand-joint tracking
  • Interaction events such as hover, grab, release, and activation
  • Egocentric RGB camera frames
  • Application state and task events

The important idea is not simply to save data. The goal is to create a synchronized session that can be analyzed later.

Quest sensors + XRI events
          |
          v
    common session clock
          |
          v
  -------------------------
  | poses.csv             |
  | hands.csv             |
  | events.csv            |
  | camera.csv            |
  | camera/*.jpg          |
  | session.json          |
  -------------------------
          |
          v
   analysis / dashboard

Key principle: continuous signals and discrete events should not be mixed into one giant table. Store each stream separately, but give every stream a common session time.

What We Will Build

Each run creates one session folder:

XRDataSessions/
└── QuestSession_20261029_142305/
    ├── session.json
    ├── poses.csv
    ├── hands.csv
    ├── events.csv
    ├── camera.csv
    └── camera/
        ├── frame_000000.jpg
        ├── frame_000001.jpg
        └── ...

The tutorial uses conservative teaching defaults:

Stream Default rate Storage
HMD + controllers 30 Hz poses.csv
Hand joints 15 Hz hands.csv
XRI / UI events event-driven events.csv
RGB camera 2 FPS JPEG + camera.csv

These rates are intentionally modest so that the logger does not become the main workload on the Quest.

Prerequisites

This tutorial builds on a Unity project you already created in this course. You need:

  • Meta Quest 3 or Quest 3S
  • The course VR project from Module D or MR project from Module E (see Prepare the Project)
  • Meta XR Core SDK and Meta Mixed Reality Utility Kit (MRUK), added in Step 1
  • Developer mode enabled on the headset
  • A Windows PC with Meta Horizon Link and a Link cable or Air Link

For current package and device requirements, review:

Camera requirement: raw passthrough-camera access is a Meta-specific feature for Quest 3 / 3S. It is different from recording or casting the composited headset view (what the user sees, including application UI). The camera image is an egocentric forward-facing RGB sensor image, not a screenshot of exactly what the user sees. Research / privacy note: RGB images and body/hand tracking can contain sensitive information. Use anonymized session IDs, avoid recording bystanders, obtain appropriate consent, and follow institutional human-subjects requirements before using this pipeline for research data collection.

Development Workflow

In class, the whole pipeline runs through Meta Horizon Link from Unity Play Mode. The recommended sequence is:

  1. Open your existing course VR or MR project.
  2. Add the data-capture scripts.
  3. Run and test through Meta Horizon Link / Unity Play Mode.
  4. Inspect the generated CSV/JPEG data on the PC.
  5. Build and run the simple Streamlit dashboard.

You do not need to build an APK to the Quest during this class. The same logger can also run as a standalone Quest app; that workflow is covered in Standalone Data Collection.

Mode Where the Unity app runs Where the data is saved Used in
Meta Horizon Link Windows PC / Unity Editor PC Application.persistentDataPath This class
Standalone Quest build Quest headset Quest Android app storage Further Study

Rebuilding and redeploying an Android APK after every small code change is slow. Link lets you press Play in Unity and test the headset directly while the application runs on the PC, and the session files land directly on the PC. Note that a Link session and a standalone Quest session do not write to the same location. Application.persistentDataPath belongs to whichever device is actually running the application.

The Link workflow assumes a Windows PC with the Meta Horizon Link application. If Link is unavailable on your computer, pair with a classmate on a Windows PC, or use the standalone workflow in Standalone Data Collection instead.

In the Meta Horizon Link PC application:

  1. Connect the Quest with Link or Air Link.
  2. Under Settings > General, select Set Meta Horizon Link as active next to OpenXR Runtime.
  3. Under Settings > Developer, enable Developer Runtime Features.
  4. In the same panel, enable Passthrough over Meta Horizon Link.
  5. Enable Passthrough Camera API permissions (Link app v2.1 or later). These options appear only after Developer Runtime Features is on; accept the consent dialog for each.
  6. Restart Unity after changing these runtime settings.

In Unity, verify OpenXR is enabled on the Desktop (Windows, Mac, Linux) platform tab used by Play Mode.

Passthrough and camera behavior over Link are for development. Performance and behavior can differ in a standalone Quest build, so validate on the headset itself before using the pipeline for real data collection (see Standalone Data Collection).

Create the Logger

Set up the project, then add the session logger, hand-joint logger, interaction loggers, and RGB camera recorder.

Prepare the Project

Do not configure a new XR project from scratch. Open a Unity project you already built from one of the course templates:

These projects already provide XR Interaction Toolkit, OpenXR, XR Hands, the XR Origin, input actions, and the OpenXR platform settings. If you need to create or reconfigure either project, refer back to Modules D and E. Keep the template’s existing XR Origin; this tutorial adds data capture around it.

Add Camera-Access Packages

The course templates do not include Meta’s camera-access components.

  1. Find Meta XR SDK on Unity Asset Store and add it to you project.

    02

  2. Go to Window > Package Manager.
  3. Add Meta XR All-in-One SDK — provides OVRManager, OVRPermissionsRequester, and PassthroughCameraAccess. 02

Use package versions compatible with the course Unity version. Do not downgrade a working course project merely to match an older screenshot.

Verify Hand Tracking

XRHandJointLogger needs the OpenXR Hand Tracking Subsystem. In Edit > Project Settings > XR Plug-in Management > OpenXR, open the Desktop tab used for Link Play Mode and confirm Hand Tracking Subsystem is enabled. On the headset, enable hand tracking under the Quest movement-tracking settings.

03

We use Unity XR Hands for the logging code because it exposes a standard hand-joint model while the project continues to use XRI.

Create the Capture Manager

  1. Create an empty GameObject named: XRDataCapture.
  2. Create a folder: Assets/Scripts/DataCapture/.
  3. Add the following scripts to that folder:
    • XRSessionLogger.cs
    • XRHandJointLogger.cs
    • XRInteractionEventLogger.cs
    • XRUIEventLogger.cs
    • QuestCameraPermission.cs
    • PassthroughCameraRecorder.cs
    • XRDataDashboard.cs (optional)

    Each script is listed in full below. You can also download them directly: XRSessionLogger.cs, XRHandJointLogger.cs, XRInteractionEventLogger.cs, XRUIEventLogger.cs, QuestCameraPermission.cs, PassthroughCameraRecorder.cs, and XRDataDashboard.cs.

  4. Attach XRSessionLogger and XRHandJointLogger to XRDataCapture.

04

XR Session Logger

This is the central session clock and file writer. It records headset/controller data and provides shared methods for the other loggers.

using System;
using System.Globalization;
using System.IO;
using UnityEngine;
using UnityEngine.XR;

public class XRSessionLogger : MonoBehaviour
{
    public static XRSessionLogger Instance { get; private set; }

    [Header("Session")]
    [SerializeField] private string sessionLabel = "QuestSession";

    [Header("Tracked Transforms")]
    [SerializeField] private Transform hmd;
    [SerializeField] private Transform leftController;
    [SerializeField] private Transform rightController;

    [Header("Sampling")]
    [SerializeField, Range(1f, 120f)] private float poseSampleRateHz = 30f;

    public string SessionDirectory { get; private set; }
    public double ElapsedSeconds => Time.realtimeSinceStartupAsDouble - startRealtime;

    public int PoseSamples { get; private set; }
    public int EventCount { get; private set; }
    public int HandRows { get; private set; }
    public int CameraFrames { get; private set; }

    private StreamWriter poseWriter;
    private StreamWriter eventWriter;
    private StreamWriter handWriter;
    private StreamWriter cameraWriter;

    private double startRealtime;
    private double nextPoseTime;
    private double nextFlushTime;

    private InputDevice headDevice;
    private InputDevice leftDevice;
    private InputDevice rightDevice;

    private static readonly CultureInfo Inv = CultureInfo.InvariantCulture;

    [Serializable]
    private class SessionMetadata
    {
        public string sessionLabel;
        public string startedUtc;
        public string unityVersion;
        public string deviceModel;
        public string operatingSystem;
        public string applicationVersion;
        public string packageName;
        public string coordinateFrame;
        public float poseSampleRateHz;
    }

    private void Awake()
    {
        if (Instance != null && Instance != this)
        {
            Destroy(gameObject);
            return;
        }

        Instance = this;
    }

    private void Start()
    {
        BeginSession();
    }

    private void Update()
    {
        if (string.IsNullOrEmpty(SessionDirectory))
            return;

        double now = Time.realtimeSinceStartupAsDouble;

        if (now >= nextPoseTime)
        {
            SamplePosesAndControllerInputs();
            nextPoseTime = now + (1.0 / poseSampleRateHz);
        }

        if (now >= nextFlushTime)
        {
            FlushAll();
            nextFlushTime = now + 1.0;
        }
    }

    private void BeginSession()
    {
        startRealtime = Time.realtimeSinceStartupAsDouble;
        nextPoseTime = startRealtime;
        nextFlushTime = startRealtime + 1.0;

        string safeLabel = MakeSafeFilename(sessionLabel);
        string folderName = $"{safeLabel}_{DateTime.Now:yyyyMMdd_HHmmss}";

        SessionDirectory = Path.Combine(
            Application.persistentDataPath,
            "XRDataSessions",
            folderName);

        Directory.CreateDirectory(SessionDirectory);
        Directory.CreateDirectory(Path.Combine(SessionDirectory, "camera"));

        poseWriter = CreateWriter("poses.csv",
            "time_utc,elapsed_s," +
            "hmd_tracked,hmd_px,hmd_py,hmd_pz,hmd_qx,hmd_qy,hmd_qz,hmd_qw," +
            "left_tracked,left_px,left_py,left_pz,left_qx,left_qy,left_qz,left_qw," +
            "left_trigger,left_grip,left_axis_x,left_axis_y,left_primary,left_secondary," +
            "right_tracked,right_px,right_py,right_pz,right_qx,right_qy,right_qz,right_qw," +
            "right_trigger,right_grip,right_axis_x,right_axis_y,right_primary,right_secondary");

        eventWriter = CreateWriter("events.csv",
            "time_utc,elapsed_s,event_type,target,interactor,details");

        handWriter = CreateWriter("hands.csv",
            "time_utc,elapsed_s,hand,joint,hand_tracked,joint_valid,px,py,pz,qx,qy,qz,qw");

        cameraWriter = CreateWriter("camera.csv",
            "time_utc,elapsed_s,camera_timestamp_utc,filename," +
            "px,py,pz,qx,qy,qz,qw,width,height," +
            "focal_x,focal_y,principal_x,principal_y");

        var metadata = new SessionMetadata
        {
            sessionLabel = sessionLabel,
            startedUtc = DateTime.UtcNow.ToString("O"),
            unityVersion = Application.unityVersion,
            deviceModel = SystemInfo.deviceModel,
            operatingSystem = SystemInfo.operatingSystem,
            applicationVersion = Application.version,
            packageName = Application.identifier,
            coordinateFrame = "Unity world coordinates; meters; left-handed; Y-up",
            poseSampleRateHz = poseSampleRateHz
        };

        File.WriteAllText(
            Path.Combine(SessionDirectory, "session.json"),
            JsonUtility.ToJson(metadata, true));

        Debug.Log($"XR data session started: {SessionDirectory}");
    }

    private StreamWriter CreateWriter(string filename, string header)
    {
        var writer = new StreamWriter(Path.Combine(SessionDirectory, filename), false);
        writer.WriteLine(header);
        writer.AutoFlush = false;
        return writer;
    }

    private void SamplePosesAndControllerInputs()
    {
        RefreshDevicesIfNeeded();

        bool headTracked = IsTracked(headDevice);
        bool leftTracked = IsTracked(leftDevice);
        bool rightTracked = IsTracked(rightDevice);

        Vector3 hmdPos = hmd != null ? hmd.position : Vector3.zero;
        Quaternion hmdRot = hmd != null ? hmd.rotation : Quaternion.identity;

        Vector3 leftPos = leftController != null ? leftController.position : Vector3.zero;
        Quaternion leftRot = leftController != null ? leftController.rotation : Quaternion.identity;

        Vector3 rightPos = rightController != null ? rightController.position : Vector3.zero;
        Quaternion rightRot = rightController != null ? rightController.rotation : Quaternion.identity;

        ReadController(leftDevice,
            out float lTrigger, out float lGrip, out Vector2 lAxis,
            out bool lPrimary, out bool lSecondary);

        ReadController(rightDevice,
            out float rTrigger, out float rGrip, out Vector2 rAxis,
            out bool rPrimary, out bool rSecondary);

        poseWriter.WriteLine(string.Join(",",
            UtcNow(), F(ElapsedSeconds),
            B(headTracked), F(hmdPos.x), F(hmdPos.y), F(hmdPos.z),
            F(hmdRot.x), F(hmdRot.y), F(hmdRot.z), F(hmdRot.w),
            B(leftTracked), F(leftPos.x), F(leftPos.y), F(leftPos.z),
            F(leftRot.x), F(leftRot.y), F(leftRot.z), F(leftRot.w),
            F(lTrigger), F(lGrip), F(lAxis.x), F(lAxis.y), B(lPrimary), B(lSecondary),
            B(rightTracked), F(rightPos.x), F(rightPos.y), F(rightPos.z),
            F(rightRot.x), F(rightRot.y), F(rightRot.z), F(rightRot.w),
            F(rTrigger), F(rGrip), F(rAxis.x), F(rAxis.y), B(rPrimary), B(rSecondary)));

        PoseSamples++;
    }

    private void RefreshDevicesIfNeeded()
    {
        if (!headDevice.isValid)
            headDevice = InputDevices.GetDeviceAtXRNode(XRNode.Head);
        if (!leftDevice.isValid)
            leftDevice = InputDevices.GetDeviceAtXRNode(XRNode.LeftHand);
        if (!rightDevice.isValid)
            rightDevice = InputDevices.GetDeviceAtXRNode(XRNode.RightHand);
    }

    private static bool IsTracked(InputDevice device)
    {
        if (!device.isValid)
            return false;
        if (device.TryGetFeatureValue(CommonUsages.isTracked, out bool tracked))
            return tracked;
        return true;
    }

    private static void ReadController(InputDevice device, out float trigger, out float grip,
        out Vector2 axis, out bool primary, out bool secondary)
    {
        trigger = 0f;
        grip = 0f;
        axis = Vector2.zero;
        primary = false;
        secondary = false;

        if (!device.isValid)
            return;

        device.TryGetFeatureValue(CommonUsages.trigger, out trigger);
        device.TryGetFeatureValue(CommonUsages.grip, out grip);
        device.TryGetFeatureValue(CommonUsages.primary2DAxis, out axis);
        device.TryGetFeatureValue(CommonUsages.primaryButton, out primary);
        device.TryGetFeatureValue(CommonUsages.secondaryButton, out secondary);
    }

    public void LogEvent(string eventType, string target, string interactor = "", string details = "")
    {
        if (eventWriter == null)
            return;

        eventWriter.WriteLine(string.Join(",", UtcNow(), F(ElapsedSeconds),
            Csv(eventType), Csv(target), Csv(interactor), Csv(details)));
        EventCount++;
    }

    public void LogHandJoint(string hand, string joint, bool handTracked, bool jointValid,
        Vector3 position, Quaternion rotation)
    {
        if (handWriter == null)
            return;

        handWriter.WriteLine(string.Join(",", UtcNow(), F(ElapsedSeconds), Csv(hand), Csv(joint),
            B(handTracked), B(jointValid), F(position.x), F(position.y), F(position.z),
            F(rotation.x), F(rotation.y), F(rotation.z), F(rotation.w)));
        HandRows++;
    }

    public void LogCameraFrame(DateTime cameraTimestamp, string filename, Pose cameraPose,
        Vector2Int resolution, Vector2 focalLength, Vector2 principalPoint)
    {
        if (cameraWriter == null)
            return;

        cameraWriter.WriteLine(string.Join(",", UtcNow(), F(ElapsedSeconds),
            Csv(cameraTimestamp.ToUniversalTime().ToString("O")), Csv(filename),
            F(cameraPose.position.x), F(cameraPose.position.y), F(cameraPose.position.z),
            F(cameraPose.rotation.x), F(cameraPose.rotation.y), F(cameraPose.rotation.z),
            F(cameraPose.rotation.w), resolution.x.ToString(Inv), resolution.y.ToString(Inv),
            F(focalLength.x), F(focalLength.y), F(principalPoint.x), F(principalPoint.y)));
        CameraFrames++;
    }

    private void OnApplicationPause(bool pause)
    {
        if (pause) FlushAll();
    }

    private void OnApplicationQuit() => CloseAll();

    private void OnDestroy()
    {
        if (Instance == this)
        {
            CloseAll();
            Instance = null;
        }
    }

    private void FlushAll()
    {
        poseWriter?.Flush();
        eventWriter?.Flush();
        handWriter?.Flush();
        cameraWriter?.Flush();
    }

    private void CloseAll()
    {
        FlushAll();
        poseWriter?.Close();
        eventWriter?.Close();
        handWriter?.Close();
        cameraWriter?.Close();
        poseWriter = null;
        eventWriter = null;
        handWriter = null;
        cameraWriter = null;
    }

    private static string UtcNow() => DateTime.UtcNow.ToString("O");
    private static string F(double value) => value.ToString("0.######", Inv);
    private static string F(float value) => value.ToString("0.######", Inv);
    private static string B(bool value) => value ? "1" : "0";

    private static string Csv(string value)
    {
        if (string.IsNullOrEmpty(value)) return "";
        return "\"" + value.Replace("\"", "\"\"") + "\"";
    }

    private static string MakeSafeFilename(string value)
    {
        if (string.IsNullOrWhiteSpace(value)) return "Session";
        foreach (char c in Path.GetInvalidFileNameChars()) value = value.Replace(c, '_');
        return value.Replace(' ', '_');
    }
}

Configure the XR Session Logger

In the Inspector:

  1. Set Session Label to an anonymous identifier such as P01 or Demo01.
  2. Assign the XR camera / HMD Transform.
  3. Assign the left controller Transform.
  4. Assign the right controller Transform.
  5. Keep Pose Sample Rate Hz = 30 for the first test.

04

The logger records Unity world-space poses. If your application moves the XR Origin through locomotion, the logged poses include that movement. This is usually desirable for task-space analysis.

Log Hand Tracking

  1. Attach XRHandJointLogger to XRDataCapture.
  2. Assign the root Transform of your XR Origin to its XR Origin field.

04

XR Hand Joint Logger

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.Hands;

public class XRHandJointLogger : MonoBehaviour
{
    [SerializeField] private Transform xrOrigin;
    [SerializeField, Range(1f, 60f)] private float handSampleRateHz = 15f;

    private XRHandSubsystem handSubsystem;
    private double nextSampleTime;
    private readonly List<XRHandSubsystem> handSubsystems = new();

    private void Update()
    {
        if (XRSessionLogger.Instance == null)
            return;

        if (handSubsystem == null || !handSubsystem.running)
            FindRunningHandSubsystem();

        if (handSubsystem == null || !handSubsystem.running)
            return;

        double now = Time.realtimeSinceStartupAsDouble;
        if (now < nextSampleTime)
            return;

        nextSampleTime = now + (1.0 / handSampleRateHz);
        LogHand("left", handSubsystem.leftHand);
        LogHand("right", handSubsystem.rightHand);
    }

    private void FindRunningHandSubsystem()
    {
        handSubsystems.Clear();
        SubsystemManager.GetSubsystems(handSubsystems);
        handSubsystem = null;
        foreach (var subsystem in handSubsystems)
        {
            if (subsystem != null && subsystem.running)
            {
                handSubsystem = subsystem;
                break;
            }
        }
    }

    private void LogHand(string label, XRHand hand)
    {
        bool handTracked = hand.isTracked;

        for (int i = XRHandJointID.BeginMarker.ToIndex();
             i < XRHandJointID.EndMarker.ToIndex(); i++)
        {
            XRHandJointID jointId = XRHandJointIDUtility.FromIndex(i);
            XRHandJoint joint = hand.GetJoint(jointId);
            bool valid = joint.TryGetPose(out Pose jointPose);

            Vector3 worldPosition = Vector3.zero;
            Quaternion worldRotation = Quaternion.identity;

            if (valid)
            {
                if (xrOrigin != null)
                {
                    worldPosition = xrOrigin.TransformPoint(jointPose.position);
                    worldRotation = xrOrigin.rotation * jointPose.rotation;
                }
                else
                {
                    worldPosition = jointPose.position;
                    worldRotation = jointPose.rotation;
                }
            }

            XRSessionLogger.Instance.LogHandJoint(label, jointId.ToString(), handTracked,
                valid, worldPosition, worldRotation);
        }
    }
}

Data Format

The logger writes one row per joint per sample. Each row includes:

  • Session time
  • Left / right hand
  • Joint name
  • Hand tracking state
  • Joint validity
  • Position
  • Quaternion rotation

Hand tracking is not valid for every joint on every frame. Do not silently replace missing tracking with a plausible pose. The joint_valid column tells your later analysis which samples are actually available.

Log XRI Interaction Events

Continuous pose data tells us where the user moved. Interaction events tell us what the user did. Logging XRI hover, select, and activate events marks the exact moments a user approached, grabbed, used, or released an object, so you can measure task time, count errors and retries, and see which objects or UI controls caused hesitation. Because each event shares the session clock, it also serves as an index into the pose, hand, and RGB streams, letting you jump straight to what the user’s head, hands, and view were doing at that moment.

XR Interaction Event Logger

For each important XR Grab Interactable, XR Simple Interactable, socket, or other XRI interactable that you want to instrument, add: XRInteractionEventLogger.

05

using UnityEngine;
using UnityEngine.XR.Interaction.Toolkit;
using UnityEngine.XR.Interaction.Toolkit.Interactables;

public class XRInteractionEventLogger : MonoBehaviour
{
    private XRBaseInteractable interactable;

    private void Awake() => interactable = GetComponent<XRBaseInteractable>();

    private void OnEnable()
    {
        if (interactable == null)
        {
            Debug.LogError(
                $"XRInteractionEventLogger on '{gameObject.name}' needs an XRI interactable " +
                "(for example, XR Grab Interactable or XR Simple Interactable) on the same GameObject. " +
                "Disabling this logger.", this);
            enabled = false;
            return;
        }

        interactable.hoverEntered.AddListener(OnHoverEntered);
        interactable.hoverExited.AddListener(OnHoverExited);
        interactable.selectEntered.AddListener(OnSelectEntered);
        interactable.selectExited.AddListener(OnSelectExited);
        interactable.activated.AddListener(OnActivated);
        interactable.deactivated.AddListener(OnDeactivated);
    }

    private void OnDisable()
    {
        if (interactable == null)
            return;

        interactable.hoverEntered.RemoveListener(OnHoverEntered);
        interactable.hoverExited.RemoveListener(OnHoverExited);
        interactable.selectEntered.RemoveListener(OnSelectEntered);
        interactable.selectExited.RemoveListener(OnSelectExited);
        interactable.activated.RemoveListener(OnActivated);
        interactable.deactivated.RemoveListener(OnDeactivated);
    }

    private void OnHoverEntered(HoverEnterEventArgs args) => Log("hover_enter", NameOf(args.interactorObject));
    private void OnHoverExited(HoverExitEventArgs args) => Log("hover_exit", NameOf(args.interactorObject));
    private void OnSelectEntered(SelectEnterEventArgs args) => Log("select_enter", NameOf(args.interactorObject));
    private void OnSelectExited(SelectExitEventArgs args) => Log("select_exit", NameOf(args.interactorObject));
    private void OnActivated(ActivateEventArgs args) => Log("activate", NameOf(args.interactorObject));
    private void OnDeactivated(DeactivateEventArgs args) => Log("deactivate", NameOf(args.interactorObject));

    private void Log(string eventType, string interactorName)
    {
        XRSessionLogger.Instance?.LogEvent(eventType, gameObject.name, interactorName);
    }

    private static string NameOf(object value)
    {
        if (value is Component component) return component.gameObject.name;
        return value != null ? value.ToString() : "unknown";
    }
}

The logger must share a GameObject with a concrete XRI interactable such as XR Grab Interactable or XR Simple Interactable. If none is present, it reports an error in the Console and disables itself instead of throwing exceptions.

Data Format

The script records:

  • hover_enter
  • hover_exit
  • select_enter
  • select_exit
  • activate
  • deactivate

The event is written with the target object’s name and the interactor name.

Log Unity UI Events

Unity UI controls such as Button, Toggle, and Slider use Unity’s UI event system rather than XRI interactable events. The XRInteractionEventLogger therefore does not capture these interactions automatically. To log UI activity, attach one XRUIEventLogger to the root Canvas. At runtime, the script automatically finds all Button, Toggle, and Slider components under that Canvas and subscribes to their events.

XR UI Event Logger

using UnityEngine;
using UnityEngine.UI;

public class XRUIEventLogger : MonoBehaviour
{
    private Button[] buttons;
    private Toggle[] toggles;
    private Slider[] sliders;

    private void Start()
    {
        buttons = GetComponentsInChildren<Button>(true);
        toggles = GetComponentsInChildren<Toggle>(true);
        sliders = GetComponentsInChildren<Slider>(true);

        foreach (Button button in buttons)
        {
            Button capturedButton = button;

            button.onClick.AddListener(() =>
                LogButton(capturedButton.gameObject.name));
        }

        foreach (Toggle toggle in toggles)
        {
            Toggle capturedToggle = toggle;

            toggle.onValueChanged.AddListener(value =>
                LogToggle(capturedToggle.gameObject.name, value));
        }

        foreach (Slider slider in sliders)
        {
            Slider capturedSlider = slider;

            slider.onValueChanged.AddListener(value =>
                LogSlider(capturedSlider.gameObject.name, value));
        }

        Debug.Log(
            $"XRUIEventLogger: monitoring {buttons.Length} buttons, " +
            $"{toggles.Length} toggles, and {sliders.Length} sliders.");
    }

    private void LogButton(string elementName)
    {
        XRSessionLogger.Instance?.LogEvent(
            "ui_click",
            elementName,
            "ui");
    }

    private void LogToggle(string elementName, bool value)
    {
        XRSessionLogger.Instance?.LogEvent(
            "ui_toggle",
            elementName,
            "ui",
            $"value={value}");
    }

    private void LogSlider(string elementName, float value)
    {
        XRSessionLogger.Instance?.LogEvent(
            "ui_slider",
            elementName,
            "ui",
            $"value={value:0.###}");
    }
}

Attach XRUIEventLogger to the root Canvas containing the UI controls you want to monitor.

05

Give UI GameObjects clear names such as StartButton, SafetyToggle, or SpeedSlider. These names are written directly into the event log and make the resulting data easier to interpret.

Data Format

UI interactions are written to the same events.csv file as the XRI interaction events. The logger automatically captures all Button, Toggle, and Slider components that exist under the Canvas when the scene starts.

If UI controls are created dynamically later at runtime, they will not be discovered automatically by this simple version of the logger.

Log RGB Camera Frames

Quest 3 exposes forward-facing RGB cameras through Meta’s Passthrough Camera API. The current Unity workflow uses MRUK’s: PassthroughCameraAccess. This component provides:

  • RGB texture
  • Camera timestamp
  • Camera pose
  • Resolution
  • Camera intrinsics
  • 2D-to-3D projection utilities

Add Meta XR Services

The course scenes already use Unity’s XR Origin and XRI. Do not add a second active XR camera rig.

  1. Create an empty GameObject named: MetaXRServices.
  2. Add one OVRManager component if the scene does not already contain one.
  3. Use: Window > Meta > Tools > Project Setup Tool and fix the required Meta Quest / Android configuration issues.
  4. In OVRManager, enable the Quest features required for passthrough and Passthrough Camera Access. Leave the camera entry under OVRManager > Permission Requests On Startup unchecked; this tutorial requests the camera permission from QuestCameraPermission instead (see Camera Permission).

Meta’s XR SDK components can be used with Unity OpenXR. OVRManager should exist only once in the scene. Visible passthrough vs. raw RGB: the Passthrough Camera API requires passthrough to be enabled. If you also want the physical world to be visible in the application, add/configure the project’s normal Meta passthrough layer/building block (for example, an OVRPassthroughLayer underlay) while keeping the existing course XR Origin. Do not add a second active camera rig.

06

Add Passthrough Camera Access

  1. Create another child under MetaXRServices named: RGBDataCapture.
  2. Add:
    • PassthroughCameraAccess
    • QuestCameraPermission
    • PassthroughCameraRecorder
  3. Configure PassthroughCameraAccess:
    • Camera Position: Left
    • Requested Resolution: 640 x 480
    • Leave the live camera stream at its normal supported frame rate

We will save only 2 frames per second to disk.

07

Camera Permission

For the Meta Horizon Link workflow, camera access is controlled by the Meta Horizon Link application on the PC.

Before opening Play Mode:

  1. Open Meta Horizon Link > Settings > Developer.
  2. Enable Developer Runtime Features.
  3. Enable Passthrough over Meta Horizon Link.
  4. Enable Passthrough Camera API permissions.
  5. Accept the consent prompts and restart Unity.

07

A standalone Quest application requires the camera permission:

horizonos.permission.HEADSET_CAMERA

The permission must be declared in the Android application and requested from the user at runtime. This is only required when deploying the application directly to the headset and is covered in the Standalone Data Collection section later in this tutorial.

Record RGB Frames + Camera Metadata

  1. Attach PassthroughCameraRecorder to RGBDataCapture.
  2. Assign the PassthroughCameraAccess component to its Camera Access field.
  3. Keep:
    • Saved Frames Per Second = 2
    • JPEG Quality = 75

07

using System.Collections;
using System.IO;
using Meta.XR;
using UnityEngine;
using UnityEngine.UI;

public class PassthroughCameraRecorder : MonoBehaviour
{
    [Header("Camera")]
    [SerializeField] private PassthroughCameraAccess cameraAccess;
    [SerializeField] private RawImage optionalPreview;

    [Header("Recording")]
    [SerializeField, Range(0.2f, 10f)] private float savedFramesPerSecond = 2f;
    [SerializeField, Range(1, 100)] private int jpegQuality = 75;

    private Texture2D snapshotTexture;
    private Vector2Int snapshotResolution;
    private string cameraDirectory;
    private double nextCaptureTime;
    private int frameIndex;

    private IEnumerator Start()
    {
        while (XRSessionLogger.Instance == null ||
               string.IsNullOrEmpty(XRSessionLogger.Instance.SessionDirectory))
            yield return null;

        cameraDirectory = Path.Combine(XRSessionLogger.Instance.SessionDirectory, "camera");
        Directory.CreateDirectory(cameraDirectory);

        if (cameraAccess == null)
        {
            Debug.LogError("PassthroughCameraRecorder: cameraAccess is not assigned.");
            yield break;
        }

        while (!cameraAccess.IsPlaying)
            yield return null;

        if (optionalPreview != null)
            optionalPreview.texture = cameraAccess.GetTexture();

        nextCaptureTime = Time.realtimeSinceStartupAsDouble;
        Debug.Log("Passthrough camera recorder is ready.");
    }

    private void Update()
    {
        if (cameraAccess == null || !cameraAccess.IsPlaying ||
            XRSessionLogger.Instance == null || string.IsNullOrEmpty(cameraDirectory))
            return;

        if (!cameraAccess.IsUpdatedThisFrame)
            return;

        double now = Time.realtimeSinceStartupAsDouble;
        if (now < nextCaptureTime)
            return;

        nextCaptureTime = now + (1.0 / savedFramesPerSecond);
        CaptureFrame();
    }

    private void CaptureFrame()
    {
        Vector2Int resolution = cameraAccess.CurrentResolution;
        if (resolution.x <= 0 || resolution.y <= 0)
            return;

        EnsureSnapshotTexture(resolution);

        var pixels = cameraAccess.GetColors();
        if (!pixels.IsCreated || pixels.Length == 0)
            return;

        snapshotTexture.SetPixelData(pixels, 0);
        snapshotTexture.Apply(false, false);

        byte[] jpg = snapshotTexture.EncodeToJPG(jpegQuality);
        string filename = $"frame_{frameIndex:D6}.jpg";
        string fullPath = Path.Combine(cameraDirectory, filename);
        File.WriteAllBytes(fullPath, jpg);

        Pose cameraPose = cameraAccess.GetCameraPose();
        var intrinsics = cameraAccess.Intrinsics;

        XRSessionLogger.Instance.LogCameraFrame(
            cameraAccess.Timestamp,
            Path.Combine("camera", filename).Replace("\\", "/"),
            cameraPose,
            resolution,
            intrinsics.FocalLength,
            intrinsics.PrincipalPoint);

        frameIndex++;
    }

    private void EnsureSnapshotTexture(Vector2Int resolution)
    {
        if (snapshotTexture != null && snapshotResolution == resolution)
            return;

        if (snapshotTexture != null)
            Destroy(snapshotTexture);

        snapshotResolution = resolution;
        snapshotTexture = new Texture2D(
            resolution.x,
            resolution.y,
            TextureFormat.RGBA32,
            false);
    }

    private void OnDestroy()
    {
        if (snapshotTexture != null)
            Destroy(snapshotTexture);
    }
}

For every saved image, camera.csv records:

  • Shared session time
  • Camera sensor timestamp
  • Image filename
  • Camera position and rotation
  • Image resolution
  • Focal length
  • Principal point

This is important because the user’s head is moving. An RGB frame without its capture-time camera pose is much less useful for spatial analysis.

GetColors() performs a CPU readback and can be expensive. At the teaching rate of 2 FPS and 640×480 it is intentionally conservative. For high-rate production capture, use an asynchronous readback or video-oriented pipeline instead of simply increasing this number.

Optional: Display the Live RGB Feed

  1. Create a small World Space Canvas with a RawImage.
  2. Assign that RawImage to PassthroughCameraRecorder > Optional Preview.
  3. When the camera becomes available, the script displays the current camera texture.

09

Optional: Add a Live Data Dashboard

Create a small World Space Canvas with one TextMeshPro text element. Then attach XRDataDashboard to the Canvas and assign the TMP text field.

using TMPro;
using UnityEngine;

public class XRDataDashboard : MonoBehaviour
{
    [SerializeField] private TMP_Text statusText;
    private float nextUpdate;

    private void Update()
    {
        if (statusText == null || XRSessionLogger.Instance == null || Time.unscaledTime < nextUpdate)
            return;

        nextUpdate = Time.unscaledTime + 0.25f;
        XRSessionLogger logger = XRSessionLogger.Instance;

        statusText.text =
            $"XR Data Capture\n" +
            $"Time: {logger.ElapsedSeconds:0.0} s\n" +
            $"Pose samples: {logger.PoseSamples}\n" +
            $"Interaction events: {logger.EventCount}\n" +
            $"Hand rows: {logger.HandRows}\n" +
            $"RGB frames: {logger.CameraFrames}\n\n" +
            $"Saved to:\n{logger.SessionDirectory}";
    }
}

The dashboard shows session duration, pose samples, interaction events, hand rows, saved RGB frames, and the current session folder.

10

Test the Data Pipeline

Run the logger from Unity Play Mode, inspect the PC-side session files, and open them in a quick dashboard.

This is the main in-class test. If you are not on Windows or cannot use Link, see Standalone Data Collection; the logger code is identical.

  1. Connect the Quest to the Windows PC using Meta Horizon Link or Air Link.
  2. Confirm the Link developer settings described earlier are enabled.
  3. Open the Unity scene.
  4. Press Play in the Unity Editor.
  5. Put on the headset.
  6. Move your head and controllers.
  7. Grab and activate a few XRI objects.
  8. Put the controllers down and briefly test hand tracking.
  9. Look around so the RGB camera saves several frames.
  10. Exit Play Mode.

The Unity Console should print a session path similar to:

XR data session started:
C:/Users/<user>/AppData/LocalLow/<company>/<product>/XRDataSessions/Demo01_...

11

The exact PC path depends on the Unity Player Settings and operating system.

Link behavior: Under Link, the Unity application is running on the PC, so Application.persistentDataPath is a PC path. The CSV and JPEG files are already on the PC. You do not use adb pull for this run.

11

Inspect the Session Files

Open the session directory printed in the Unity Console.

You should see:

session.json
poses.csv
hands.csv
events.csv
camera.csv
camera/

poses.csv

Contains synchronized HMD and controller data, one row per sample (30 Hz by default). Column names start with hmd_, left_, or right_; left_ and right_ refer to the left and right controllers. In the table, * stands for one of these prefixes.

Column Meaning
time_utc Absolute UTC timestamp
elapsed_s Seconds since the logging session started; the main synchronization clock
*_tracked 1 if that HMD/controller was tracked at this sample, 0 otherwise
*_px, *_py, *_pz Position in Unity world coordinates, in meters
*_qx, *_qy, *_qz, *_qw Orientation as a quaternion
*_trigger, *_grip Analog controller values, normally 0 to 1 (controllers only)
*_axis_x, *_axis_y Thumbstick / primary 2D-axis values, normally -1 to 1 (controllers only)
*_primary, *_secondary Button state: 1 pressed, 0 not pressed (controllers only)

hands.csv

Uses a long-table format: one row represents one joint of one hand at one sample time, so every sample produces many rows for each tracked hand. This is easier to filter by joint or hand than creating hundreds of hand columns in one row.

Column Meaning
time_utc Absolute UTC timestamp
elapsed_s Shared session time in seconds
hand left or right
joint Tracked hand joint, such as Wrist, IndexProximal, or IndexTip
hand_tracked 1 when the hand is currently tracked, 0 otherwise
joint_valid 1 when Unity has a valid pose for this joint; 0 means the position/orientation should not be used
px, py, pz Joint position in Unity world coordinates, in meters
qx, qy, qz, qw Joint orientation as a quaternion

events.csv

Contains event-driven interaction data: a row is written only when something happens, not at a fixed sampling rate.

Column Meaning
time_utc Absolute UTC timestamp
elapsed_s Shared session time in seconds
event_type Event such as hover_enter, select_enter, select_exit, activate, ui_click, or ui_toggle
target Name of the GameObject or UI element involved
interactor Controller or hand interactor that caused the event, or ui for UI events
details Optional extra information, such as a toggle state or slider value

camera.csv

Maps each JPEG to its capture metadata, one row per saved frame.

Column Meaning
time_utc Time the logger wrote the record
elapsed_s Shared session time in seconds
camera_timestamp_utc Timestamp of the captured camera frame
filename Corresponding JPEG under the camera/ folder
px, py, pz Camera position in Unity world coordinates, in meters
qx, qy, qz, qw Camera orientation as a quaternion
width, height RGB image resolution in pixels
focal_x, focal_y Camera focal length, in pixels
principal_x, principal_y Image coordinates of the camera’s principal point, in pixels

session.json

Stores session-level metadata: the session label, start time, Unity version, device model and operating system, application version and package name, coordinate-frame convention, and configured pose sampling rate.

Because every stream contains elapsed_s, data from different files can be aligned by session time. For example, a grab event at 12.438 s can be matched to the nearest controller pose, hand sample, and RGB frame.

Optional: Build a Quick Dashboard

A dashboard is useful because it gives immediate feedback that the data pipeline is working. For this course, keep the dashboard simple. The goal is inspection, not a full analytics system.

Download dashboard.py and requirements-dashboard.txt into the same folder on your PC, then install the dependencies:

pip install -r requirements-dashboard.txt

This is equivalent to pip install streamlit pandas.

You can also create dashboard.py manually:

from pathlib import Path
import pandas as pd
import streamlit as st

st.set_page_config(page_title="XR Session Dashboard", layout="wide")
st.title("XR Session Dashboard")

session_input = st.text_input("Session folder", value=".")
session = Path(session_input).expanduser()

if not session.exists():
    st.warning("Choose a valid session folder.")
    st.stop()

poses_path = session / "poses.csv"
events_path = session / "events.csv"
hands_path = session / "hands.csv"
camera_path = session / "camera.csv"

poses = pd.read_csv(poses_path) if poses_path.exists() else pd.DataFrame()
events = pd.read_csv(events_path) if events_path.exists() else pd.DataFrame()
hands = pd.read_csv(hands_path) if hands_path.exists() else pd.DataFrame()
camera = pd.read_csv(camera_path) if camera_path.exists() else pd.DataFrame()

c1, c2, c3, c4 = st.columns(4)
c1.metric("Pose samples", len(poses))
c2.metric("Interaction events", len(events))
c3.metric("Hand rows", len(hands))
c4.metric("RGB frames", len(camera))

if not poses.empty:
    st.subheader("HMD position")
    hmd = poses.set_index("elapsed_s")[["hmd_px", "hmd_py", "hmd_pz"]]
    st.line_chart(hmd)

if not events.empty:
    st.subheader("Interaction events")
    st.bar_chart(events["event_type"].value_counts())
    st.dataframe(events.tail(25))

if not hands.empty:
    st.subheader("Hand tracking")
    tracked = hands.groupby(["hand", "joint_valid"]).size().unstack(fill_value=0)
    st.dataframe(tracked)

camera_dir = session / "camera"
if camera_dir.exists():
    images = sorted(camera_dir.glob("*.jpg"))
    if images:
        st.subheader("Latest RGB frame")
        st.image(str(images[-1]), caption=images[-1].name)

Run it from the same folder:

streamlit run dashboard.py

Paste the session-folder path from the Unity Console into the dashboard. Within a few seconds you should be able to see:

  • total pose samples
  • number of interaction events
  • hand-tracking rows
  • number of RGB frames
  • HMD position over time
  • interaction-event counts
  • the latest RGB image

Key Considerations

At this point, you have captured a synchronized multimodal session through Meta Horizon Link and inspected its pose, hand, event, and RGB data in a simple dashboard. The sections below step back from the individual scripts to cover the design decisions, troubleshooting, and data-collection practices that make such sessions reliable and useful for engineering analysis.

Understand the Data Pipeline

The most important part of this tutorial is not the individual scripts. It is the architecture.

Continuous Signals

Examples:

  • Head pose
  • Controller pose
  • Hand joints
  • Trigger values

These are sampled at a defined frequency.

Discrete Events

Examples:

  • Hover entered
  • Object grabbed
  • Object released
  • Button clicked

These are logged only when they occur.

Camera Frames

RGB images are large, so they are saved separately as JPEG files. The CSV stores only the image’s timestamp, filename, and camera metadata.

Synchronization

All streams contain the same session variable:

elapsed_s

Suppose a grab occurs at:

12.438 s

You can find:

  • the nearest HMD/controller sample
  • the nearest hand-joint sample
  • the nearest RGB frame

This lets you reconstruct what was happening around the interaction.

For spatial computer-vision work, also use the camera’s own timestamp and GetCameraPose() recorded in camera.csv.

Coordinate Frames

This tutorial stores poses in Unity world coordinates:

  • Units: meters
  • Y axis: up
  • Positions include movement of the XR Origin

Hand-joint poses from XR Hands are initially relative to the XR tracking origin, so XRHandJointLogger transforms them into Unity world space before saving.

Always record the coordinate-frame convention with the dataset. A column named x is meaningless if nobody knows what frame x belongs to.

Troubleshooting

Most pipeline problems come from a missing setting, permission, or component reference rather than from the logging code itself. Start by checking the Unity Console for the printed session path and any errors, then use the symptoms below to narrow down which stream is failing.

No CSV files appear

  • Confirm XRSessionLogger is active.
  • Check the Unity Console or device log for the printed SessionDirectory.
  • If running through Link, look on the PC, not in Quest Android storage.
  • If running a standalone build (Further Study), look under the Quest application’s Android files directory.
  • Verify the app exited normally or paused long enough for the writers to flush.

Controllers move, but input values stay at zero

  • Verify the Quest controllers are active.
  • Check OpenXR controller profiles and the project’s XRI input setup.
  • Inspect the left_tracked / right_tracked columns.

hands.csv contains invalid joints

  • Verify XR Hands is installed.
  • Enable the OpenXR Hand Tracking Subsystem.
  • Enable hand tracking on the Quest.
  • Put the controllers down if the headset is configured to auto-switch between controllers and hands.

RGB frames are not saved

  • Quest 3 / 3S is required for the Passthrough Camera API.
  • For Link testing, verify Meta Horizon Link Developer Runtime Features and Passthrough Camera API permissions are enabled, then restart Unity.
  • Verify passthrough / camera access is enabled in the Meta project settings.
  • Verify horizonos.permission.HEADSET_CAMERA exists.
  • In a standalone build, accept the runtime permission prompt.
  • Confirm PassthroughCameraAccess.IsPlaying becomes true.
  • Test on the physical headset. XR Simulator does not provide Passthrough Camera API data.

RGB capture reduces frame rate

  • Keep the teaching configuration at 640×480 and 2 FPS.
  • Do not call GetColors() every rendered frame.
  • Reduce the saved frame rate before reducing the XR application’s display frame rate.

Putting It into Practice

A working pipeline is only the first step. Apply the practices below whenever you collect XR data, use the in-class exercise to confirm that every stream is captured and synchronized, and then consider how the same session structure can support larger engineering studies.

Data Collection Best Practices

For useful XR datasets:

  1. Use anonymous session IDs.
  2. Record metadata such as application version and device model.
  3. Keep a shared session clock.
  4. Record tracking validity, not just pose values.
  5. Separate continuous data from events.
  6. Save camera metadata with the images.
  7. Choose sampling rates intentionally.
  8. Test performance in the headset.
  9. Document the coordinate frame.
  10. Respect privacy and research requirements.

Where This Leads

Once this pipeline works, it can support more advanced engineering applications:

  • Compare controller and hand interaction strategies
  • Analyze workspace reach and ergonomics
  • Measure task time and error sequences
  • Synchronize human interaction with robot or machine telemetry
  • Detect real-world objects from RGB
  • Build AR task-recognition systems
  • Quantify training behavior
  • Study human-robot collaboration
  • Create replayable multimodal datasets for research

The same architecture works in both VR and passthrough AR/MR. The main difference is which sensors and environment information are available.

Standalone Data Collection

This section is not required in class. Once the Link workflow works, the same logger can be deployed directly to Quest for untethered experiments, such as field studies or sessions where a PC tether would interfere with the task. No code changes are needed.

Standalone behavior: In a standalone build, the application runs on Quest, so Application.persistentDataPath is in the Android app’s storage. The files are written on the headset, not on your PC.

Build and Run on Quest

  1. In the course project, switch the build target to Android and confirm OpenXR and the Hand Tracking Subsystem are enabled on the Android tab, as set up in Modules D and E.
  2. Connect the Quest 3 or Quest 3S and build and run the application.
  3. Accept the camera permission prompt inside the headset (requested by QuestCameraPermission).
  4. Disconnect the cable, record a short session, and exit the application normally so the files are flushed.

Retrieve the Session with ADB

Sessions are stored under:

/sdcard/Android/data/<package-name>/files/XRDataSessions/

Replace <package-name> with the application’s package identifier from Unity Player Settings. Then list and pull a session:

adb devices
adb shell ls /sdcard/Android/data/<package-name>/files/XRDataSessions
adb pull /sdcard/Android/data/<package-name>/files/XRDataSessions/<session-folder> .

You can also use Meta Quest Developer Hub to browse the files, but adb pull is repeatable and easy to script for research workflows. Point the same Streamlit dashboard at the pulled folder; the dashboard does not care whether the session came from Link or a standalone headset.

The logger code is the same. The difference is where the application executes and therefore where its files are written.

  Link / Play Mode Standalone Quest
Application executes on PC Quest
Data location PC Quest Android storage
Need ADB to retrieve files No Yes
Fast code iteration Excellent Slower
Final performance validation No Yes
Untethered data collection No Yes

References & Documentations