HMGMA Lab: XR-Guided Component Installation and Verification

Build a compact Unity XR prototype that guides a worker through the installation and verification of a virtual engineering component.

Associated professional lecture: XR for Industrial Training & Real-Time Task Support

Placeholder for XR-guided manufacturing lab hero image

Hero image placeholder — to be replaced

Estimated duration

  • Required VR lab: approximately 60–75 minutes after Unity and the starter project are available
  • Optional passthrough MR extension: approximately 30–45 additional minutes
  • Unity installation and initial hardware setup are not included in the estimate

Participants are not expected to complete all of Modules D and E during this lab.

Lab Overview

This lab demonstrates how a simple engineering procedure can be represented through XR interaction, task state, guidance, feedback, and verification. The required VR experience is a training prototype that walks a participant through inspecting, installing, and verifying one virtual engineering component. The optional passthrough extension reframes the same concept as contextual task support in a physical workspace.

You will work in a prepared Unity project or scene rather than configure every XR subsystem from an empty project. The lab reuses patterns from Module D and, for the optional extension, Module E passthrough sessions.

Item Detail
Required mode VR
Optional mode Passthrough mixed reality
Platform Unity
Primary framework XR Interaction Toolkit
Suggested hardware Meta Quest 3 or compatible OpenXR headset
Headset-free early testing XR Device Simulator where appropriate
Expected required-lab duration 60–75 minutes
Difficulty Introductory to intermediate
Final result One guided component-installation interaction

Learning Outcomes

By completing the lab, participants should be able to:

  • Represent a compact engineering procedure as a sequence of XR task states. Connect instruction text, status labels, and interaction events to a small state machine that mirrors a real work step.
  • Configure a virtual engineering component for grabbing with Unity’s XR Interaction Toolkit. Apply collider, Rigidbody, and XR Grab Interactable settings suitable for industrial-scale manipulation.
  • Configure a socket that accepts and aligns the component at its intended installation location. Position an attach transform that reflects the final installed pose from multiple viewpoints.
  • Connect interaction events to instructions, status changes, and completion logic. Use socket selection and UI activation events rather than release-only heuristics.
  • Create a readable world-space instruction and verification interface. Build a compact panel with clear labels, status text, and verification controls.
  • Add clear feedback for correct actions and blocked actions. Use visual, audio, and optional haptic cues that communicate task state.
  • Test the required interaction using a simulator or supported VR headset. Validate reach, readability, and interaction flow before judging industrial readiness.
  • Explain how the same interaction could be adapted for passthrough task support. Distinguish VR training from MR contextual overlays anchored to physical surfaces.
  • Identify additional validation required before an XR prototype could be used in real industrial training. Recognize gaps in work instructions, geometry, assessment, safety, and operational support.

Manufacturing Scenario

A worker must install a small engineering component—such as a sensor module, electrical control module, mounting bracket, inspection cover, or connector assembly—onto a fixture and confirm that the installation has been completed. The XR experience presents the current step, allows the worker to manipulate the component, detects correct placement, and unlocks a verification action.

This lab uses neutral object names only. It does not represent an actual proprietary production process.

Work sequence

  1. Inspect: Review the component and the installation location.
  2. Install: Pick up the component and place it into the designated socket.
  3. Verify: Activate the verification control after correct installation.

Task states

State Meaning
Ready Task initialized; participant may inspect the workstation
Component selected Participant has grabbed the engineering component
Component installed Component is socketed in the correct installation location
Verification blocked Participant attempted verification before installation
Task complete Verification succeeded; installation confirmed

Verification blocked occurs when the participant attempts to verify the task before the component is installed. This provides corrective feedback without requiring a second incorrect socket or additional mandatory objects.

Final Lab Outcome

When the required lab is complete, the prototype must demonstrate:

  • A world-space panel shows the current task step.
  • The engineering component can be grabbed and released.
  • The component snaps into the correct installation socket.
  • Correct placement updates the instruction and system status.
  • Verification is unavailable or rejected before installation.
  • Verification succeeds after correct installation.
  • Completion produces clear visual feedback.
  • Audio or haptic feedback is included where available.
  • The task can be reset for another participant.

Keep the scope small: One workstation, one component, one socket, one verification action, and one complete interaction loop are sufficient. See the Assignment D required features for a related compact-interaction model.

Before You Begin

This tutorial assumes one of the following:

  • A provided starter project with the XR rig and workstation already configured, or
  • A Unity VR Template project configured according to Module D1

Instructor preparation — resource placeholders

The following resources are not yet available. Replace these placeholders before distributing the lab to participants:

  • TODO: Add HMGMA lab starter project download link
  • TODO: Add completed reference project link, if one will be provided
  • TODO: Add supported Unity version
  • TODO: Add required XR Interaction Toolkit version
  • TODO: Add recommended headset/runtime details

Prerequisite checklist

  • Unity and required platform modules are installed.
  • The project opens without compilation errors.
  • The scene contains a functional XR Origin.
  • Controller or simulator input is working.
  • The scene contains an XR Interaction Manager.
  • The required XRI input actions are enabled.
  • Participants know how to select GameObjects and add components in the Inspector.
  • Participants can consult Modules B and C when they need help with Unity components, UI, or C#.

Required VR Lab Tutorial

Part 1 — Open and Inspect the Prepared Workstation

Purpose: Confirm that the prepared scene is functional and identify the objects you will configure during the lab.

  1. Open the provided project and the HMGMA lab scene.
  2. Enter Play Mode before changing anything.
  3. Confirm that head and controller movement or simulation input works.
  4. Identify these objects in the Hierarchy:
    • XR Origin
    • XR Interaction Manager
    • EngineeringComponent
    • InstallationFixture
    • InstallSocket
    • VerificationPanel
    • TaskController
  5. Exit Play Mode.
  6. Save a personal copy of the scene before editing.

Do not rebuild the XR rig from scratch. The prepared project should already include the VR Template setup, XR Interaction Toolkit, and XR Device Simulator configuration from Module D1.

Check your work

  • Scene opens without errors.
  • XR input responds.
  • The workstation is visible at an appropriate scale.
  • The component is reachable.
  • The instruction panel is readable.
  • You understand which objects will be edited.

Why this matters: In industrial training, technicians must trust that the digital environment matches the physical task layout. Inspecting scale, reach, and object naming before editing prevents wasted configuration time and catches setup issues early.

Part 2 — Configure the Engineering Component for Grabbing

Purpose: Make EngineeringComponent grabbable with predictable near- and far-interaction behavior.

  1. Select EngineeringComponent in the Hierarchy.
  2. Verify or add a valid Collider that matches the component’s visible geometry.
  3. Verify or add a Rigidbody. Use a kinematic or dynamic setup consistent with your project’s XRI movement type.
  4. Verify or add XR Grab Interactable.
  5. Set an appropriate XRI Interaction Layer compatible with the participant’s interactor.
  6. Choose a suitable Movement Type (commonly Instantaneous or Kinematic for precise placement tasks).
  7. Enable Track Position and Track Rotation unless a constrained interaction requires otherwise.
  8. If the default pivot creates an unnatural grip, create a child Attach Transform and assign it on the grab interactable.

Component roles

  • The collider defines the interactive physical region.
  • The Rigidbody allows the object to participate in Unity physics.
  • XR Grab Interactable provides the XRI grabbing behavior.
  • The XRI interaction layer must be compatible with the participant’s interactor.
  • The attach transform should make the component feel natural when held.

Do not require a custom grab script for this lab.

Check your work

  • The component highlights or responds when targeted.
  • The component can be grabbed.
  • It follows the hand or controller predictably.
  • It releases without jumping or rotating unexpectedly.
  • Its scale and reach distance feel plausible.

Reference: XR Grab Interactable, Grab Transformers, Preserving Orientation

Why this matters: Grabbing is the primary manipulation action in assembly training. An unnatural grip or unstable release undermines confidence and makes socket placement harder to evaluate.

Part 3 — Configure the Installation Socket

Purpose: Create one installation socket that accepts only the engineering component and snaps it to the correct final pose.

  1. Inspect or create a child object named InstallSocket under InstallationFixture.
  2. Add a trigger collider sized for practical placement—not so large that the part snaps from an unrealistic distance.
  3. Add XR Socket Interactor.
  4. Create an Attach Transform child and position it at the intended installed pose.
  5. Set the Interaction Layer Mask so the socket accepts only EngineeringComponent.
  6. Assign the attach transform to the socket.

Positioning the attach transform

  1. Temporarily position the component in its intended final installed pose.
  2. Align the attach transform with that pose.
  3. Verify its local axes.
  4. Assign the transform to the socket.
  5. Test from more than one viewing angle.

Use one installation socket only. Correct visual alignment in one camera view is not sufficient; inspect orientation and clearance from multiple viewpoints.

Check your work

  • The socket detects only the intended component.
  • The component snaps into a stable final pose.
  • The final orientation is correct.
  • The part is not visibly floating or intersecting the fixture.
  • The socket does not capture the part from an excessive distance.

Reference: XR Socket Interactor, Socket Implementation, Snapping with Constraints

Why this matters: In manufacturing training, incorrect alignment tolerance teaches the wrong assembly behavior. The socket should represent a meaningful installation constraint, not approximate placement.

Part 4 — Connect the Task States

Purpose: Link grab, socket, and verification events to instruction text and status labels through a small task controller.

Use a compact task-state controller rather than a large architecture. Wire interaction events to public methods on TaskController; do not infer installation completion from component release alone.

Task logic

State Instruction Status Verification
Initial Inspect the component and installation location. Ready Available visually but rejects premature use
Component selected Place the component into the highlighted installation location. Component selected Still blocked
Component installed Confirm the installation using the verification control. Component installed Valid
Premature verification Install the component before verification. Verification blocked Rejected with warning
Completed Installation verified. Task complete Success feedback

When verification is attempted before installation, show a brief warning state and optionally play a short error sound. Do not mark the task complete.

After successful verification, update the status indicator, change instruction text, and optionally play a completion sound and haptic pulse.

Event wiring

  • The component’s grab or select event may call OnComponentSelected().
  • The socket’s Select Entered event should call OnComponentInstalled().
  • The verification button’s OnClick event should call TryVerifyInstallation().
  • The reset button’s OnClick event should call ResetTask().

Example controller

using TMPro;
using UnityEngine;

public enum GuidedTaskState
{
    Ready,
    ComponentSelected,
    ComponentInstalled,
    VerificationBlocked,
    TaskComplete
}

public class GuidedInstallationTask : MonoBehaviour
{
    [Header("UI References")]
    [SerializeField] private TMP_Text instructionText;
    [SerializeField] private TMP_Text statusText;

    [Header("Feedback")]
    [SerializeField] private Renderer statusIndicator;
    [SerializeField] private Color readyColor = Color.gray;
    [SerializeField] private Color warningColor = Color.yellow;
    [SerializeField] private Color successColor = Color.green;
    [SerializeField] private AudioSource audioSource;
    [SerializeField] private AudioClip blockedClip;
    [SerializeField] private AudioClip completeClip;

    private GuidedTaskState currentState = GuidedTaskState.Ready;
    private bool isInstalled;

    private void Start()
    {
        ResetTask();
    }

    public void OnComponentSelected()
    {
        if (currentState == GuidedTaskState.TaskComplete)
            return;

        currentState = GuidedTaskState.ComponentSelected;
        UpdateUI(
            "Place the component into the highlighted installation location.",
            "Component selected");
    }

    public void OnComponentInstalled()
    {
        isInstalled = true;
        currentState = GuidedTaskState.ComponentInstalled;
        UpdateUI(
            "Confirm the installation using the verification control.",
            "Component installed");
    }

    public void TryVerifyInstallation()
    {
        if (currentState == GuidedTaskState.TaskComplete)
            return;

        if (!isInstalled)
        {
            currentState = GuidedTaskState.VerificationBlocked;
            UpdateUI(
                "Install the component before verification.",
                "Verification blocked");
            SetIndicator(warningColor);
            PlayClip(blockedClip);
            return;
        }

        currentState = GuidedTaskState.TaskComplete;
        UpdateUI("Installation verified.", "Task complete");
        SetIndicator(successColor);
        PlayClip(completeClip);
    }

    public void ResetTask()
    {
        isInstalled = false;
        currentState = GuidedTaskState.Ready;
        UpdateUI(
            "Inspect the component and installation location.",
            "Ready");
        SetIndicator(readyColor);
    }

    private void UpdateUI(string instruction, string status)
    {
        if (instructionText != null)
            instructionText.text = instruction;

        if (statusText != null)
            statusText.text = status;
    }

    private void SetIndicator(Color color)
    {
        if (statusIndicator != null)
            statusIndicator.material.color = color;
    }

    private void PlayClip(AudioClip clip)
    {
        if (audioSource != null && clip != null)
            audioSource.PlayOneShot(clip);
    }
}

Attach this script to TaskController and assign references in the Inspector. Extend it with haptic calls if your project already includes Simple Haptic Feedback.

Check your work

  • Instructions update when the component is grabbed.
  • Status changes only after successful socket selection—not on release alone.
  • Premature verification shows blocked status and warning feedback.
  • Successful verification marks the task complete.
  • Reset returns the task to Ready.

Reference: Common Activation Paths, Event-Driven Design, Contextual Interactions, Custom Unity Events

Why this matters: Industrial procedures are governed by state and interlocks. A verification step should not succeed until prerequisite actions are confirmed—mirroring real work-instruction logic.

Part 5 — Create the Instruction and Verification Panel

Purpose: Build a readable world-space interface for instructions, status, verification, and reset.

Configure VerificationPanel as a compact world-space Canvas containing:

  • Current step or instruction text
  • Current task status
  • A verification button labeled VERIFY
  • A reset button labeled RESET
  • Optional compact progress text such as Step 2 of 3

Configuration checklist

  1. Set the Canvas Render Mode to World Space.
  2. Confirm the scene EventSystem uses XR UI Input Module.
  3. Add Tracked Device Graphic Raycaster to the Canvas.
  4. Confirm the XR interactor supports UI interaction.
  5. Size text for readability at the expected viewing distance.
  6. Make buttons large enough for ray or poke interaction.
  7. Position the panel so it does not obstruct the component or installation area.
  8. Wire the verification button to call TryVerifyInstallation() once per activation.

Use direct labels such as READY, INSTALL COMPONENT, and TASK COMPLETE in status or instruction text. Do not create a dense dashboard.

Check your work

  • Panel text is readable in Play Mode.
  • VERIFY and RESET respond to XR UI input.
  • VERIFY invokes the task controller correctly.
  • Panel placement does not block the installation area.

Reference: UI as Input, Interactive UI Elements

Why this matters: Technicians rely on work instructions and status displays during assembly. A compact, legible panel models how XR can present the current step without replacing approved documentation.

Part 6 — Add Feedback

Purpose: Communicate task state through visual, audio, and optional haptic cues.

Visual feedback is required. Audio and haptics are recommended when available.

Interaction Recommended feedback
Component targeting Subtle hover highlight
Component grab Short selection sound or light haptic pulse
Correct installation Fixture or socket success color; confirmation sound; optional haptic pulse
Premature verification Warning text; brief warning color; short error sound
Task completion Success status; clear completion sound; optional completion haptic pulse

Feedback should communicate task state, not merely make the prototype feel more game-like.

Avoid:

  • Constant vibration
  • Excessively loud sounds
  • Long celebratory animations
  • Flashing effects
  • Red/green color alone without supporting text
  • Feedback that implies a real machine is safe when the prototype is only a simulation

Check your work

  • Hover and grab feedback are noticeable but not distracting.
  • Installation success is visually distinct from blocked verification.
  • Completion feedback is clear and brief.
  • Audio levels are comfortable in headset or speakers.

Reference: Simple Haptic Feedback, 3D Spatial Audio

Why this matters: In training environments, feedback confirms whether an action was recognized and whether the next step is allowed—similar to indicator lights, torque confirmations, or scan acknowledgments on a production line.

Part 7 — Test the Complete Interaction

Purpose: Validate the full interaction loop before considering the lab complete.

Required test sequence

  1. Start or reset the task.
  2. Attempt verification before installing the component.
  3. Confirm that verification is blocked and corrective feedback appears.
  4. Grab the component.
  5. Confirm that the instruction updates.
  6. Place the component into the installation socket.
  7. Confirm that the socket accepts and aligns it.
  8. Confirm that the instruction changes to verification.
  9. Activate the verification control.
  10. Confirm that the system reports task completion.
  11. Reset the task.
  12. Repeat the sequence without errors.

Validation table

Test Expected result Pass / needs attention
Scene initialization Task starts in Ready with inspect instruction  
Component grab Component follows controller; status updates  
Component release (not socketed) Component remains grabbable; installation not marked complete  
Socket acceptance Component snaps to final pose  
Final alignment Orientation and clearance look correct from multiple views  
Blocked verification Warning status and feedback before installation  
Successful verification Task complete after correct installation  
Visual feedback State changes are visible without color-only reliance  
Audio/haptic feedback Cues play at grab, block, and completion if configured  
Reset behavior Task returns to Ready for another run  
UI readability Instruction and status text readable at working distance  
Interaction reach Component and buttons reachable without strain  
Simulator or headset behavior Input and interaction work on chosen test platform  

Testing notes:

  • Simulator testing is acceptable for early development.
  • Final judgments about reach, comfort, scale, haptics, passthrough, and spatial readability require suitable hardware testing.
  • A functioning prototype is not automatically a validated industrial training system.

Reference: Structured Core Tasks for a related compact validation approach.

Optional Extension: Passthrough Mixed Reality Task Support

Complete the VR lab first. This extension requires compatible passthrough hardware and may require additional device configuration and standalone testing.

This optional section adapts the same workstation concept for passthrough mixed reality. It is not required to complete the lab.

Objective

  • Display the physical environment using passthrough.
  • Identify or select a suitable tabletop or work surface.
  • Preview the virtual workstation on the surface.
  • Require explicit user confirmation.
  • Create or use an anchor for the confirmed placement.
  • Place the virtual component, fixture, or instruction panel under the anchored content root.
  • Walk around the placement and inspect its stability.
  • Use the panel as a contextual task-support interface.

Concepts to distinguish

Concept Role
Detected plane Environmental surface estimate from AR Foundation
Raycast or candidate pose Proposed placement location—not yet confirmed
Confirmed placement User-approved pose before anchoring
Anchor Trackable reference that stabilizes content relative to the physical environment
Persistent content Content parented under an anchor for the session

A raycast hit is not already an anchor. Anchoring does not prove accurate physical registration with real machinery.

Extension steps

  1. Open the provided MR scene or Unity MR Template-based scene.
  2. Confirm passthrough works on the supported headset.
  3. Identify a tabletop or suitable surface using plane detection.
  4. Preview the workstation or task panel at the candidate pose using the Spatial Panel Manipulator.
  5. Confirm placement explicitly.
  6. Anchor the content root using anchors and placement and anchoring workflow.
  7. Verify scale, orientation, readability, and physical clearance.
  8. Walk around the content and inspect stability.
  9. Reset and repeat placement.
  10. Document limitations such as drift, imperfect plane detection, and registration error.

Do not implement marker tracking, camera-frame computer vision, QR recognition, object recognition, persistent cloud anchors, shared anchors, or precision registration with real machinery for this extension.

Reference: What Is Passthrough AR?

Safety: Passthrough mixed reality does not eliminate the need for physical-space awareness. Keep the test area clear, preserve access to boundaries and exit paths, and do not treat virtual overlays as certified safety indicators.

Industrial Deployment Reflection

Before this prototype could become a real training or task-support application in a manufacturing facility, many additional validations would be required.

Consider:

  • Approved work instructions
  • Accurate CAD and component geometry
  • Product and process variants
  • Realistic tolerances and constraints
  • Common errors and recovery procedures
  • Assessment and qualification criteria
  • Accessibility and worker differences
  • Hardware deployment and sanitation
  • Content ownership and revision control
  • IT, cybersecurity, and system integration
  • Training effectiveness and transfer to real work
  • Safety and human-factors validation
  • Operational support and maintenance

Reflection prompts

Write three short items:

  1. One change required for technical validity
  2. One change required for workforce adoption
  3. One measure that would indicate the system is creating value

Troubleshooting

The component cannot be grabbed

Check:

  • Collider present and enabled
  • Rigidbody present
  • XR Grab Interactable present and enabled
  • Matching XRI interaction layers between interactor and interactable
  • Physics layers not blocking interaction casts
  • Enabled input actions
  • Active interactor in the scene
  • Play Mode Console errors

The component will not enter the socket

Check:

  • Socket trigger collider present
  • Is Trigger enabled on the socket collider
  • XR Socket Interactor present and active
  • Matching interaction layers between component and socket
  • Socket detection volume not too small or obstructed
  • Another interactor is not still holding the object
  • The part is not already selected by another socket

The component snaps into the wrong orientation

Check:

  • Attach transform position and rotation
  • Local axes on the attach transform
  • Object pivot versus visual mesh center
  • Temporary final-pose alignment method
  • Component scale
  • Socket and fixture orientation

The verification button does not respond

Check:

  • Canvas set to World Space
  • EventSystem uses XR UI Input Module
  • Canvas has Tracked Device Graphic Raycaster
  • XR interactor has UI interaction enabled
  • Button Graphic has Raycast Target enabled
  • Button OnClick wired to TryVerifyInstallation()
  • Task-controller reference assigned

Verification succeeds too early

Check:

  • Installation state initialized to false in ResetTask()
  • TryVerifyInstallation() checks installation state before completing
  • State changes only from successful socket Select Entered
  • Component release does not call the installation-complete method

Haptics do not play

Check:

  • HapticImpulsePlayer on the controller
  • Simple Haptic Feedback configured on the interactor or interactable
  • Hardware support on the target headset
  • OpenXR input configuration
  • Testing on actual headset—not simulator only
  • Amplitude and duration values above zero

Passthrough content drifts or is misaligned

Check:

  • Placement was confirmed before anchoring
  • Content pivot and scale
  • Anchor parenting hierarchy
  • Tracking quality and lighting conditions
  • Validation from multiple viewpoints
  • Whether expected accuracy exceeds platform capabilities

Completion Checklist

Participants have completed the required lab when:

  • The scene loads without errors.
  • The component can be grabbed.
  • The component installs into the correct socket.
  • The task panel updates during the sequence.
  • Premature verification is rejected.
  • Correct installation enables successful verification.
  • Completion feedback is clear.
  • The task can be reset.
  • The participant can explain the difference between the VR training prototype and the optional passthrough task-support extension.

Key Takeaways

  • Industrial XR experiences should be organized around clear task states and decisions.
  • Standard XR Interaction Toolkit components can support a useful prototype without extensive custom code.
  • Correct placement should be detected through the installation interaction, not inferred from object release.
  • Guidance and feedback should clarify task state without overwhelming the worker.
  • A working XR prototype still requires engineering, learning, usability, safety, and operational validation before deployment.