Platforms

NVIDIA XR AI Enters Public Beta with Spatially-Aware Agents for Industrial AR Glasses

NVIDIA's first production SDK purpose-built for AR hardware lets developers wire real-time scene understanding, depth sensing, and enterprise knowledge retrieval into agents that perceive and act on the factory floor.

Mainstay Communications Team · September 21, 2026 · 4 min read

In brief

NVIDIA released XR AI, a developer library for building multimodal AI agents on AR glasses and XR devices, into public beta on June 16, 2026, coinciding with the Augmented World Expo in Long Beach, California. The SDK routes video, audio, depth, and sensor data from wearable hardware into NVIDIA's model stack, connecting agents to enterprise knowledge systems via the Model Context Protocol. Siemens is among the first industrial partners exploring the framework for factory maintenance workflows, and hardware maker VITURE unveiled the first device built natively on the platform at the same event.

What XR AI Is

NVIDIA describes XR AI as a developer library that connects the real-world inputs of AR glasses and XR headsets to AI models, enterprise data, and accelerated computing. The framework ingests video, audio, depth, pose, and sensor data from wearable hardware and routes it through a modular processing stack the company calls the XR Media Hub.

The architecture separates into five layers: media transport, model services, tool access, agent orchestration, and client delivery. That separation lets developers swap hardware clients, models, MCP servers, orchestration frameworks, and deployment environments without rebuilding the agent from scratch.

Video pixels stay in shared memory while lightweight metadata flows through the pipeline, reducing unnecessary inference and data movement, a design choice that matters on hardware running on battery-constrained glasses.

The Model Stack

Visual understanding relies on NVIDIA's Cosmos-Reason1-7B vision-language model, which analyzes live camera frames to interpret the wearer's environment and task context. Speech-to-text runs through Parakeet-TDT-0.6b-v3, converting voice queries into text before the VLM processes the frame.

For latency-sensitive responses the framework calls Llama-3.1-Nemotron-Nano-8B-v1. Deeper tool-calling workflows route to NVIDIA-Nemotron-3-Nano-30B-A3B. The dual-model approach lets smaller models acknowledge quickly while larger models handle reasoning that takes longer.

Agent orchestration runs on the NVIDIA NeMo Agent Toolkit, implementing ReAct patterns. Enterprise data connects through Model Context Protocol servers. Pre-built MCP servers cover visual question answering, video analysis, vector and spatial utilities, and transcript retrieval. Custom servers can reach RAG pipelines, databases, digital twins, and asset-management systems.

NVIDIA Metropolis and its Video Search and Summarization service handle video understanding and searchable visual knowledge capture, supporting reporting, training, compliance, and retrieval workflows. NVIDIA NeMo Retriever provides the enterprise knowledge retrieval and RAG layer.

Industrial Use Cases

Siemens is exploring XR AI alongside NVIDIA DGX Spark for factory maintenance applications. The scenario NVIDIA describes in its technical blog involves an engineer wearing lightweight AR glasses who asks an AI agent about a programmable logic controller fault and receives real-time guidance that pulls from industrial systems, digital twins, and automation workflows, hands-free and without leaving the task.

The use case maps directly to the pain points of industrial maintenance: documentation buried in PDFs, procedures that require both hands, and experts who can't always be on-site. The agent layer adds real-time reasoning over those knowledge stores without requiring the engineer to stop work to consult a screen.

Rana (AutoBio) is deploying XR AI through its LabOS system for scientific research. LabOS provides hands-free guidance for stem cell therapy and gene-editing workflows at Stanford's Cong Lab and Princeton's Wang Lab, running on smart glasses from Meta, Rokid, and VITURE.

The Surreality Lab at UPMC is implementing a surgical assistance pipeline designed to surface information for operating room teams without pulling the surgeon's attention from the patient. Innoactive is applying the framework to automotive design-review workflows, and Atlantic Studios has built an interactive Titanic exploration experience on the same platform.

The First Dedicated Device

VITURE unveiled the Helix at AWE 2026 on June 16 as the first AI safety glasses built natively on NVIDIA's XR AI solution. The device carries a 12MP first-person camera, a four-microphone array, stereo speakers, Wi-Fi, and Bluetooth 5.3 connectivity. Battery life runs 60-plus minutes, with charge-while-using support. The frame design targets industrial, clinical, and life-sciences environments; ANSI Z87.1-2025 certification is in progress. The Helix runs standalone with no companion phone required.

Reservations for the first production batch are open at $600 and enterprise pilot allocations are invite-only. Shipping is expected in Q1 2027.

Supported Hardware and Deployment

Beyond VITURE, XR AI currently supports Meta and Rokid smart glasses. The framework targets AI glasses, AR headsets, XR headsets, mobile devices, web clients, and CloudXR-powered experiences. Spatial rendering through CloudXR lets agents create and manipulate objects in the wearer's physical environment via MCP tool calls.

Deployment targets span cloud, data center, and edge environments, including NVIDIA DGX Spark and DGX Station systems and RTX PRO workstations. Developer resources are available at developer.nvidia.com/xr/xr-ai.

Sources

Companies mentioned

Mainstay Digital · Spatial and 3D visualization for industrial companies

Back to Newsroom