X-Mobility: End-To-End Generalizable Navigation via World Modeling

Code | Paper

Description:

X-Mobility is an E2E generalizable navigation model using world modeling. It enables the robot navigation in an E2E manner by taking RGB image, robot state and (optional) route as inputs, then outputs the action commands for the robot directly.

This model is ready for commercial/non-commercial use.

License/Terms of Use

NVIDIA Open Model License https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/.

Model Architecture:

Architecture Type: Vision Transformer, Multilayer Perceptron

Network Architecture: X-Mobility introduces a learned state representation for robot navigation to capture relevant mobility features. It leverages a state estimation network with recurrent modules to handle non-Markovian effects, combining visual and robot state information for robust belief updates. A multi-task decoding framework enables RGB reconstruction and semantic segmentation, ensuring the latent state supports downstream tasks. The action policy network fuses the latent state with route encoding using self-attention, enabling effective action policy learning for linear and angular velocities, along with an optional navigation path for enhanced safety and control.

Input:

Input Type(s): Image, Robot state, Route

Input Format(s): RGB, Floating points, Floating points

Input Parameters: (2D, 1D, 1D)

Other Properties Related to Input: Image size (width, height) being a multiplier of (5, 8) Robot states contains robot’s current linear velocity of size (1) Route is list of consecutive 2D segments from robot position towards the final destination of size (num_segments, 4)

Output:

Output Type(s): Actions

Output Format: Floating points

Output Parameters: (1D)

Other Properties Related to Output: Actions consist of desired linear and angular velocities of size (6)

Software Integration:

Runtime Engine(s): TensorRT

Supported Hardware Microarchitecture Compatibility: All of the below:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Jetson
• NVIDIA Hopper
• NVIDIA Lovelace
• NVIDIA Pascal
• NVIDIA Turing
• NVIDIA Volta
• NVIDIA A100

[Preferred/Supported] Operating System(s):

• Linux

Model Version(s):

This is the initial version of the model, version 1.0.

Training, Testing, and Evaluation Datasets:

Training Dataset:

Link: https://huggingface.co/datasets/nvidia/X-Mobility

Data Collection Method by dataset: Automated

Labeling Method by dataset: Automated

Properties: 100,000 total frames using Nav2 160,000 total frames using random actions Warehouse environment data Specific sensor type: Camera for images, wheel encoder for speed

Dataset License(s): https://huggingface.co/datasets/nvidia/X-Mobility/blob/main/LICENSE

Testing Dataset:

Link: https://huggingface.co/datasets/nvidia/X-Mobility

Data Collection Method by dataset: Automated

Labeling Method by dataset: Automated

Properties: 100,000 total frames using Nav2 160,000 total frames using random actions Warehouse environment data Specific sensor type: Camera for images, wheel encoder for speed

Dataset License(s): https://huggingface.co/datasets/nvidia/X-Mobility/blob/main/LICENSE

Evaluation Dataset:

Link: https://huggingface.co/datasets/nvidia/X-Mobility

Data Collection Method by dataset: Automated

Labeling Method by dataset: Automated

Properties : 100,000 total frames using Nav2 160,000 total frames using random actions Warehouse environment data Specific sensor type: Camera for images, wheel encoder for speed

Dataset License(s): https://huggingface.co/datasets/nvidia/X-Mobility/blob/main/LICENSE

Inference:

Engine: Tensor(RT)

Test Hardware: Jetson Orin

Ethical Considerations :

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Model Limitations:

This model is not tested or intended for use in mission critical applications that require functional safety. The use of the model in those applications is at the user's own risk and sole responsibility, including taking the necessary steps to add needed guardrails or safety mechanisms.

Risk: Model underperformance in highly dynamic environments. Mitigation: Enhance dataset with dynamic obstacle scenarios and fine-tune models accordingly.

Risk: Integration challenges in specific customer environments. Mitigation: Provide detailed integration guides and support, leveraging NVIDIA's ecosystem.

Risk: Limited initial support for certain robot embodiments. Mitigation: Expand testing and validation across a wider range of robot platforms.