ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation Models

🌐 Project Page  |  🤗 Hugging Face  |  📔 Technical Report  |  🤖 GitHub

Overview

ACE-Brain-0.6-9B is a 9B vision-language model that strengthens spatial grounding for physical foundation models. Given one or more images and a natural-language query, it identifies task-relevant objects, counts instances, resolves spatial relationships, and localizes interaction targets.

Built on Qwen3.5-9B, ACE-Brain-0.6-9B is fine-tuned on approximately 1.5 million training samples. A unified data pipeline converts robot manipulation recordings, egocentric interactions, object and part annotations, and spatial data into verifiable grounding supervision. Training updates the language-model parameters while keeping the vision encoder frozen.

On seven public grounding benchmarks, ACE-Brain-0.6-9B achieves a reported Avg-7 of 75.41, compared with 73.68 for GPT-6 Astra. It exceeds Astra on four of seven benchmarks, with remaining gaps on PixMo-Count, RefSpatial, and PointArena.

Key Features

• Unified grounding interface: Supports points, point lists, bounding boxes, integer counts, and categorical answers (yes, no, or none) for target identification, multi-instance perception, spatial grounding, and interaction grounding.
• Robot-centered data pipeline: Automatically annotates manipulation recordings and combines them with complementary visual sources to broaden grounding coverage.
• Verifiable supervision: Checks object labels, requires complete instance sets for counting, verifies absent-target questions, and excludes evaluation images and episodes from training.
• Scaling with diverse data: Grounding performance improves across progressively larger and more diverse training releases under a fixed backbone and training recipe.
• Strong robot-scene performance: Achieves 93.42 on RobotGrounding, the team's held-out evaluation set covering referring, counting, absent targets, spatial judgments, and cross-view correspondence.

Performance Highlights

Results below are reported in Table 1 of the technical report. Higher is better; bold marks the best score among the models shown.

Avg-7 is the unweighted mean of the seven public benchmark scores. RobotGrounding is evaluated separately and is not included in Avg-7. It contains 2,591 questions and reports a macro average over 14 scoring groups.

Inference Example

Install a Transformers version with Qwen3.5 support:

pip install -U "transformers>=5.2.0" accelerate torch torchvision pillow

The example uses the checkpoint and processor files from this repository in non-thinking mode. Replace scene.jpg and the query with your own image and grounding instruction.

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

model_id = "ACERobotics/ACE-Brain-0.6-9B"

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
).eval()

image = Image.open("scene.jpg").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {
                "type": "text",
                "text": (
                    "Point to the red cup in the image. "
                    "Return a point as [[x, y]], with integer coordinates "
                    "normalized to [0, 1000]. Return none if it is absent."
                ),
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

answer_ids = generated_ids[:, inputs["input_ids"].shape[1]:]
answer = processor.batch_decode(
    answer_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(answer)

Grounding coordinates are integers normalized to [0, 1000] along each image axis. Depending on the query, the model can return a point, a list of points, a box, a count, or a categorical answer. Free-space grounding identifies unoccupied 2D regions and does not by itself establish 3D placement feasibility.

The example uses greedy decoding for simplicity. The reported benchmarks use temperature 0.2 and a 128-token output budget, except PixMo-Count, which uses 1,024 tokens. See the report for the full evaluation protocol and the Transformers Qwen3.5 documentation for the model interface.

Citation

If you find ACE-Brain-0.6-9B useful for your research and applications, please refer to our technical report:

ACE-Brain Team. ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation Models.

Read the technical report (PDF). A formal BibTeX entry will be added when the arXiv identifier is available.