Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
†Equal contribution
ECCV 2026
†Equal contribution
Large Multimodal Models (LMMs) succeed on images and short videos, yet scaling them to long videos remains hard under frame-centric tokenization. 3D geometry offers a natural compression mechanism: depth and camera pose fuse multi-view observations into a persistent, world-aligned representation. Recent 3D LMMs improve spatial reasoning, but still lag specialists on grounding and segmentation — often because geometry-aware decoding is bottlenecked through language tokens, proposal selection, or lightweight grounding queries.
We introduce Qwen-3D, a geometry-aware LMM built on Qwen2.5-VL that (i) compresses visual tokens with multi-view geometric cues, (ii) applies 3D Rotary Positional Embeddings so attention operates in scene space, and (iii) connects contextualized backbone features to a query-based segmentation decoder for dense language-to-geometry grounding. Across grounding, segmentation, and VQA, Qwen-3D surpasses existing 3D LMMs and several proprietary 2D models, while joint 2D+3D training preserves strong 2D vision–language performance.
Existing 3D LMMs improve high-level spatial reasoning, yet remain far behind specialist detectors and segmentors.
A core issue is the interface between language reasoning and dense geometric prediction: most models decode 3D outputs
as text coordinates, select from detector proposals, or pass a single <REF> token into a lightweight head.
These interfaces create an information bottleneck between rich visual representations and precise 3D grounding.
Decoding object information in 3D LMMs. (i) Text-space decoding generates coordinates as tokens. (ii) Proposal selection is limited by external detectors. (iii) Special-token decoding compresses grounding into one latent. (iv) Qwen-3D routes full contextualized vision–language features into a mask decoder for dense 2D/3D grounding.
Qwen-3D extends Qwen2.5-VL with three tightly coupled components: a world-space scene representation, geometry-aware vision–language attention, and a shared mask-based grounding decoder.
Multi-view RGB features are unprojected with depth and camera pose, then voxel-pooled (~5cm) so redundant views collapse into compact, persistent 3D tokens.
Multimodal RoPE is extended to \((t, x, y, z)\), letting attention reason over nearby 3D locations instead of independent image frames — with full attention over visual tokens.
Object queries attend to the full visual and language token streams, predicting segmentation masks and text-span grounding scores without a single-token bottleneck.
Qwen-3D architecture. Multi-view RGB-D features from the frozen Qwen ViT are unprojected into world coordinates, voxel-pooled, and fused with text via LoRA-adapted VL attention with 3D Rotary PE. Shared tokens feed a 3D mask decoder (grounding / instance segmentation) and a text decoder (open-ended QA).
Training jointly supervises mask loss, text-span grounding loss, and text generation. The model shares parameters across 2D and 3D: 2D tokens use grid PE, while 3D tokens use world coordinates; we optionally lift 2D images to pseudo-3D with reconstruction models to improve alignment. Only ~50M trainable parameters are introduced (LoRA + mask decoder).
On ScanRefer, Qwen-3D-3B reaches 62.2 / 55.3 Acc@25/50 and the 7B variant reaches 62.8 / 54.7 — matching specialist UniVLG closely and establishing a new state of the art among 3D LMMs. The 3B model is already competitive on VQA; the 7B model leads ScanQA and SQA3D EM@1.
| Method | ScanRefer @25 | ScanRefer @50 | ScanQA EM@1 | SQA3D EM@1 |
|---|---|---|---|---|
| Experts | ||||
| UniVLG | 63.5 | 56.4 | 25.7 | 50.2 |
| Locate-3D | 61.1 | 50.9 | — | — |
| 2D VLMs | ||||
| GPT-4o | 48.2 | 32.5 | 18.0 | — |
| Qwen2.5-VL-7B | 41.6 | 14.9 | — | — |
| 3D LMMs | ||||
| Video-3D LLM-7B | 58.1 | 51.7 | 30.1 | 58.6 |
| LLaVA-3D-7B | 50.1 | 42.7 | 27.0 | 55.6 |
| Grounded 3D-LLM | 48.6 | 44.0 | — | — |
| Qwen-3D-3B (Ours) | 62.2 | 55.3 | 28.0 | 55.4 |
| Qwen-3D-7B (Ours) | 62.8 | 54.7 | 31.5 | 59.6 |
| Locate-3D / ScanNet++ (OOD) | ||
|---|---|---|
| Model | Acc@25 | Acc@50 |
| UniVLG | 32.3 | 24.6 |
| Video-3D LLM | 33.2 | 27.6 |
| Qwen-3D-3B | 55.7 | 43.4 |
| ScanNet200 Instance Seg. | ||
|---|---|---|
| Model | mAP | mAP25 |
| UniVLG (lang-prompted) | 27.9 | 46.1 |
| Grounded-3D-LLM | 12.1 | 16.8 |
| Qwen-3D-3B | 25.3 | 41.5 |
Controlled ablations on ScanRefer (Top-1@0.5, 50-scene subset) show that the full text/visual token interface, aligned 3D RoPE, full attention over visual tokens, and LoRA finetuning of the VLM are all critical.
| Ablation | Variant | Top-1 Acc. |
|---|---|---|
| Decoder interface | Qwen-3D with <REF> token | 39.7 |
| Decoder interface | Full text + visual tokens (ours) | 53.5 |
| Positional embedding | 2D / Vanilla 3D | 53.2 / 49.9 |
| Positional embedding | Aligned 3D RoPE (ours) | 53.5 |
| Attention mask | Causal | 36.4 |
| Attention mask | Full (ours) | 53.5 |
| VLM tuning | Frozen | 37.0 |
| VLM tuning | LoRA finetune (ours) | 53.5 |
Joint 2D+3D training also preserves backbone strength on RefCOCO / RealWorldQA (e.g., RefCOCO 88.1 vs. Qwen2.5-VL-3B 89.1), showing geometry-aware reasoning can be added without collapsing 2D competence.
Qwen-3D unifies 3D referential grounding, instance segmentation, situated VQA, and strong 2D grounding/segmentation under one architecture.
3D referential grounding on SR3D, NR3D, and ScanRefer.
Instance segmentation on COCO (2D) and ScanNet200 (3D).
Visual question answering on ScanQA, SQA3D, and RealWorldQA.
2D referential grounding retained after 3D adaptation.
@inproceedings{lin2026qwen3d,
title = {Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding},
author = {Lin, Lucy and Jain, Ayush and Liu, Yifan and Fragkiadaki, Katerina},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}