Geometry in the token, not in the pipeline
A VLM answers "is the chair in front of the sofa" from a grid of 2D patch features that carry no depth, so it answers from word statistics. We lift each patch to a metric 3D point and cluster the points into voxel tokens with a soft assignment. The question-answering loss trains the depth, the camera, and the grouping together.
- Reported
- Spatial reasoning improves with more spatial QA data, or with detection and segmentation estimators added at inference.
- Found
- Language loss alone learns a text shortcut: 41.5% on binary relations, below chance. Moving geometry into the tokenizer reaches 81.33% on SpatialRGPT-Bench at 1B parameters in 3.6 GB, with no region proposal at inference.