When a computer-use agent clicks the wrong place, which elementary spatial skill actually broke?
Every instruction points between two candidates: the target and its contrastive rival. Across 18,886 predictions, the click usually lands on neither. Pick a primitive:
Every item has a twin. The screenshot and anchor stay fixed; the relation expression changes; the correct target moves to the other candidate. A model that always clicks the salient element scores one twin and fails the other, so its pair-level score is zero. The instructions below are a real pair from the benchmark (pair 58, a UI-Vision toolbar). Try it:
Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations. Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset (Fleiss κ = 0.94 for well-formedness, κ = 0.79 for target selection) and select the instruction-consistent candidate on 96.9% of retained items.
Nineteen vision-language models reach at most 32% strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls in. Predictions fall outside both candidates on 60-92% of items. Conditional on landing on a candidate, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion. Most failures reflect candidate localization, not relation understanding.
Benchmark accuracy correlates with ScreenSpot-Pro grounding across ten models (Spearman ρ = +0.74, p = 0.015). Marking the two designated candidates raises selection accuracy by 35-57 percentage points: an oracle diagnostic that supplies the candidate set, not a deployable method.
| Primitive | Off-candidate | Target selection* |
|---|
*Forced-choice accuracy among predictions that land on a candidate. 0.50 is the two-candidate reference level, not the chance level of unconstrained clicking. Alignment is sensitive to the inclusion rule (0.728 strict, n=173; 0.529 permissive, n=1021) and we do not interpret it.
The benchmark draws from two sources: 290 real items mined from UI-Vision screenshots (VS Code, Office, GRASS GIS, file managers, video editors) and 704 synthetic items rendered with exact ground truth, the only way to obtain containment and occlusion cleanly.
On real screenshots, 96.2% of predictions land on neither candidate, pooled over all 19 models. Among the few clicks that do reach a candidate, the target is still chosen on 0.75 of them (0.91 for GPT-5 and Claude Opus 4.7). So the near-zero accuracy on real GUIs is a search failure, and the relation word still carries signal when search succeeds.
All results in the paper are reported separately by source. In 253 of 497 pairs the instruction template also varies lexically between twins (verb, head noun, anchor quoting); this variation is balanced across relation terms and cannot be exploited as a shortcut.
On 60-92% of items the click lands outside both candidates. The headline "below-chance" accuracy is off-candidate clicking, not a preference for the wrong element.
Even when a model commits to a candidate, containment and occlusion split at 0.548 and 0.554, indistinguishable from a coin flip between the two boxes.
Benchmark accuracy tracks downstream ScreenSpot-Pro grounding (ρ = +0.74 across ten models). We report this as an exploratory association at n = 10.
Oracle two-candidate Set-of-Mark lifts GPT-5 from 30% to 87% by handing the model both candidates. It measures discrimination given perfect localization, not a fix.
@inproceedings{jahin2026guiprimitives,
title = {{GUI-PRIMITIVES}: Diagnosing Spatial Reasoning Failures
in Vision-Language {GUI} Grounding},
author = {Jahin, Md Abrar and Parvez, Md Rizwan},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods
in Natural Language Processing (EMNLP)},
year = {2026}
}