LEGO-Puzzles: Testing Multi-Step Spatial Reasoning in AI
LEGO-Puzzles is a benchmark, not a brick generator. It tests whether multimodal models understand views and rotations, infer the next assembly state and maintain a valid plan across several steps. Results here follow the updated arXiv paper; the authors' GitHub repository provides the evaluation code.
Separate elementary skill from full planning
The Elementary set has 1,100 visual question-answer pairs, 100 for each of 11 task types drawn from 407 assembly instructions. Level 1 checks height, adjacency, rotation and multiple views. Level 2 asks for one-step rotation status, position, next state and dependencies. Level 3 asks which intermediate state is correct, how images should be ordered and which state does not belong.
The Planning set then asks for an explicit assembly sequence. Choosing a plausible next image is easier than maintaining a consistent series of placements through to the target.
Increase the horizon and add plausible wrong steps
Plan-k-Step-VQA varies plan length from one to eight steps. It extends image ordering and adds distractors from backward-reasoning and outlier tasks, giving 2k choices for a k-step plan. Exact sequence match, set match ignoring order and overlap with correct steps distinguish wrong step selection from wrong ordering.
A second task generates each intermediate image. Evaluation enumerates valid assembly orders under part-dependency constraints, matches images against candidate states and accepts only a full legal trajectory. A visually attractive frame alone is insufficient.
The updated paper shows a steep multi-step gap
The updated paper evaluates 29 multimodal models on Elementary tasks. GPT-5 Plan-k-Step-VQA exact match falls from 90% at one step to 0% at eight; five expert human participants solve the tested planning questions perfectly. In the reported image-generation test, no evaluated model completes a correct three-step trajectory, even when some individual images look plausible.
These are measurements for selected tasks, models and prompts. The repository README still summarizes an earlier 20-model release; the linked updated arXiv paper is the source for the 29-model and long-horizon results.
Review visual build steps as a sequence
Check part count and orientation, whether dependencies are respected, and whether each image can follow the preceding one without pieces disappearing or moving. A polished final render cannot validate hidden joins or intermediate states.
BrickGPT.online creates AI visual boards and videos from a finished model image. The study does not test this website or certify its outputs; it explains why generated steps need inspection before being used as construction instructions.
Paper, code and dataset
LEGO-Puzzles is by Kexian Tang and collaborators. Their repository includes dataset access and a VLMEvalKit evaluation workflow. Check current repository instructions before reproducing scores, since the paper and README describe different release snapshots.
BrickGPT Lab