Inside the BrickGPT Paper: Text to Stable Brick Structures
Ava Pun and colleagues' BrickGPT generates brick assemblies from text. Its key idea is to represent a model as an ordered sequence of bricks, then check each proposed placement and repair unstable structures. This research project is separate from BrickGPT.online's image-to-visual-board workflow.
The problem is more than matching a prompt
A text-to-3D image can resemble a brick model while hiding collisions, unsupported parts or impossible connections. The paper targets assemblies that remain stable on a fixed baseplate within its defined grid and brick library. The authors demonstrate manual and robotic assembly, but do not claim that arbitrary prompts or real-world building conditions are solved.
The authors released the paper, model, dataset and code. Sharing the name BrickGPT does not mean this independent website runs their model.
Build stable text-to-brick training data
StableText2Brick contains more than 47,000 structures from over 28,000 distinct ShapeNetCore objects across 21 categories. The authors voxelize source meshes into a 20 x 20 x 20 grid, generate multiple layouts per object and retain structures that pass stability analysis. They render each assembly from 24 viewpoints and use GPT-4o to write geometry-focused descriptions without color.
Ordinary 3D datasets do not identify which individual brick placements are valid, stable and describable in text. This dataset remains bounded by its selected shapes, grid and brick types.
Predict a sequence of individual bricks
Each brick becomes a compact line such as 1x2 (16,18,0): footprint dimensions and grid position. Bricks are ordered in a raster scan from bottom upward. This plain-text representation exposes dimensions and coordinates to the validator without redundant 3D-format fields.
The team fine-tunes Llama-3.2-1B-Instruct on caption-and-brick-sequence instructions. At inference, the model proposes the next brick given the prompt and previously accepted bricks. Next-token prediction alone cannot guarantee valid geometry or equilibrium.
Reject invalid placements and roll back unstable builds
Fast checks reject malformed lines, unavailable brick dimensions, out-of-bounds placements and collisions. After generation, a force-based solver checks static equilibrium and connection stress. If a brick is unstable, the system removes it and subsequent bricks, then resumes from an earlier stable prefix.
The analysis models gravity, contact and connection forces, force and torque balance, and friction capacity. The official repository recommends Gurobi. Its optional connectivity-only fallback is simpler and less accurate: connectivity is not a physical stability proof.
- The fine-tuned model proposes bricks in sequence from a text prompt.
- Placement checks prune malformed, unavailable or colliding candidates.
- A full stability check can trigger rollback and regeneration.
What the experiments establish
The paper evaluates 250 held-out prompts against base language models and text-to-3D-to-brick pipelines. Ablations show why rejection sampling and physics-aware rollback matter. In that research setup, median generation time is 40.8 seconds and median rollbacks are two; these are not speed promises for other hardware or products.
The method assumes axis-aligned, one-unit-high bricks on a fixed baseplate and discretized grid. Texturing and per-brick color are separate appearance stages. A stable structure is different from an exact parts inventory or a complete instruction manual.
Original paper and code
Generating Physically Stable and Buildable Brick Structures from Text is by Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu and Jun-Yan Zhu (ICCV 2025). BrickGPT.online is an independent product that starts with a finished model image and creates a visual steps board and video.
BrickGPT Lab