How Microsoft taught neural networks to assemble 3D models in seconds

Think about how long it usually takes to create a quality 3D asset. Even if you don't model by hand but use first-generation neural networks, the process often feels like a lottery: textures will "drift," geometry turns into a mess, or the GPU chokes on memory. Microsoft rolled out TRELLIS.2, and this is probably one of the most sensible attempts to make generating 3D objects from a single image truly fast and high-quality.

What it is and why we need it

TRELLIS.2 is a large generative model with 4 billion parameters. To put it simply, it takes a regular 2D image and turns it into a full-fledged 3D object. But unlike many academic projects that output a "point cloud" or a broken mesh, here you get a render-ready asset with PBR materials (color, metallic, roughness).

What's interesting is that the developers abandoned the conventional Signed Distance Fields (SDF) in favor of a structure they called O-Voxel. This allowed the model to "understand" complex topologies: clothing, tree leaves, or objects with internal cavities that previous algorithms would choke on.

What sets TRELLIS.2 apart from the rest

I often see projects that promise "photorealism," but in reality they require an H100 and half an hour for a single chair. The situation here is different.

Speed and resolution

The model runs on a Sparse 3D VAE with 16x spatial compression. This makes it possible to generate objects at 512³ resolution in just 3 seconds. If you need something more serious, like 1024³, you'll have to wait about 17 seconds. For comparison: many alternatives spend minutes on this, and the result often looks worse.

Working with materials

This isn't just a colored figurine. The model generates full surface attributes:

• Base Color
• Roughness
• Metallic
• Opacity

The last point is especially nice. Now you can generate glass objects or semi-transparent fabrics, and they'll look decent when rendered.

Topology versatility

Thanks to O-Voxel, the model doesn't try to "patch" all the holes in an object. It easily handles open surfaces (like a thin sheet of paper or a dress hem) without corrupting geometry where surfaces don't touch.

The technical side of things

Under the hood, TRELLIS.2 runs on a DiT (Diffusion Transformer). To make all of this fly, the Microsoft team wrote several helper libraries that come bundled:

1. FlexGEMM — sparse convolution implementation on Triton.
2. CuMesh — CUDA utilities for fast mesh processing, UV unwrapping, and decimation.
3. O-Voxel — the core itself for working with their new data representation.

Interestingly, the conversion process from mesh to voxels and back takes mere milliseconds on GPU. This is critical if you plan to embed such a solution into a game dev pipeline.

How to run this yourself

I'll say upfront: the project is demanding. You'll need Linux and an NVIDIA GPU with at least 24 GB of VRAM (RTX 3090/4090 or server A100/H100). On Windows, you'll probably only get it running through WSL2, but the developers tested strictly on Linux.

Installation looks standard for projects like this, but requires compiling CUDA extensions:

git clone -b main https://github.com/microsoft/TRELLIS.2.git --recursive
cd TRELLIS.2
# Скрипт создаст окружение и поставит все зависимости, включая специфичные либы
. ./setup.sh --new-env --basic --flash-attn --nvdiffrast --nvdiffrec --cumesh --o-voxel --flexgemm

If everything went smoothly, you can run generation in just a few lines of code:

from trellis2.pipelines import Trellis2ImageTo3DPipeline
from PIL import Image

# Загружаем модель (веса подтянутся с Hugging Face)
pipeline = Trellis2ImageTo3DPipeline.from_pretrained("microsoft/TRELLIS.2-4B")
pipeline.cuda()

# Берем картинку и запускаем магию
image = Image.open("your_image.png")
mesh = pipeline.run(image)[0]

# Экспортируем в GLB с текстурами
# (в коде репозитория есть подробный пример в example.py)

Is it worth trying

The project looks like a solid tool for anyone who needs to quickly populate scenes with unique objects. Yes, the system requirements are steep, but 4 billion parameters and real PBR are worth it.

What I especially like is that Microsoft released not just weights and inference, but also the full training code. This means we'll soon see fine-tuned models for specific styles or object categories. If you work with 3D generation or just like poking around fresh SOTA models, TRELLIS.2 definitely deserves an evening of your time to study. Just make sure you have CUDA 12.4 installed, otherwise the setup process will turn into an exciting quest to fix compilation errors.