AivexaNewsSearch
AI news for builders and product teamsChecked every hour

The Definitive Fine-Tuning Guide for 3D Generation Models

Collected Nov 12, 2026

Hugging Face published a fine-tuning guide for TRELLIS.2, a 3D generation model, accompanied by two Docker images and a companion repository. The containers can run either on Hugging Face Jobs cloud hardware or locally with Docker, and they handle both preprocessing and LoRA training. The walkthrough is built around a concrete experiment: adapting the pretrained model to the footwear category using a 188-object subset of the Google Scanned Objects (GSO) dataset, drawn from GSO's 1,030 total objects. The motivating failure was specific—the base model generally produced plausible shoes but got the soles wrong.

The pipeline and where adapters go

TRELLIS.2 generates an object through five stages: background removal, image encoding, coarse structure prediction, detailed geometry, and material generation, after which the result is exported as a textured mesh. BiRefNet strips the background, and DINOv3-L encodes the image into feature tokens that condition the later stages. The sparse-structure flow DiT predicts coarse shape as a small 16³ × 8 latent, which a VAE decoder turns into a 64 × 64 × 64 occupancy grid marking which voxels belong to the object. The shape flow DiT then combines that grid with the DINOv3 features to produce geometry at 512 × 512 × 512 resolution in the O-Voxel format. Finally the texture flow DiT predicts per-voxel base color, metallic, roughness and alpha, and the mesh is exported as GLB.

The authors add LoRA adapters only to the sparse-structure flow DiT and the shape flow DiT, since those control shape. The texture flow DiT is frozen: the dataset contains diffuse color maps with baked lighting rather than the separate material maps that stage needs, and training on them could teach the model to treat lighting as a surface property or cause catastrophic forgetting. In total the configuration trains roughly 9.3 million adapter parameters. The sparse-structure flow DiT gets rank-8 adapters in its attention and MLP layers, while the shape flow DiT gets rank-16 adapters in its attention layers. Rank controls adapter matrix size, and a higher rank lets the adapter learn more.

Why LoRA over full fine-tuning

With only 188 objects, full fine-tuning is expensive and risks overwriting the model's learned priors. LoRA trains small adapter matrices while the original weights stay frozen, cutting trainable parameters and the memory needed for gradients and optimizer state. Hugging Face's PEFT library supplies the adapters. Because TRELLIS.2 uses a custom forward pass, the standard PeftModel wrapper does not apply; the guide instead calls inject_adapter_in_model() to attach adapters directly while preserving the model's interface. Trainable weights are kept in fp32 for numerical stability, with bf16 autocasting in the forward pass to reduce memory use without changing how frozen backbone weights are stored.

Preprocessing and data preparation

The preprocessing container renders 16 views of each object in Blender running headless via xvfb-run, then passes each mesh through the pipeline using frozen TRELLIS.2 encoders to produce training latents that are saved for reuse. During training one random view per sample serves as the conditioning image. Nine of the 188 objects are held out for evaluation, leaving 179 for training, so the adapters can be checked on objects never seen during training. For the full 188-object run, preprocessing took about an hour on an a100-large job. Two commands are given for Jobs: one for a sparse-structure run (config "ss") and one for a shape run (config "shape"), each with a 12h timeout, with options to push checkpoints to a chosen Hub repo and to log metrics to a Trackio space. Training uses batch size 4 with 4 gradient accumulation steps, an effective batch size of 16, and an example setting of max_steps=3000 with i_save=500. The same flags apply under Docker; only the container invocation changes, adding --gpus all, a shared memory size of 16g, and mounted cache and work volumes. A preprocessed footwear dataset is published so people can train without running preprocessing themselves, and the trained adapters live in the suvadityamuk/trellis-2-finetune-gso repository.

Practical constraints and my reading

Two access requirements are worth noting before starting: a Hugging Face token, and access to DINOv3, which is a gated model on the Hub. Users bringing their own data must upload meshes to the Hub first. The design reflects a sensible general pattern for 3D pipelines—freeze the decoders and material stages, target the structure-predicting transformers, and let cheap adapters absorb category-specific corrections. In practice, that means fine-tuning a 3D generator now looks closer to fine-tuning an image model with LoRA than to retraining a whole pipeline, though the preprocessing step (rendering and latent encoding) is a real upfront cost that the saved-latent approach amortizes across runs. The guide does not present quantitative before-and-after metrics on sole accuracy in the supplied material, so whether the adapters actually fix the described failure is something a reader would need to verify against the released checkpoints and held-out split.

Why it matters: Teams building robotics, simulation or media tools often need generated 3D assets to follow category-specific rules, and this lowers the barrier to specializing a strong open 3D model without owning the full training stack. The container-based workflow and Hub integration mean a developer can move from a folder of meshes to a shareable LoRA adapter with a few commands, which my inference suggests will make custom 3D asset pipelines more practical for small teams. The frozen-material caveat also signals where the approach stops: categories whose problems are textural rather than geometric would need datasets with proper material maps and a different training plan.

Read at Hugging Face Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt