3D Generation Interview Cheat Sheet

§0 TL;DR Cheat Sheet

1. Three representations: NeRF (implicit neural field + volume rendering), 3DGS (explicit Gaussian point cloud + rasterization), Mesh / SDF (explicit surface / implicit distance field). Sweet spot for quality vs speed: 3DGS (Kerbl 2023 SIGGRAPH Best Paper).
2. NeRF core equation: $C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt$, where $T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)$ is the transmittance. Discretization yields $\alpha$-compositing: $C \approx \sum_i T_i (1-e^{-\sigma_i\delta_i})\mathbf{c}_i$.
3. Instant-NGP (Müller 2022 SIGGRAPH): multi-resolution hash grid + tiny MLP, roughly a 4+ order-of-magnitude speedup; hash collisions are automatically disambiguated by the MLP over colliding entries (suppressed jointly by the loss and multi-scale redundancy).
4. 3DGS core: scene represented as a set of 3D Gaussians $\{\mu_i, \Sigma_i, \alpha_i, c_i(\mathbf{d})\}$, differentiable rasterization projects the 3D covariance to 2D via Jacobian $J$: $\Sigma' = J W \Sigma W^\top J^\top$, then performs front-to-back alpha-blending after depth sorting.
5. DreamFusion SDS (Poole et al. 2022 arXiv → ICLR 2023 Outstanding): supervise a 3D representation using a pretrained 2D diffusion: $\nabla_\theta \mathcal{L}_\text{SDS} = \mathbb{E}_{t,\epsilon}[w(t)(\epsilon_\phi(x_t;y,t)-\epsilon)\,\partial x/\partial \theta]$, deliberately dropping the U-Net Jacobian of $x_t$ to make training simulation-free. Price: mode-seeking → over-saturation / Janus.
6. VSD (Wang 2023 NeurIPS, ProlificDreamer): treats the 3D parameters $\theta$ as a random variable $\mu(\theta)$ and minimizes the KL between the noised rendered image distributions: $\mathbb{E}_t\big[D_\text{KL}\big(q_\mu^t(x_t|y)\,\|\,p_\phi^t(x_t|y)\big)\big]$. The gradient form is a relative score $\nabla_\theta \approx (\epsilon_\phi(x_t;y,t) - \epsilon_\psi(x_t;y,t,\pi))\,\partial x/\partial\theta$, where $\epsilon_\psi$ is a LoRA-finetuned auxiliary score. CFG can drop from 100 to 7.5.
7. Single-image / Few-view 3D: Zero-1-to-3 (Liu 2023 ICCV) uses viewpoint-conditioned diffusion; SyncDreamer / MVDream learn joint multi-view consistency; TripoSR / InstantMesh / Stable Fast 3D push image-to-mesh to the seconds regime (TripoSR ~0.5 s, InstantMesh ~10 s).
8. Feed-forward reconstruction: DUSt3R (arXiv:2312.14132) replaces SfM with a single forward pass — it regresses pointmaps directly, both images' 3D points landing in the first image's camera frame, so pose becomes an output rather than an input; the loss divides by the mean distance of valid points to the origin, hence up-to-scale by default. VGGT (2503.11651, CVPR 2025 Best Paper) draws four heads (camera / depth / point map / track) off one backbone. COLMAP thereby drops from "mandatory" to "accuracy reference" (§7, §12.1).
9. 3D Foundation Models (2024-26 open source): TRELLIS's SLAT = sparse voxel coordinates + per-voxel latent ($N=64$, $L\approx 20\text{K}$ active voxels), two-stage rectified flow, one latent decoding into 3DGS / field / mesh; Hunyuan3D iterates 2.0 → 2.1 → 2.5 → Omni / Studio / Buffalo (there is no 3.0), shape→texture two-stage; CLAY (arXiv:2406.13897) is multi-resolution VAE + latent DiT.
10. Native mesh generation and the latent-representation axis: whether the mesh comes from isosurface extraction (marching cubes, FlexiCubes) or from a direct O-Voxel conversion as in TRELLIS.2, none of these outputs is guaranteed to have edge flow suited to editing, rigging and deformation — that, not raw quality, is why artists reject it. MeshGPT → MeshAnything V2 (AMT) → BPT → TreeMeshGPT compresses tokens against a baseline of 9 per face, while Meshtron instead answers with architecture (hourglass + sliding window). Latent representations (SLAT / sparse voxel / VecSet / triplane) can all be decoded by coordinate query; what they decide is not "how many output formats" but where compute and memory go.
11. Embodied AI key applications: Sim2Real asset generation, NeRF/3DGS as differentiable simulators, language-conditioned 3D affordance. Common interview crossovers: NeRF SLAM, Gaussian-Splat scene editing, 3D physics consistency.

§1 Intuitive comparison of the three representations

The first multiple-choice question in 3D generation is representation — pick the wrong one and the entire downstream pipeline is wasted.

§2 NeRF: derivation of volume rendering (must-know)

2.1　Continuous volume-rendering equation

NeRF (Mildenhall 2020 ECCV Best Paper Honorable Mention) represents a scene as a 5D neural field $f_\theta : (\mathbf{x}, \mathbf{d}) \to (\sigma, \mathbf{c})$:

• Input: 3D position $\mathbf{x} \in \mathbb{R}^3$ + view direction $\mathbf{d} \in \mathbb{S}^2$
• Output: volume density $\sigma \ge 0$ (direction-independent) + color $\mathbf{c} \in \mathbb{R}^3$ (direction-dependent, captures specular reflection)

For camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, integrate along $t \in [t_n, t_f]$ to get pixel color:

$$\boxed{\;C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt\;}$$

where transmittance (the probability that the ray has not been blocked between $t_n$ and $t$) is

$$\boxed{\;T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)\;}$$

2.2　Why this form? — derivation from physics

Consider a light ray traveling through a participating medium. Over $[t, t+dt]$:

• Probability of absorption / scattering out of the ray: $\sigma(\mathbf{r}(t))\,dt$
• Color contribution emitted by the medium at this point: $\mathbf{c}(\mathbf{r}(t),\mathbf{d})$

Let $T(t)$ be the survival probability of the ray from $t_n$ to $t$. From $t \to t + dt$, the survival probability changes as

$$T(t+dt) = T(t)\,(1 - \sigma\,dt) \;\Rightarrow\; \frac{dT}{dt} = -\sigma(t)\,T(t)$$

This is a first-order ODE with initial value $T(t_n) = 1$, solving to

$$T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)$$

The color contribution to the pixel at each depth $t$ = survival probability × absorption probability × local color:

$$dC = T(t)\,\sigma(t)\,\mathbf{c}(t)\,dt$$

Integrating gives $C(\mathbf{r})$.

2.3　Discretization: $\alpha$-compositing (must-derive)

Continuous integration is impossible in practice. Slice $[t_n, t_f]$ into $N$ segments with $\delta_i = t_{i+1} - t_i$, and assume $\sigma, \mathbf{c}$ are constants $\sigma_i, \mathbf{c}_i$ within each segment.

Within-segment transmittance decay: on $[t_i, t_{i+1}]$, $T$ satisfies $dT/dt = -\sigma_i T$, so

$$\frac{T(t_{i+1})}{T(t_i)} = e^{-\sigma_i \delta_i}$$

This gives inter-segment transmittance:

$$T_i := T(t_i) = \prod_{j=1}^{i-1} e^{-\sigma_j \delta_j} = \exp\!\Big(\!-\!\sum_{j=1}^{i-1}\sigma_j\delta_j\Big)$$

Within-segment color contribution (integral, not a simple rectangle):

$$\int_{t_i}^{t_{i+1}} T(t)\sigma_i\mathbf{c}_i\,dt = T_i\,\mathbf{c}_i \int_0^{\delta_i} \sigma_i e^{-\sigma_i s}\,ds = T_i\,\mathbf{c}_i\,(1 - e^{-\sigma_i\delta_i})$$

Let $\alpha_i := 1 - e^{-\sigma_i \delta_i}$ (segment opacity). Combining yields the discrete NeRF equation:

$$\boxed{\;C(\mathbf{r}) \approx \sum_{i=1}^{N} T_i\,\alpha_i\,\mathbf{c}_i,\quad T_i = \prod_{j<i}(1-\alpha_j),\quad \alpha_i = 1 - e^{-\sigma_i\delta_i}\;}$$

This is exactly the graphics front-to-back alpha-compositing formula. Key point: $\alpha_i = 1 - e^{-\sigma_i\delta_i}$, not $\sigma_i\delta_i$; they are approximately equal when $\sigma_i\delta_i$ is small (first-order Taylor), but differ noticeably when large (saturating to 1 vs diverging linearly).

2.4　Positional encoding $\gamma(p)$: representing high-frequency detail

MLPs default to a low-frequency bias (NTK analysis); fitting $f(\mathbf{x})$ directly produces blur. NeRF uses positional encoding to lift the frequency:

$$\gamma(p) = \big(\sin(2^0\pi p),\cos(2^0\pi p),\sin(2^1\pi p),\cos(2^1\pi p),\dots,\sin(2^{L-1}\pi p),\cos(2^{L-1}\pi p)\big)$$

Use $L=10$ for $\mathbf{x}$ (60 dims) and $L=4$ for $\mathbf{d}$ (24 dims). Tancik et al. 2020 "Fourier Features" later gave an NTK explanation: high-frequency $\sin/\cos$ slow the kernel decay, allowing the MLP to learn high frequencies.

2.5　Hierarchical sampling: coarse → fine

• Coarse network: uniformly sample $N_c = 64$ points, render to get weights $w_i = T_i \alpha_i$
• Fine network: normalize $w$ to a PDF and importance-sample $N_f = 128$ new points (importance sampling: regions near the surface have larger weights and should be densely sampled)
• Composite the final color with all $N_c + N_f$ coarse + fine points
• Loss: $\mathcal{L} = \|C_c - C_\text{gt}\|^2 + \|C_f - C_\text{gt}\|^2$ (supervise both networks; the coarse one provides a well-defined gradient for the sampler)

2.6　NeRF training code (core 30 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

def positional_encoding(x: torch.Tensor, L: int) -> torch.Tensor:
    """ x: [..., D]; returns [..., D*2*L]; NeRF γ(p) does not include the raw x """
    freqs = 2.0 ** torch.arange(L, device=x.device, dtype=x.dtype) * torch.pi
    args = x.unsqueeze(-1) * freqs                # [..., D, L]
    pe = torch.stack([torch.sin(args), torch.cos(args)], dim=-1)  # [..., D, L, 2]
    return pe.flatten(-3)                          # [..., D*2*L]

def volume_render(sigma: torch.Tensor, color: torch.Tensor, t_vals: torch.Tensor,
                  ray_d: torch.Tensor):
    """
    NeRF discrete α-compositing
        sigma:   [B, N]        volume density (>= 0; usually after softplus / ReLU)
        color:   [B, N, 3]     color
        t_vals:  [B, N]        t values of sampled points along the ray (monotonically increasing)
        ray_d:   [B, 3]        ray direction (used to convert Δt → true distance)
    Returns C: [B, 3], weights: [B, N], depth: [B]
    """
    # δ_i = t_{i+1} - t_i; pad the last segment with 1e10 (absorbs the remainder out to infinity)
    deltas = t_vals[..., 1:] - t_vals[..., :-1]
    delta_far = torch.full_like(deltas[..., :1], 1e10)
    deltas = torch.cat([deltas, delta_far], dim=-1)            # [B, N]
    deltas = deltas * torch.norm(ray_d[:, None, :], dim=-1)    # convert t-spacing to real Euclidean distance

    alpha = 1.0 - torch.exp(-sigma * deltas)                   # [B, N]
    # T_i = ∏_{j<i} (1 - α_j) — use cumprod, shifted by one so T_1 = 1
    T = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]),
                                 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]
    weights = T * alpha                                        # [B, N]
    C = (weights[..., None] * color).sum(dim=-2)               # [B, 3]
    depth = (weights * t_vals).sum(dim=-1)                     # [B]
    return C, weights, depth

2.7　Mip-NeRF / Mip-NeRF 360 (anti-aliasing)

Vanilla NeRF aliases badly at low resolution / when zoomed (the same pixel corresponds to cones of different scales, but NeRF treats them as rays).

• Mip-NeRF (Barron 2021 ICCV): treats the ray as a cone (view frustum), uses Integrated Positional Encoding (IPE) — closed-form Gaussian expectation of PE over the cone segment $\mathbb{E}_{\mathbf{x}\sim\mathcal{N}(\mu,\Sigma)}[\gamma(\mathbf{x})]$. For frequency vector $\boldsymbol{\omega} = 2^k\pi\,\mathbf{e}$, $\mathbb{E}_{\mathbf{x}\sim\mathcal{N}(\boldsymbol{\mu},\Sigma)}[\sin(\boldsymbol{\omega}^\top\mathbf{x})] = \sin(\boldsymbol{\omega}^\top\boldsymbol{\mu})\,e^{-\frac{1}{2}\boldsymbol{\omega}^\top\Sigma\,\boldsymbol{\omega}}$ (the scalar case is $e^{-\frac{1}{2}\omega^2\sigma^2}$); high-frequency coefficients are automatically attenuated by the cone covariance $\Sigma$ via $e^{-\frac{1}{2}\boldsymbol{\omega}^\top\Sigma\boldsymbol{\omega}}$, naturally achieving multi-scale behavior.
• Mip-NeRF 360 (Barron 2022 CVPR): for unbounded scenes, applies contraction $f(x) = (2 - 1/\|x\|)\,x/\|x\|$ for $\|x\| > 1$, squashing infinity into a ball; adds distortion / proposal MLP losses.

2.8　NeuS / VolSDF: volume rendering + SDF (key to mesh extraction)

NeRF is density-based; mesh extraction requires choosing a $\sigma$ threshold (unstable). NeuS (Wang 2021 NeurIPS) replaces density with SDF $d(\mathbf{x})$:

$$\sigma(t) = \max\!\left(\frac{-\frac{d}{dt}\Phi_s(d(\mathbf{r}(t)))}{\Phi_s(d(\mathbf{r}(t)))},\; 0\right),\quad \Phi_s(d) = (1 + e^{-sd})^{-1}$$

where $\Phi_s$ is the sigmoid and $s$ is a learnable "sharpness". Properties: the weight peaks at the surface ($d=0$); Marching Cubes can directly extract the mesh (the mesh is $\{d = 0\}$).

VolSDF (Yariv 2021 NeurIPS) uses Laplace CDF $\sigma = \alpha\,\Phi(-d/\beta)$, with a similar idea.

§3 Instant-NGP: roughly 4+ orders of magnitude faster (must-know)

Training a vanilla NeRF on one scene takes 1-2 days. Instant-NGP (Müller 2022 SIGGRAPH Best Paper) fits a simple scene in 5 seconds.

3.1　Core idea: multi-resolution hash grid

Replace "dense grid vs big MLP" with "sparse hash grid + tiny MLP".

• $L$ resolution levels (e.g. $L = 16$); the $\ell$-th level has $N_\ell = \lfloor N_\min \cdot b^\ell \rfloor$ grid points, geometric progression ($b \approx 1.38$; typical paper values $N_\min = 16$, $N_\max \in [512, 2048]$ depending on scene size)
• Each level uses a hash function to map grid-point coordinates into a fixed-size feature table ($T = 2^{14}$–$2^{24}$, typically $T = 2^{19} = 524288$)
• Query point $\mathbf{x}$: trilinear-interpolate at the 8 corner points per level → concatenate to a $L \times F$-dim feature ($F = 2$)
• Feed into a tiny MLP (2 layers, hidden 64) to output $\sigma, \mathbf{c}$

3.2　Hash function

$$\text{hash}(\mathbf{x}) = \bigg(\bigoplus_{i=1}^{d} x_i \cdot \pi_i\bigg) \bmod T$$

$\pi_i$ are large coprime constants ($\pi_1 = 1, \pi_2 = 2654435761, \pi_3 = 805459861$ — note $\pi_1 = 1$ is not prime; it is set deliberately in the paper so that the first dimension maps directly without scrambling). $\oplus$ is XOR. This is a spatial hash, commonly used in physics-simulation BVHs.

3.3　How are hash collisions disambiguated? (L3 high-frequency follow-up)

When $N_\ell^d > T$ (inevitable at fine levels), multiple grid points map to the same entry → collision. Why does it still work?

1. Multi-resolution redundancy: features at coarse levels are unique ($N_\ell^d \le T$); fine levels add detail. The MLP can recover structure from coarse features, with fine levels only responsible for detail.
2. Sparsity prevails: most of space is empty (most voxels in a NeRF scene are background), and meaningful queries concentrate near surfaces, so "valid colliding grid-point pairs" are rare.
3. Gradients auto-disambiguate: early in training, sample points in empty regions with non-zero density still receive a gradient through the volume-rendering photometric loss (gradient magnitude $\propto \delta_i e^{-\sigma_i\delta_i}$, pushing $\sigma$ toward 0); as $\sigma\to 0$ that gradient magnitude naturally decays, and once the occupancy grid updates it skips sampling there entirely — so it's not that empty-region entries "never get a gradient signal" from the start.
4. MLP post-processing: the tiny MLP learns a classification/regression on the $L \times F$ concatenated features; surface points with collisions can be disambiguated via non-colliding features from other levels.

3.4　Instant-NGP training equations

Parameters: hash table $\theta_\text{hash} \in \mathbb{R}^{L \times T \times F}$ + MLP weights $\theta_\text{MLP}$. The loss is still photometric MSE, but the 5-seconds-vs-1-day gap comes from:

• Tiny MLP: 100× fewer parameters, ~50× faster forward
• Hash table: sparse activations, cache-friendly
• Fused CUDA kernels: tiny-cuda-nn fuses forward + backward
• Occupancy grid: a coarse occupancy grid skips sampling in empty regions, avoiding wasted queries

3.5　Plenoxels / TensoRF (contemporary explicit methods)

Plenoxels (Fridovich-Keil 2022 CVPR): pure voxel grid + spherical harmonics SH coefficients + density, no MLP at all, directly gradient-descend on voxels; speed similar to Instant-NGP but heavy on VRAM. TensoRF (Chen 2022 ECCV): compresses the 3D grid field via VM / CP decomposition, reducing parameters from $O(N^3)$ to $O(N)$ for CP or $O(N^2)$ for VM.

§4 3D Gaussian Splatting: explicit differentiable rasterization (current workhorse)

3DGS (Kerbl 2023 SIGGRAPH Best Paper) addressed NeRF's two big pain points: slow rendering and hard editing.

4.1　Scene representation

Scene = a set of 3D Gaussians $\{G_i\}$, each:

• Mean $\mu_i \in \mathbb{R}^3$ (position)
• Covariance $\Sigma_i \in \mathbb{R}^{3\times 3}$ (shape), decomposed as $\Sigma = R S S^\top R^\top$ (rotation $R$ + diagonal scaling $S$)
• Opacity $\alpha_i \in [0, 1]$
• Color $c_i(\mathbf{d})$ via SH coefficients ($\ell = 3$, 16 coefficients per channel, 48 parameters total)

Why $R S S R^\top$ and not learn $\Sigma$ directly? — to ensure $\Sigma$ is positive definite. Learning $\Sigma$ as a raw matrix under gradient descent escapes the positive-semidefinite cone; the decomposition only requires $R$ to be orthogonal (parameterized via quaternion $q$) and $S$ to be positive (parameterized via $\exp(s)$), so it is satisfied automatically.

4.2　3D → 2D projection Jacobian (L3 must-derive)

To splat a 3D Gaussian to the screen for rasterization, the 3D covariance $\Sigma$ must be projected to a 2D covariance $\Sigma'$.

Step 1: World → Camera: rigid transform $W \in SE(3)$. $\Sigma_\text{cam} = W \Sigma W^\top$ (where $W$ takes the rotation part; translation does not affect covariance).

Step 2: Camera → Screen: perspective projection is nonlinear:

$$\pi(\mathbf{x}) = \begin{pmatrix} f_x\,x/z \\ f_y\,y/z \end{pmatrix}$$

The covariance of a nonlinear map is approximated via first-order Taylor. Compute the Jacobian at the mean $\mu_\text{cam} = (x, y, z)$:

$$J = \frac{\partial \pi}{\partial \mathbf{x}}\bigg|_{\mu_\text{cam}} = \begin{pmatrix}\dfrac{f_x}{z} & 0 & -\dfrac{f_x\,x}{z^2}\\[2pt] 0 & \dfrac{f_y}{z} & -\dfrac{f_y\,y}{z^2}\end{pmatrix} \in \mathbb{R}^{2\times 3}$$

Step 3: 2D covariance (core formula):

$$\boxed{\;\Sigma' = J\,W\,\Sigma\,W^\top\,J^\top \in \mathbb{R}^{2\times 2}\;}$$

Derivation: if $\mathbf{x} \sim \mathcal{N}(\mu, \Sigma)$, then first-order $\pi(\mathbf{x}) \approx \pi(\mu) + J(\mathbf{x} - \mu)$, so $\text{Cov}[\pi(\mathbf{x})] \approx J\,\Sigma_\text{cam}\,J^\top = J W \Sigma W^\top J^\top$. This is the classic corollary of EWA splatting (Zwicker 2001).

4.3　Differentiable rasterization: tile-based front-to-back alpha-blending

The color at pixel $\mathbf{p}$:

$$C(\mathbf{p}) = \sum_{i \in \mathcal{N}(\mathbf{p})}\,c_i\,\alpha_i\,G_i'(\mathbf{p}) \prod_{j < i}\big(1 - \alpha_j\,G_j'(\mathbf{p})\big)$$

where $G_i'(\mathbf{p}) = \exp\!\big(-\tfrac{1}{2}(\mathbf{p} - \mu_i')^\top \Sigma_i'^{-1} (\mathbf{p} - \mu_i')\big)$ is the value of the 2D Gaussian at the pixel, and $\mathcal{N}(\mathbf{p})$ are the Gaussians covering $\mathbf{p}$ sorted by depth.

Key engineering:

1. Tile partitioning: split the screen into $16\times 16$ tiles, sort Gaussians by depth within each tile, render in parallel
2. GPU sort: radix sort, with composite key (tile_id, depth)
3. Front-to-back early stop: exit when accumulated $\prod(1 - \alpha G') < 10^{-4}$
4. CUDA kernel: the authors release diff-gaussian-rasterization; both forward and backward are manual derivatives

4.4　3DGS forward pass (PyTorch reference implementation)

def quat_to_rot(q: torch.Tensor) -> torch.Tensor:
    """ q: [N, 4] (w, x, y, z) already normalized; returns R: [N, 3, 3] """
    w, x, y, z = q.unbind(-1)
    R = torch.stack([
        1 - 2*(y*y + z*z),   2*(x*y - w*z),     2*(x*z + w*y),
        2*(x*y + w*z),       1 - 2*(x*x + z*z), 2*(y*z - w*x),
        2*(x*z - w*y),       2*(y*z + w*x),     1 - 2*(x*x + y*y),
    ], dim=-1).reshape(-1, 3, 3)
    return R

def gaussian_splat_forward(
    means3D: torch.Tensor,        # [N, 3]  Gaussian centers (world)
    scales: torch.Tensor,          # [N, 3]  log-scale (take exp for true scale)
    quats: torch.Tensor,           # [N, 4]  quaternion (will be normalized)
    opacities: torch.Tensor,       # [N, 1]  σ(logit) → α
    colors: torch.Tensor,          # [N, 3]  (simplified to RGB here, no SH expansion)
    viewmat: torch.Tensor,         # [4, 4]  world→camera
    K: torch.Tensor,               # [3, 3]  intrinsics (fx, fy, cx, cy)
    H: int, W: int,
):
    """ Pedagogical forward: no tile sort / CUDA, just shows the math.
        Real production uses gsplat / diff-gaussian-rasterization. """
    N = means3D.shape[0]
    device = means3D.device

    # --- 1. World → Camera ---
    homo = torch.cat([means3D, torch.ones(N, 1, device=device)], dim=-1)
    mu_cam = (homo @ viewmat.T)[:, :3]                          # [N, 3]
    z = mu_cam[:, 2].clamp(min=1e-4)                            # guard against div-by-zero

    # --- 2. Covariance (3D) ---
    q = quats / quats.norm(dim=-1, keepdim=True)
    R = quat_to_rot(q)                                          # [N, 3, 3]
    S = torch.diag_embed(torch.exp(scales))                     # [N, 3, 3]
    cov3D = R @ S @ S.transpose(-1, -2) @ R.transpose(-1, -2)   # [N, 3, 3]

    # Apply the World→Cam rotation part W_rot (3x3) to the covariance
    W_rot = viewmat[:3, :3]
    cov_cam = W_rot @ cov3D @ W_rot.T                           # [N, 3, 3]

    # --- 3. Projection Jacobian J (2x3) ---
    fx, fy = K[0, 0], K[1, 1]
    x_c, y_c, z_c = mu_cam[:, 0], mu_cam[:, 1], z
    J = torch.zeros(N, 2, 3, device=device)
    J[:, 0, 0] = fx / z_c
    J[:, 0, 2] = -fx * x_c / z_c**2
    J[:, 1, 1] = fy / z_c
    J[:, 1, 2] = -fy * y_c / z_c**2

    # --- 4. 2D covariance Σ' = J W Σ W^T J^T ---
    cov2D = J @ cov_cam @ J.transpose(-1, -2)                   # [N, 2, 2]
    cov2D = cov2D + 0.3 * torch.eye(2, device=device)           # low-pass filter (anti-aliasing)

    # --- 5. 2D center (pixel coordinates) ---
    cx, cy = K[0, 2], K[1, 2]
    mu2D = torch.stack([fx * x_c / z_c + cx, fy * y_c / z_c + cy], dim=-1)  # [N, 2]

    # --- 6. Depth sort (front to back) ---
    depth = z_c
    order = depth.argsort()                                     # ascending z
    mu2D, cov2D = mu2D[order], cov2D[order]
    colors_o = colors[order]
    alphas = torch.sigmoid(opacities[order]).squeeze(-1)        # [N]

    # --- 7. Pixel loop (pedagogical full-image loop; real impl uses tile + CUDA) ---
    yy, xx = torch.meshgrid(torch.arange(H, device=device),
                            torch.arange(W, device=device), indexing='ij')
    pix = torch.stack([xx, yy], dim=-1).float()                 # [H, W, 2]

    img = torch.zeros(H, W, 3, device=device)
    T_acc = torch.ones(H, W, device=device)
    inv_cov = torch.linalg.inv(cov2D)                           # [N, 2, 2]

    for i in range(mu2D.shape[0]):
        diff = pix - mu2D[i]                                    # [H, W, 2]
        # 2D Gaussian: exp(-0.5 (diff)^T Σ^-1 (diff))
        G = torch.exp(-0.5 * (diff @ inv_cov[i] * diff).sum(-1))  # [H, W]
        contrib = alphas[i] * G                                 # [H, W]
        contrib = contrib.clamp(max=0.99)                       # numerical safety
        img = img + (T_acc * contrib).unsqueeze(-1) * colors_o[i]
        T_acc = T_acc * (1 - contrib)
        if (T_acc < 1e-4).all():                                # early stop
            break

    return img

4.5　Adaptive density control (frequent interview topic)

3DGS initializes from sparse SfM (COLMAP) point clouds, but training must "spread" more Gaussians.

Heuristic conditions: gradient norm > τ_pos (e.g. $2 \times 10^{-4}$), scale > τ_scale (1% of scene scale).

def densify_and_prune(gaussians, grad_thresh=2e-4, scale_thresh=0.01,
                      max_screen_size=None):
    """ Pedagogical densify decisions (simplified; real gsplat also has screen-size triggers).
        Assume gaussians exposes the following 1D fields (N = current Gaussian count):
          xyz_grad_accum: [N]  ‖accumulated xyz gradient norm‖
          denom:          [N]  accumulation count (avoid /0)
          scales:         [N, 3]  log-scale
          opacities:      [N]  ∈ (0, 1) after sigmoid
          screen_size:    [N]  last-render screen-projected size (optional)
    """
    grad_norm  = gaussians.xyz_grad_accum / gaussians.denom.clamp(min=1)      # [N]
    mean_scale = gaussians.scales.exp().max(dim=-1).values                    # [N]

    # ⚠️ Compute BOTH the clone / split masks on the original N Gaussians first,
    #    before any append/delete — otherwise once clone_at appends and the array
    #    grows, the N-length split_mask is misaligned with the new array
    #    (mask-length mismatch / acting on the wrong objects).
    clone_mask = (grad_norm > grad_thresh) & (mean_scale <= scale_thresh)     # [N]
    split_mask = (grad_norm > grad_thresh) & (mean_scale >  scale_thresh)     # [N]

    # CLONE: high gradient + small scale — duplicate in place (new_xyz = original xyz, no
    # positional offset); keep original (append at the end), and the two overlapping
    # Gaussians separate naturally through subsequent independent gradient descent
    gaussians.clone_at(clone_mask)

    # SPLIT: high gradient + large scale — this is the step that actually samples a new
    # position: draw via torch.normal with the original Gaussian's scale as the std +
    # rotation, then divide scale by 1.6, splitting into 2 children, removing original at the end
    # split_at applies explicitly to the original N only: pass original_n, do not rely on implicit alignment with the post-append array length
    gaussians.split_at(split_mask, n=2, scale_div=1.6, original_n=split_mask.shape[0])

    # PRUNE: low opacity / too large on screen / already marked by split
    # ⚠️ Note: new Gaussians from clone are appended to the end; length has changed. The mask only applies to the original N.
    prune_mask = (gaussians.opacities[:split_mask.shape[0]] < 0.005) | split_mask
    if max_screen_size is not None:
        prune_mask = prune_mask | (gaussians.screen_size[:split_mask.shape[0]] > max_screen_size)
    gaussians.remove_original(prune_mask)   # delete only the originals flagged among the first N

    gaussians.reset_grad_accum()
    return gaussians

4.6　2DGS / Surfels (surface-aligned)

3DGS ellipsoids are not surface-aware; mesh extraction needs SuGaR / GSDF post-processing. 2DGS (Huang 2024 SIGGRAPH) degenerates the 3D ellipsoid into a 2D disk (one axis = 0), directly aligning with the surface, which works better with normal/depth supervision and produces clearly higher-quality meshes.

4.7　Dynamic 4DGS

Dynamic 3DGS (Luiten 2024 3DV, arXiv:2308.09713) uses a set of persistent Gaussians: color / opacity / size stay fixed across frames, only the position $\mu(t)$ and rotation move over time, with a local-rigidity regularizer constraining neighborhood rigidity; 4DGS (Wu 2024 CVPR / Yang 2024 ICLR) writes $\mu(t), \Sigma(t)$ as functions of time (MLP or spline); SC-GS (Huang 2024) drives dense Gaussians from sparse control points (analogous to LBS).

§5 Mesh extraction: Marching Cubes / DMTet

After NeRF / 3DGS reconstruction, downstream (simulator, AR, 3D printing) often needs a mesh.

5.1　Marching Cubes (classic must-know)

Input: 3D scalar field $f(\mathbf{x})$ (density / SDF) + threshold $\tau$. Output: triangle mesh of the level set $\{f = \tau\}$.

Algorithm skeleton:

1. Voxelize space (8 corners per voxel)
2. For each voxel, binarize the 8 corners ($f > \tau$ as 1, else 0) → 256 possible configurations
3. Look up table: each configuration has predefined isosurface triangle patches + vertex positions on edges
4. Linear interpolation for precise vertices: between edge endpoints $\mathbf{a}, \mathbf{b}$, interpolate $t = (\tau - f(\mathbf{a})) / (f(\mathbf{b}) - f(\mathbf{a}))$, vertex $= \mathbf{a} + t(\mathbf{b} - \mathbf{a})$
5. Merge all voxel triangles → full mesh

def marching_cubes_sketch(density: torch.Tensor, threshold: float):
    """ Real implementations use mcubes / scikit-image / pytorch3d; this shows the idea """
    from skimage.measure import marching_cubes
    # density: [Nx, Ny, Nz]; detach() cuts the autograd graph, cpu().numpy() moves to host
    verts, faces, normals, _ = marching_cubes(
        density.detach().cpu().numpy(),
        level=threshold,
        spacing=(1.0, 1.0, 1.0),
        gradient_direction='descent',  # normal direction; descent = surface faces low density
    )
    return verts, faces, normals

5.2　Differentiable: DMTet / FlexiCubes (end-to-end mesh learning)

Marching Cubes is not differentiable (the lookup table is discrete).

• DMTet (Shen 2021 NeurIPS / Munkberg 2022 CVPR): uses a deformable tetrahedral grid, each tetrahedron has 4 vertex SDFs + position offsets that are differentiable. Marching Tetrahedra replaces MC; topology is determined by SDF signs and geometry by vertex positions: within a region of fixed topology, geometry (vertex positions) is differentiable with respect to the SDF values and offsets; but at the instant a vertex SDF crosses zero and triggers a switch in that tetrahedron's look-up case, the topology itself is not differentiable (a measure-zero set, akin to the kink in ReLU at 0) — it should not be described unconditionally as "fully differentiable."
• FlexiCubes (Shen 2023 SIGGRAPH): generalizes dual marching cubes by introducing extra learnable parameters (dual vertex offsets / interpolation weights) to fix quality artifacts.

Typical use: Magic3D and Fantasia3D after DreamFusion use DMTet to learn meshes + textures under SDS supervision.

§6 SDS Loss: supervising 3D with 2D diffusion (DreamFusion family)

6.1　Problem setup

We want to generate 3D assets but have no 3D training data — 3D data is scarce (ShapeNet ~50k objects, Objaverse-XL 10M but uneven quality). Pretrained 2D diffusion (Stable Diffusion, Imagen) is abundant. Can we use 2D diffusion as a teacher to supervise 3D?

DreamFusion (Poole et al. 2022 arXiv → ICLR 2023 Outstanding Paper) proposed Score Distillation Sampling (SDS).

6.2　Setup

• 3D representation $\theta$ (NeRF parameters / DMTet vertices / 3DGS point cloud)
• Differentiable renderer $g(\theta, \pi) \to x$, where $x$ is an image ($\pi$ is the camera viewpoint)
• Pretrained 2D diffusion $\epsilon_\phi(x_t; y, t)$ ($y$ is the text prompt)

Goal: make $g(\theta, \pi)$ look like a "photo of $y$", i.e. $g(\theta, \pi)$ lies on the data manifold learned by the diffusion model.

6.3　SDS gradient derivation (L3 must-know)

Intuition: backprop the diffusion training loss into $\theta$. Naive idea: treat the rendered image $x = g(\theta, \pi)$ as a training sample and minimize

$$\mathcal{L}_\text{diff}(\theta) = \mathbb{E}_{t, \epsilon}\Big[w(t)\big\|\epsilon_\phi(x_t; y, t) - \epsilon\big\|^2\Big],\quad x_t = \alpha_t x + \sigma_t \epsilon$$

Take the gradient w.r.t. $\theta$ (chain rule):

$$\nabla_\theta \mathcal{L}_\text{diff} = \mathbb{E}\Big[w(t)\,2\big(\epsilon_\phi(x_t; y, t) - \epsilon\big)\,\underbrace{\frac{\partial \epsilon_\phi(x_t;y,t)}{\partial x_t}}_{\text{U-Net Jacobian}}\,\alpha_t\,\underbrace{\frac{\partial x}{\partial \theta}}_{\text{renderer Jacobian}}\Big]$$

Problem: the U-Net Jacobian $\partial \epsilon_\phi / \partial x_t$ is expensive to compute and numerically poor (the diffusion model is large and not trained for second-order stability).

SDS trick: drop the U-Net Jacobian entirely, giving

$$\boxed{\;\nabla_\theta \mathcal{L}_\text{SDS} \;=\; \mathbb{E}_{t, \epsilon}\Big[w(t)\,\big(\epsilon_\phi(x_t; y, t) - \epsilon\big)\,\frac{\partial x}{\partial \theta}\Big]\;}$$

(The original DreamFusion paper writes it as $\partial L/\partial \theta$; $\alpha_t$ and the constant 2 are absorbed into $w(t)$.)

6.4　Why does dropping the Jacobian still work?

First explanation (original DreamFusion, score view): $\epsilon_\phi(x_t; y, t)/\sigma_t \approx -\nabla_{x_t}\log p_\phi(x_t|y)$ (score). The SDS gradient = (predicted score - noise) × renderer Jacobian, i.e. it pushes the rendered image toward high-probability regions.

Second explanation (mode-seeking): SDS is equivalent to a mode-seeking form of $\mathbb{E}_t[D_\text{KL}(q(x_t|\theta) \,\|\, p_\phi(x_t|y))]$: drive toward high-probability regions of $p_\phi(\cdot|y)$.

6.5　SDS side effects: over-saturation / mode collapse / Janus

• Over-saturation: oversaturated colors, excessive contrast ("plastic-y look")
• Over-smoothing: blurred details
• Mode collapse: objects converge to "canonical" single forms
• Janus problem: 3D objects grow a "front face" in every view (faces on the back of heads; animals with heads on both sides)

Root cause: SDS is equivalent to a mode-seeking KL, and only large CFG (DreamFusion default 100) can escape the mean-mode. CFG=100 sharpens the distribution to the extreme → over-saturation.

6.6　SDS code (core 30 lines)

def sds_loss(
    renderer,                 # θ → x (B, 3, H, W)
    theta,                    # 3D parameters (NeRF / 3DGS / DMTet)
    prompt_emb,               # text embedding (cond) [B, L, D]
    uncond_emb,               # text embedding (uncond / null) [B, L, D]
    unet,                     # frozen 2D diffusion U-Net (e.g. SD); returns noise pred Tensor
    alpha_cumprod: torch.Tensor,  # [T_max] precomputed bar-alpha schedule
    cfg_scale: float = 100.0,
    t_range: tuple = (0.02, 0.98),
):
    """ Score Distillation Sampling loss (DreamFusion).
        Convention: unet(x_t, t, encoder_hidden_states=emb) -> [B, 3, H, W] noise pred.
        If using diffusers UNet2DConditionModel, wrap to take .sample.
        Returns a grad surrogate, can be backward'd directly. """
    x = renderer(theta)                                  # [B, 3, H, W]
    B = x.shape[0]
    device, dtype = x.device, x.dtype
    T_max = alpha_cumprod.shape[0]                       # usually 1000

    # 1. Sample t and noise, forward add noise
    t = torch.randint(int(t_range[0] * T_max), int(t_range[1] * T_max),
                      (B,), device=device)
    noise = torch.randn_like(x)
    abar = alpha_cumprod.to(device=device, dtype=dtype)[t].view(B, 1, 1, 1)
    x_t = abar.sqrt() * x + (1 - abar).sqrt() * noise

    # 2. U-Net predicts noise (cond / uncond), CFG combination; key: no autograd through diffusion
    with torch.no_grad():
        eps_uncond = unet(x_t, t, encoder_hidden_states=uncond_emb)
        eps_cond   = unet(x_t, t, encoder_hidden_states=prompt_emb)
        eps_pred = eps_uncond + cfg_scale * (eps_cond - eps_uncond)

    # 3. SDS gradient: w(t)(ε_pred - ε) · ∂x/∂θ; w(t) = σ_t² is a common choice
    grad = ((1 - abar) * (eps_pred - noise)).detach()
    # backward of (grad · x) gives grad · ∂x/∂θ
    return (grad * x).sum() / B

6.7　VSD: variational SDS (ProlificDreamer, NeurIPS 2023 Spotlight)

VSD (Wang 2023 NeurIPS) views SDS as a special case of "point estimation for a single $\theta$" and generalizes to variational inference over a distribution $\mu(\theta)$ on $\theta$.

Setup

• Treat the 3D parameters $\theta$ as a latent random variable with distribution $\mu(\theta)$
• Goal: align the rendered-image distribution with the diffusion-learned prior distribution (not mode alignment)

Objective and gradient

ProlificDreamer writes the objective as a KL:

$$\min_{\mu}\; D_\text{KL}\!\Big(q_\mu^t(x_t|y)\;\Big\|\;p_\phi^t(x_t|y)\Big),\quad t\sim\mathcal{U}[0,1]$$

where $q_\mu^t$ is the distribution induced by "rendering from $\theta\sim\mu$ + adding noise to time $t$". The variational gradient (Wang et al. 2023, Theorem 2, abbreviated) gives the update direction in $\theta$ as a relative score:

$$\boxed{\;\nabla_\theta \mathcal{L}_\text{VSD} \;=\; \mathbb{E}_{t,\epsilon}\Big[\,w(t)\,\big(\epsilon_\phi(x_t;y,t) \;-\; \epsilon_\psi(x_t;y,t,\pi)\big)\,\frac{\partial x}{\partial \theta}\,\Big]\;}$$

Compared to SDS: replace the raw noise $\epsilon$ with an auxiliary score $\epsilon_\psi$. $\epsilon_\psi$ is a LoRA-finetuned score network that online-minimizes a score-matching loss to track the score of the current $q_\mu^t$; it is the necessary term corresponding to the $q$ distribution's own score (an entropy term) that arises from differentiating the KL objective $D_\text{KL}(q_\mu^t\|p_\phi^t)$ with respect to $\theta$ — it is not a zero-mean, variance-only-reducing control variate. Dropping it degrades the gradient objective from "distribution matching" back to SDS's mode-seeking (it changes the expected gradient direction, not just the noise), so it cannot be equated with a value baseline in RL actor-critic, which only reduces variance without changing the expected gradient. Note: the above is the gradient form, not a squared-loss form; the paper does not have an implementable form "write a scalar loss $\|\epsilon_\phi-\epsilon_\psi\|^2$ and then differentiate" — $\epsilon_\psi$ depends on $\mu$, which would drop the key term of the KL.

Intuition for why over-saturation is mitigated

• SDS: pushes the rendered $x$ toward modes of the prior $p_\phi(\cdot|y)$ (mean-mode → needs CFG=100 → over-saturation)
• VSD: the auxiliary score tracks the score of the current rendered distribution; the update direction = from "where I am now" to "where the prior is" (relative gradient), without needing huge CFG. CFG can drop to 7.5 (the standard diffusion default), avoiding extreme sharpening
• Empirically: VSD has more natural colors and more complex geometry, and can maintain multiple modes simultaneously (ProlificDreamer reports 50k-step training yielding photorealistic Buddha statues, etc.)

6.8　SDS derivative family: mesh / 3DGS + SDS

§7 Feed-forward reconstruction: from COLMAP to pointmap regression

Before 2024, the first step of "multi-view → 3D" was almost always SfM: COLMAP running SIFT → matching → incremental SfM → bundle adjustment, tens of minutes to hours, and frequently failing on texture-less surfaces, low overlap, and dynamic objects. DUSt3R (Wang 2024 CVPR, arXiv:2312.14132, Naver) replaced that entire pipeline with a single network forward pass: no explicit matching, no triangulation, no bundle adjustment — it regresses pointmaps directly. By the time VGGT won CVPR 2025 Best Paper, this line had become the default question direction for 3D vision roles.

7.1　DUSt3R: pointmaps and the confidence-weighted loss (must-derive)

A pointmap $X \in \mathbb{R}^{H\times W\times 3}$ is a dense "one 3D point per pixel" map whose points are expressed in one designated camera frame. It constrains geometry, intrinsics and pose at once, but each is recovered differently:

• Intrinsics: pair the 3D points of $X^{1,1}$ with their own pixel coordinates and fit a projection model.
• Pose: pair the 3D points of $X^{2,1}$ with the pixel coordinates of $I^2$ and solve PnP — it rests on 2D-3D correspondence.
• Depth: the $z$ component of $X^{2,1}$ is not $I^2$'s depth, because those points live in $I^1$'s frame; to get $I^2$'s depth you must first transform the points back into $I^2$'s camera frame and then take $z$.

Given two images $I^1, I^2$, DUSt3R encodes them with weight-shared ViT encoders, lets two decoders cross-attend to each other, and outputs two pointmaps $X^{1,1}$ and $X^{2,1}$. The superscripts are the point: both pointmaps are expressed in the camera frame of $I^1$. Both images' pixels now live in one coordinate system, so pose stops being an input and becomes something you can solve for from the output. That one sentence is the origin of the whole feed-forward reconstruction family.

Regression loss. For valid pixels $i \in \mathcal{D}^v$ of view $v \in \{1, 2\}$ (pixels where GT is defined), the per-point Euclidean error is

$$\ell_{\text{regr}}(v,i)=\Big\lVert \tfrac{1}{z}X^{v,1}_i-\tfrac{1}{\bar z}\bar X^{v,1}_i\Big\rVert$$

where $\bar X$ is the GT. Each side is divided by a scale factor — $z$ from the prediction, $\bar z$ from the GT — both defined identically:

$$z=\text{norm}(X^1,X^2)=\frac{1}{\lvert\mathcal{D}^1\rvert+\lvert\mathcal{D}^2\rvert}\sum_{v}\sum_{i\in\mathcal{D}^v}\lVert X^v_i\rVert$$

that is, the mean distance of all valid points to the origin. After dividing by it the loss is invariant to global scaling: the network is only asked to get the shape right, scale is free. This is exactly where "DUSt3R-style training is up-to-scale by default" comes from.

Getting absolute scale is done in more than one way, and MapAnything is one of them: it keeps this scale-invariant geometric supervision and separately predicts a global scale with its own loss. Normalisation itself also differs — VGGT normalizes only the GT, without DUSt3R's prediction-side normalization, but what it learns is still the normalised scale, not a metric route. The use_metric switch in the code below (setting $z=\bar z$) is the most direct teaching variant, pressing absolute scale straight into the regression term — easy to reason about, but not what either paper does.

Confidence weighting. Real data contains sky, specularities, transparency and moving objects — pixels where the GT itself is untrustworthy. DUSt3R has the network emit an extra confidence map $C^{v,1}$, turning the loss into

$$\boxed{\;\mathcal{L}_{\text{conf}}=\sum_{v}\sum_{i\in\mathcal{D}^v}\Big[\,C^{v,1}_i\,\ell_{\text{regr}}(v,i)\;-\;\alpha\log C^{v,1}_i\,\Big]\;},\qquad C=1+\exp(\check C)\gt 1$$

where $\check C$ is the raw output of the confidence head and the $1+\exp(\cdot)$ parameterization guarantees $C\gt 1$ (every pixel is counted at least once). The paper calls $-\alpha\log C$ a regularization term.

Why $-\alpha\log C$ is mandatory (this is the question). Look at one pixel's loss as a function of $C$ on the domain $C\gt 1$: $f(C)=C\ell-\alpha\log C$.

• Drop the $\log$ term: $f(C)=C\ell$ is non-decreasing in $C$ (since $\ell \ge 0$), so the optimum pushes $C$ to the lower end of the domain, $C\to 1$, and the per-pixel loss falls back to the plain regression loss $\ell$. Geometry is still learned — what degenerates is only the confidence head: it becomes a constant carrying no information. Blocking that degeneracy is exactly what the regularizer is for.
• Keep the $\log$ term: $f'(C)=\ell-\alpha/C$. For $\ell\lt \alpha$ the stationary point $C^\star=\alpha/\ell\gt 1$ lies inside the domain; for $\ell\ge\alpha$, $f'(C)\gt 0$ throughout and the infimum is still at $C\to 1^+$.
• Eliminate $C$ (taking the infimum over $C$ — an optimization elimination, not a probabilistic marginalization):

$$\inf_{C\gt 1} f(C)=\begin{cases}\alpha+\alpha\log(\ell/\alpha), & 0\lt \ell\lt \alpha\\[2pt] \ell, & \ell\ge\alpha\end{cases}$$

• So what does it actually do: the $\ell\lt \alpha$ branch is reshaped into a logarithm, while the $\ell\ge\alpha$ branch stays linear in $\ell$. It is not an outlier suppressor that squashes large residuals into a log — large residuals are still counted linearly. $\alpha$ is where the boundary sits: raising $C$ only pays off once a pixel's residual drops below $\alpha$ (buying a negative $\alpha\log(\ell/\alpha)$). So $C$ learns "can I get this pixel right", and on sky, specularities and moving objects — where it cannot — $C$ sits near 1 and earns no reward.

import torch

def dust3r_conf_loss(pred_pts, gt_pts, conf_raw, valid, alpha=0.2, use_metric=False):
    """ DUSt3R confidence-weighted pointmap loss (pedagogical version).
        pred_pts: [B, V, H, W, 3]  pointmaps from both heads, in the I^1 camera frame
        gt_pts:   [B, V, H, W, 3]  GT in the same frame (V = 2); invalid pixels may be NaN
        conf_raw: [B, V, H, W]     raw confidence-head output Ĉ (pre-activation)
        valid:    [B, V, H, W]     bool, the set of pixels D^v where GT is defined
        use_metric: teaching variant setting z = z̄ to press absolute scale into the
                    regression term (not what the paper does)
        Note: the paper's L_conf sums over valid pixels; this returns a per-sample mean
        over valid pixels, then a batch mean, so values compare across batches.
    """
    eps = 1e-8
    vm = valid.unsqueeze(-1)                                       # [B, V, H, W, 1]
    zero = torch.zeros((), dtype=pred_pts.dtype, device=pred_pts.device)
    # select before computing: invalid GT depth is often NaN, and NaN * 0 is still NaN
    pred_v = torch.where(vm, pred_pts, zero)
    gt_v = torch.where(vm, gt_pts, zero)

    n = valid.flatten(1).sum(1).to(pred_pts.dtype).clamp(min=1.0)  # [B]  |D^1| + |D^2|
    # norm(.) = mean distance of valid points to the origin; one for pred, one for gt
    z_bar = gt_v.norm(dim=-1).flatten(1).sum(1) / n                # [B]  z̄
    z = z_bar if use_metric else pred_v.norm(dim=-1).flatten(1).sum(1) / n

    bshape = (-1,) + (1,) * (pred_pts.dim() - 1)                   # [B, 1, 1, 1, 1]
    l_regr = (pred_v / z.view(bshape).clamp(min=eps)
              - gt_v / z_bar.view(bshape).clamp(min=eps)).norm(dim=-1)   # [B, V, H, W]

    C = 1.0 + conf_raw.exp()                                       # C > 1, every pixel counted
    per_pix = C * l_regr - alpha * C.log()                         # drop the log term -> C collapses to 1
    per_pix = torch.where(valid, per_pix, torch.zeros((), dtype=per_pix.dtype,
                                                      device=per_pix.device))
    return per_pix.flatten(1).sum(1).div(n).mean()

MASt3R (2406.09756) adds a dense local-feature head on top of DUSt3R, fusing "regress a pointmap" and "do pixel-level matching" into one model with much better matching accuracy; the follow-up MASt3R-SfM uses it to replace COLMAP's matching front end.

7.2　The shared recipe and VGGT's four heads

The recipe across this family is broadly uniform, but every line has exceptions and a good answer names them:

1. Views exchange information through attention, rather than hand-crafted matching followed by geometric solving. But "one forward pass consumes all views" is not universal: DUSt3R itself is a pairwise model whose multi-view use still needs a global alignment stage (and does not require running every pair); Fast3R is the one that truly emits all pointmaps in a single pass, though its camera parameters are recovered from the pointmaps afterwards.
2. Outputs land in one common frame, usually the camera frame of some reference image. π³ is the exception: its raw outputs are per-view local pointmaps in each view's own camera frame plus the corresponding poses, and the common frame is assembled from those poses.
3. Pose is usually an output rather than a required input — no SfM initialization needed.
4. Scale depends on how it was trained: DUSt3R-style two-sided (prediction + GT) normalization → up-to-scale; VGGT normalizes only the GT. Metric output has more than one route — MapAnything obtains it by predicting a separate global scale, and you can also simply train on metric geometric supervision (CUT3R does exactly that; MASt3R is likewise trained on metric data).

VGGT (Wang 2025 CVPR Best Paper, arXiv:2503.11651, Oxford + Meta) is the current reference implementation. It processes N images with alternating frame-wise / global attention and draws four heads off one shared backbone:

Why VGGT displaced "COLMAP first" as the interview answer: seconds instead of hours; no dependence on pre-calibrated intrinsics and far more tolerance for limited overlap; four quantities out in one pass, directly usable downstream (3DGS initialization, SLAM, robot mapping). But this is not "COLMAP is dead" — see §12.1 for accuracy and benchmark GT.

7.3　Branch structure: which directions the family is growing in

• Fast3R (2501.13928, Meta, CVPR 2025): DUSt3R is pairwise, so multi-view use means "run some image pairs, then globally align" — two stages. Fast3R pushes all N images through one forward pass of a transformer straight to pointmaps — demonstrated up to 1000+ images — dropping the global alignment; camera parameters are then recovered from the pointmaps in post-processing.
• CUT3R (2501.12387, Berkeley): goes recurrent — it maintains a persistent state, updating it and emitting that frame's pointmap as each frame arrives. That makes it online / streaming, able to build while capturing and to accept a single image; its pointmaps and poses are metric.
• π³ (2507.13347): the methods above all nominate a reference view as the coordinate origin, so results depend on which one was picked and a degenerate reference drags everything with it. π³ is permutation-equivariant: the output is equivariant to input ordering, so no single view is privileged.
• MapAnything (2509.13414, Meta + CMU, 3DV 2026): unifies a pile of tasks with a factored metric representation (ray directions + depth + pose + scale), and its inputs are optional — use intrinsics if you have them, poses if you have them, run with nothing if you have neither. Scale comes from a separately predicted global factor.
• Depth Anything 3 (2511.10647, ByteDance): subtraction in the other direction — no bespoke architecture, no four heads, just a plain transformer with a single depth-ray prediction target. On the authors' own benchmark the paper reports average relative improvements over VGGT of 44.3% on camera pose and 25.1% on geometry. Open weights.
• VGGT-Ω (2605.15195, CVPR 2026 Oral): cuts VGGT's training memory to 30% and scales the supervised training data to 15× (two separate facts, not an equal-budget exchange), extending coverage to dynamic scenes (the VGGT line originally assumes static ones). It predicts depth and cameras. Weights unverified.

MV-DUSt3R+ (2412.06974) is another transition route from DUSt3R to multi-view (multi-view decoder blocks + cross-reference-view fusion); a systematic survey of the family is arXiv:2507.08448.

7.4　Feed-forward 3DGS

The line above regresses geometry; a parallel line regresses a directly renderable 3DGS — replacing the 10-30 minutes of per-scene optimization from §4 with a single forward pass. Note that this line is not uniform about needing input poses:

• LGM (2402.05054, Tang 2024 ECCV): an asymmetric U-Net takes 4 multi-view images and emits one 3D Gaussian per pixel, producing an asset in seconds. Those 4 images come from preset, fixed orbit camera poses that the upstream multi-view generator renders to.
• GS-LRM (2404.19702): swaps LRM's transformer output for Gaussian parameters — per-pixel Gaussians, patch tokens in, Gaussians out, from 2-4 posed views, in 0.23 s on an A100. This is the pivot from "LRM outputs a triplane" to "LRM outputs 3DGS" (LRM itself is in §8.3).
• Long-LRM (2410.12781): pushes the input to 32 posed views at 960×540, reconstructing a room-scale scene in roughly 1 second on an A100 — about 800× faster than per-scene optimization, at comparable or better quality. Followed by Long-LRM++ (2512.10267).
• DepthSplat (2410.13862, CVPR 2025): feeds pretrained monocular depth features into multi-view Gaussian prediction so depth and splatting reinforce each other — the depth prior supplies exactly the constraint that sparse views lack most. Its inputs are calibrated multi-view images.
• NoPoSplat (2410.24207): the Gaussian network itself needs no pose input — it defines the first input image's camera frame as canonical space, emits all Gaussians there, and injects intrinsics as tokens. The "pose-free" claim is scoped to that network: rendering target views and evaluating still involve a separate pose-estimation procedure.
• AnySplat (2505.23716): an uncalibrated image collection goes in and Gaussians + intrinsics + extrinsics come out — the point where the geometry-regression line and the 3DGS-regression line merge.

§8 Single-image / Few-view 3D generation

A more practical setting: given one image, generate 3D.

8.1　Zero-1-to-3 paradigm (novel view via diffusion)

Zero-1-to-3 (Liu 2023 ICCV): finetune Stable Diffusion on Objaverse so it accepts (input view, target camera) → output novel view.

• Input: single image $x$ + relative camera pose $\Delta R, \Delta T$
• Diffusion conditioning: image embedding (CLIP) + camera embedding (sinusoidal)
• Output: the image from the $\Delta R, \Delta T$ viewpoint

Usage: given one input view, sample 16-32 novel views, then reconstruct via NeRF / 3DGS.

Derivatives:

• Zero-1-to-3++ (Shi 2023): fixed generation of 6 anchor views (azimuth uniformly spaced every 60° at 30°/90°/150°/210°/270°/330°, elevation alternating between -10° and 20° — i.e. interleaved elevation + uniform azimuth, not "one north-pole view + four eye-level views + one top view"), reducing randomness
• SyncDreamer (Liu 2024 ICLR): joint prediction of multiple views in latent space (cross-attention lets views see each other), ensuring 3D consistency
• MVDream (Shi 2024 ICLR): text-to-multi-view, generates 4 views simultaneously; followed by SDS refinement

8.2　One-2-3-45 / InstantMesh / TripoSR / Stable Fast 3D

LRM (Hong et al. 2023 arXiv → ICLR 2024) setting: treat the image as tokens + Plucker ray embedding, transformer outputs a NeRF triplane. This is the parent model of TripoSR / InstantMesh.

8.3　LRM Triplane representation (high-frequency interview topic)

• Triplane (Chan 2022 EG3D): 3 axis-aligned 2D planes (XY, YZ, XZ), total $3 \times C \times N \times N$ dim
• Query 3D point $(x, y, z)$: bilinearly interpolate on each plane → element-wise sum of the three plane features (EG3D uses sum, not concat) → small MLP → $(\sigma, \mathbf{c})$
• Advantages: less VRAM than a voxel grid ($O(N^2)$ vs $O(N^3)$), denser than a hash grid making it suitable as transformer output
• LRM / TripoSR / InstantMesh all let the transformer directly regress triplane tokens

§9 3D Foundation Models (the 2024–26 open-source wave)

9.1　TRELLIS: SLAT and two-stage rectified flow

TRELLIS (Xiang 2024, arXiv:2412.01506, Microsoft) is the most complete open attempt at a "Stable Diffusion for 3D", and the one worth explaining precisely in an interview — because its latent design directly determines its engineering shape.

SLAT (Structured LATents) is defined as

$$\boldsymbol z=\{(\boldsymbol z_i,\boldsymbol p_i)\}_{i=1}^{L},\qquad \boldsymbol p_i\in\{0,1,\dots,N-1\}^3,\quad N=64$$

a set of "latent vector + the voxel coordinate it sits at". $N=64$ is the voxel resolution ($64^3\approx 2.6\times10^5$ cells), and $L\approx 20\text{K}$ is the number of occupied active voxels — roughly 8% occupancy.

Get the division of labour right: $\boldsymbol p_i$ only fixes the sparse support — which cells contain material, at $64^3$ granularity. The actual fine geometry and appearance are both encoded in $\boldsymbol z_i$, since a 64-resolution voxel grid could never express sharp edges and smooth surfaces on its own. $\boldsymbol z_i$ comes from projecting each active voxel into multi-view renderings, sampling and aggregating from DINOv2 feature maps, then compressing with a sparse VAE. So SLAT encodes geometry and appearance jointly from the start.

Two rectified-flow stages (the order cannot be swapped):

1. Structure stage: generate which voxels are active, i.e. the sparse structure $\{\boldsymbol p_i\}$ itself.
2. Latent stage: generate $\{\boldsymbol z_i\}$ on the already-determined active voxels.

The direct payoff of splitting is that the second stage runs flow over ~20K tokens instead of the $64^3\approx262\text{K}$ cells — fix the skeleton first, then fill in content, and the attention and forward compute on the 92% empty cells simply never happens.

Multiple decoders: three decoders were trained on the same SLAT, producing 3DGS / Radiance Field / Mesh (the mesh path goes through FlexiCubes, not naive marching cubes — see §5.2).

Model sizes are 342M / 1.1B / 2B, trained on roughly 500K assets filtered from four public datasets (Objaverse(-XL), ABO, 3D-FUTURE, HSSD). Open weights.

TRELLIS.2 (2512.14692) is its successor. Its Sparse Compression VAE takes O-Voxel as its native representation — sparse voxels carrying geometry and PBR material together; the VAE compresses O-Voxel into a tighter latent, three flow models split the generation work on that compressed latent, and meshes come out through its own dual-grid conversion. Weights unverified.

9.2　The Hunyuan3D family: 2.0 → 2.1 → 2.5 → Omni / Studio / Buffalo

Hunyuan3D takes the shape-then-texture two-stage route and is the fastest-iterating line in the Chinese open-source ecosystem.

• Hunyuan3D-1 (Yang 2024): stage 1 text/image → multi-view (Zero-1-to-3 family), stage 2 multi-view → mesh (LRM-like).
• Hunyuan3D 2.0 (2501.12202): Hunyuan3D-DiT — ShapeVAE encodes a point cloud into a vecset shape latent, flow matching runs on that latent, and the decoder produces an SDF; Hunyuan3D-Paint does multi-view texture diffusion plus UV-space refinement, outputting RGB textures. Open weights.
• Hunyuan3D 2.1 (2506.15442): PBR material generation enters the main line at this version, along with the released training code — the latter matters far more than the version number for anyone reproducing it.
• Hunyuan3D 2.5 (2506.16504): the geometry model becomes LATTICE, 10B parameters, with clearly better detail and sharp edges. Weights unverified.
• Hunyuan3D-Omni (2509.21245): controllable generation — beyond image/text it accepts pose / bounding box / voxel control signals. Weights unverified.
• Hunyuan3D Studio (2509.12815): a production-oriented asset toolchain (retopology, UV, texture integration). Weights unverified.
• Hunyuan3D Buffalo 1.0 (2608.02711): unifies generation, understanding and editing in one model. Weights unverified.

9.3　The latent-representation axis: CLAY / Direct3D / Dora / Step1X-3D

Sorting these by release date teaches nothing; sorting them by latent representation and decoding target is the structure an interview wants.

• CLAY (Zhang 2024 SIGGRAPH, 2406.13897): multi-resolution VAE + latent DiT. Its shape VAE extends the vector-set representation of 3DShape2VecSet (Zhang et al. 2023, arXiv:2301.11445), placing it in the VecSet lineage; the DiT diffuses on that latent and decodes an occupancy field, from which marching cubes extracts the surface, followed by a PBR texture stage.
• Direct3D (2405.14832): D3D-VAE encodes the mesh into an explicit triplane latent and D3D-DiT diffuses on the triplane. Triplane lets you reuse the entire 2D conv/attention toolchain; the cost is projection ambiguity across the three planes (thin elongated structures overlap on some plane).
• Direct3D-S2 (2505.17412): switches to a sparse voxel latent with spatial sparse attention (SSA), bringing 1024³ training within reach of 8 GPUs. The reported 3.9× forward / 9.6× backward figures are the SSA operator against FlashAttention-2, not an end-to-end model speedup.
• Dora (2412.17808, CVPR 2025): stays on VecSet and changes the sampling and attention — Sharp Edge Sampling tilts the sampling budget toward geometric sharp edges, paired with dual cross-attention. The result: Dora-VAE uses 1,280 latent codes to match the dense XCube-VAE's reconstruction quality on Dora-bench, where the latter needs >10,000. An order-of-magnitude drop in latent count directly sets the DiT's sequence length. Open weights.
• Step1X-3D (2505.07747): geometry via a hybrid VAE-DiT with a VecSet latent decoded to TSDF, trained on 2M curated assets (filtered from 5M+); code, weights and the training pipeline for both the geometry and texture stages are fully open (Apache-2.0). Open weights.
• Meta AssetGen 2 (2605.26137): a closed product line pushing single-asset generation to roughly 30 seconds on its H100 deployment, optimized for production usability rather than paper metrics.

Rodin (Microsoft 2023, commercial): an early product-grade text-to-3D-avatar system, diffusion on triplane, aimed at characters / avatars.

9.4　Comparison table: latent representation is the axis

The "Weights" column says "Open weights" only where verified, "Weights unverified" where no official release was found, and "—" where this tutorial did not check.

9.5　Native mesh generation: why artists reject isosurface-extracted output

The mesh output of the methods above comes mainly by two different routes: extracting an isosurface from a field — naive marching cubes (§5.1) or a differentiable variant such as FlexiCubes (§5.2, which is what TRELLIS uses); or, as in TRELLIS.2, converting O-Voxel directly through a dual grid, never passing through a field at all. For both routes the problem is not extraction quality but topology: the triangle layout follows the extraction or voxel grid rather than the shape's structure. They can be UV-unwrapped, decimated and turned into LODs — none of that is impossible, and methods like FlexiCubes preserve sharp edges to a considerable degree. What hurts is that the output is not guaranteed to have edge flow suited to editing, rigging and deformation: there are no edge loops organized along silhouettes and creases, so reshaping and skinning are awkward, and decimation has to fight the existing triangulation instead of following it. A hand-built artist-created mesh typically has a few hundred to a few thousand faces, with every edge hugging a geometric feature.

Hence native mesh generation: no isosurface, no voxel conversion — autoregressively emit the face sequence directly. The main thread of this line is token compression: sequence length = face count × tokens per face, and transformers are quadratic. The baseline everyone is measured against is the naive sequence of 9 coordinate tokens per face (3 vertices × 3 coordinates) — until Meshtron, which changes tack and carries the long sequence with architecture instead of compressing further.

• MeshGPT (2311.15475): the origin. Graph-convolutional encoding plus residual quantization compresses each face into 6 codebook tokens, generated autoregressively by a decoder-only transformer.
• MeshAnything (2406.10163): adds shape conditioning — obtain a coarse shape by any means (reconstruction, generation), then have the AR model "re-model" it into an artist-style mesh, with hundreds of times fewer faces than isosurface extraction gives.
• MeshAnything V2 (2408.02555): introduces Adjacent Mesh Tokenization (AMT) — since adjacent faces share vertices, most faces need only one new vertex encoded instead of three, bringing sequence length to about half the naive one and doubling the achievable face limit.
• BPT (2411.07025): Blocked and Patchified Tokenization uses block indexing (coordinates split into block id + in-block offset) together with vertex sharing inside a patch, shortening the naive sequence by ~75% and pushing generatable faces past 8K.
• TreeMeshGPT (2503.11629, CVPR 2025): replaces "order the faces somehow" with autoregression along an adjacency tree — decoding pops an edge from a stack of expandable edges and grows a face outward. The compressed sequence is about 22% of the original length (≈2 tokens/face) — that is "22% remaining", not "22% fewer". Its sequence construction also constrains face orientation, markedly reducing normal flips, though not eliminating post-processing entirely.
• Meshtron (2412.09548): does not compress tokens but takes the long sequence head-on — hourglass architecture + sliding-window attention reaching 64K faces @ 1024 coordinate resolution. Weights unverified.
• DeepMesh (2503.15265): attaches DPO to AR mesh generation, writing "topology humans prefer" straight into the objective — among the first post-training work in mesh generation.

9.6　Part-level and sim-ready: from a part-less whole to a usable asset

A generated asset usually lacks explicit part and joint structure. The drawer a robot must pull, the wheel it must turn, the button it must press are not separately addressable in the output — the simulator cannot assign them mass, friction, or joints. This is the closest and most blocked link between 3D generation and Embodied AI deployment.

Step one: split into parts. Four works with different routes and one goal:

• PartGen (2412.18608): multi-view diffusion → part-level segmentation + completion of the occluded portions (a segmented part is itself incomplete and must be generatively completed).
• HoloPart (2504.07943): formalizes this as 3D part amodal segmentation — decompose a whole mesh into semantically complete parts, each completed into an independent closed body.
• PartCrafter (2506.05573): instead of "whole first, then split", it denoises multiple parts jointly — spatial relations between parts are learned in that joint training rather than fixed by post-hoc alignment.
• PartPacker (2506.09980, NVIDIA): dual volume packing packs multiple parts into two volume fields generated at once, sidestepping the trouble that a variable part count causes for a network's output dimension.

Step two: geometry → parts → joints → materials → URDF. Splitting parts only solves geometry; sim-ready is three steps further: what joint connects two parts (revolute / prismatic / fixed), where its axis sits, what its limits are; each part's physical material (mass, friction, restitution); and finally packaging into a URDF / MJCF the simulator can read.

• EmbodiedGen (2506.10600): one of the most complete open works walking this whole chain, emitting URDF assets with physical attributes that plug into IsaacSim / MuJoCo.
• Artiverse (2605.24403): 5.4K articulated objects across 88 categories. For reference, PartNet-Mobility holds 2,346 — articulated assets remain at the thousands scale, and while Artiverse more than doubles that, it is still three orders of magnitude below the geometry layer.

§10 Complexity / resource comparison

§11 Comparison with related methods & Embodied AI applications

11.1　Key differences between 3D and 2D generation

11.2　Embodied AI / AR / VR practical routes

§12 Engineering practice & common footguns

12.1　COLMAP or feed-forward?

COLMAP: input multi-view → output intrinsics $K$ + extrinsics $\{R_i, t_i\}$ + sparse point cloud; standard pipeline SIFT → matching → incremental SfM → bundle adjustment. Common pitfalls: SfM fails on texture-less / specular objects; dynamic objects pollute extrinsics.

After §7 this step is no longer mandatory, but it is nowhere near deletable. The division of labour is clear:

• Accuracy depends on conditions, not on the brand name. COLMAP is strong on static scenes with enough texture and overlap; but monocular SfM is itself up-to-scale — absolute scale has to come from a calibration target, a known baseline or another sensor, and COLMAP does not supply it.
• Benchmarks do not share one GT convention. Some reconstruction benchmarks do derive their poses from COLMAP; others use sensor ground truth or synthetic data. Match whichever convention the baseline you want to compare against used, rather than asserting that "the benchmark is COLMAP".
• Feed-forward models deliver seconds instead of hours, and still produce results where COLMAP struggles: low overlap, few views, texture-less. Note that unknown intrinsics are not a COLMAP failure condition — it can self-calibrate, it just drifts more easily.
• Connecting the two costs engineering. VGGT-X (2509.25191) existing at all is the evidence: feeding VGGT's outputs into large-scale 3DGS training needed extra work on memory, point-cloud noise and pose accuracy before it ran — this is not merely "swap the pose source".

12.2　Numerical stability (general NeRF/3DGS)

12.3　Multi-machine distributed & evaluation metrics

Distributed: NeRF / Instant-NGP / 3DGS are single-GPU standard; large 3DGS scenes use chunk-wise (VastGaussian, CityGaussian); SDS/VSD runs 2 SD forwards per iter, 8×A100 gives significant speedup; TRELLIS / Hunyuan3D training is large-scale multi-node DDP.

Commonly used evaluation sets: GSO (Google Scanned Objects, 2204.11918), 1000+ real scanned objects, the most common evaluation set for single-image-to-3D; Toys4K (2101.07296), ~4K objects across 105 categories, good category diversity, common for few-shot and part-level experiments; Objaverse-XL (2307.05663), 10M+ objects, treated by most work as a training source. A dataset does not carry a "train" or "test" identity of its own — the same assets are training data in one paper and evaluation data in another, so when you see a score, ask how the split was drawn; the generalization claim holds only on that split.

§13 25 frequently-asked interview questions

Sorted into 3 tiers by difficulty (L1 must-know / L2 advanced / L3 top labs). Each question links to answer points + footguns.

L1 must-know (asked at every 3D / vision role)

L2 advanced (research-oriented roles)

L3 top-lab (top-conference / industry research roles)

§A Appendix: full code skeleton + references

A.1　Complete from-scratch code includes

volume_render() (NeRF α-compositing with numerical stability) · positional_encoding() (γ(p) Fourier features) · gaussian_splat_forward() (3DGS pedagogical forward + projection Jacobian) · densify_and_prune() (3DGS densification heuristics) · sds_loss() (SDS gradient surrogate) · dust3r_conf_loss() (DUSt3R confidence-weighted pointmap loss, with scale normalization and a metric switch) · marching_cubes_sketch() (mesh extraction interface using scikit-image).

A.2　Key papers reading list

• NeRF family: Mildenhall 2020 ECCV (HM); Müller Instant-NGP SIGGRAPH 2022 Best; Barron Mip-NeRF / 360 ICCV 2021 / CVPR 2022; Wang NeuS + Yariv VolSDF NeurIPS 2021; Fridovich-Keil Plenoxels CVPR 2022; Chen TensoRF ECCV 2022.
• 3DGS family: Kerbl 3D Gaussian Splatting SIGGRAPH 2023 Best; Huang 2D Gaussian Splatting SIGGRAPH 2024; Luiten Dynamic 3DGS 3DV 2024; Wu 4DGS CVPR 2024; Guédon SuGaR CVPR 2024.
• Mesh / SDF: Shen DMTet NeurIPS 2021 / FlexiCubes SIGGRAPH 2023.
• SDS family: Poole DreamFusion arXiv 2022.09 → ICLR 2023 Outstanding; Wang ProlificDreamer (VSD) NeurIPS 2023 Spotlight; Lin Magic3D CVPR 2023; Chen Fantasia3D ICCV 2023; Tang DreamGaussian ICLR 2024; Yi GaussianDreamer CVPR 2024.
• Single-image 3D: Liu Zero-1-to-3 ICCV 2023 / One-2-3-45 NeurIPS 2023 / SyncDreamer ICLR 2024; Shi Zero-1-to-3++ arXiv 2023 / MVDream ICLR 2024; Hong LRM arXiv 2023.11 → ICLR 2024; Tochilkin TripoSR arXiv 2024; Xu InstantMesh arXiv 2024; Boss Stable Fast 3D arXiv 2024.
• 3D Foundation Models: Xiang TRELLIS 2412.01506 (Microsoft) / TRELLIS.2 2512.14692; Tencent Hunyuan3D 2.0 2501.12202 / 2.1 2506.15442 / 2.5 2506.16504 / Omni 2509.21245 / Studio 2509.12815 / Buffalo 1.0 2608.02711; Zhang CLAY SIGGRAPH 2024 (2406.13897); Direct3D 2405.14832 / Direct3D-S2 2505.17412; Dora 2412.17808 (CVPR 2025); Step1X-3D 2505.07747; Meta AssetGen 2 2605.26137.
• Feed-forward reconstruction: DUSt3R 2312.14132 (CVPR 2024) / MASt3R 2406.09756; VGGT 2503.11651 (CVPR 2025 Best Paper); Fast3R 2501.13928 (CVPR 2025); CUT3R 2501.12387; π³ 2507.13347; MapAnything 2509.13414 (3DV 2026); Depth Anything 3 2511.10647; VGGT-Ω 2605.15195 (CVPR 2026 Oral); MV-DUSt3R+ 2412.06974; VGGT-X 2509.25191; survey 2507.08448.
• Feed-forward 3DGS: LGM 2402.05054; GS-LRM 2404.19702; Long-LRM 2410.12781 / Long-LRM++ 2512.10267; DepthSplat 2410.13862 (CVPR 2025); NoPoSplat 2410.24207; AnySplat 2505.23716.
• Native mesh generation: MeshGPT 2311.15475; MeshAnything 2406.10163 / V2 2408.02555; BPT 2411.07025; TreeMeshGPT 2503.11629 (CVPR 2025); Meshtron 2412.09548; DeepMesh 2503.15265; Nexus 2607.13563.
• Part-level / sim-ready: PartGen 2412.18608; HoloPart 2504.07943; PartCrafter 2506.05573; PartPacker 2506.09980 (NVIDIA); EmbodiedGen 2506.10600; Artiverse 2605.24403.
• Datasets / evaluation: Objaverse-XL 2307.05663 (10M+); GSO 2204.11918 (real scanned objects); Toys4K 2101.07296 (~4K objects / 105 categories); ULIP 2212.05171 / ULIP-2 2305.08275 (3D-image-text alignment).

A.3　Common Embodied AI / AR / VR follow-ups

3DGS connected to a physics engine → first extract mesh via 2DGS / SuGaR → IsaacSim / MuJoCo; dynamic NeRF → 4DGS / D-NeRF / K-Planes; real-time AR 3DGS → mobile-friendly (PostShot, Luma) + pruning / quantization; insufficient 3D data → Objaverse-XL (TRELLIS), 2D distillation (DreamFusion family), or multi-view heuristics (MVDream); a fresh capture to reconstruct → run VGGT first for structure, then decide whether COLMAP is worth waiting for (§12.1); a generated asset heading into a simulator → mesh alone is not enough, parts, joints and physical materials are still missing (§9.6).

3D Generation Quick Reference · Main references: Mildenhall 2020 (NeRF), Müller 2022 (Instant-NGP), Kerbl 2023 (3DGS), Poole 2022/ICLR 2023 (DreamFusion), Wang 2023 (VSD), Wang 2024 (DUSt3R), Wang 2025 (VGGT), Xiang 2024 (TRELLIS), Tencent 2025-26 (Hunyuan3D 2.x / Omni / Buffalo). Covers: NeRF volume-rendering derivation, Instant-NGP hash grid, 3DGS projection Jacobian, SDS / VSD gradient derivation, DUSt3R's confidence-weighted loss, feed-forward reconstruction and feed-forward 3DGS, single-image 3D, the latent-representation axis across 3D foundation models, native mesh generation, and part-level / sim-ready assets. Essential for Embodied AI / AR / VR.