Image to 3D
Asset Generator
Upload a reference image. Get a downloadable GLB. Forge3D connects a Next.js frontend to a FastAPI orchestration layer to a local ComfyUI + TRELLIS 2 inference server — three layers, one seamless pipeline.
What Is Forge3D?
Forge3D is a web tool that takes a reference image and turns it into a downloadable 3D asset (.glb) using an AI generation pipeline. The user uploads a photo or illustration, confirms the file details, and waits while the system processes the mesh — then downloads a ready-to-use 3D model.
The frontend and the inference server never communicate directly. A FastAPI layer in the middle owns the job queue, the ComfyUI workflow injection, the WebSocket connection that detects completion, and the file delivery back to the browser.
- Three-panel UI: input, interactive 3D canvas, history + tools
- Real-time polling with staged status (queued → generating → done)
- WebSocket completion detection — no polling the inference server
- TRELLIS 2 running locally via ComfyUI on a dedicated GPU machine
- History tab with localStorage thumbnail previews
System Overview
The system is built across three distinct layers, each with a single responsibility. The Next.js frontend handles all UI and user interaction. The FastAPI backend owns orchestration — job lifecycle, workflow patching, and file delivery. ComfyUI + TRELLIS 2 runs entirely on a local GPU machine and is never exposed to the browser.
1. Next.js Frontend
App Router + TypeScript + Tailwind CSS v4. Three-panel layout: image upload on the left, interactive GLB viewer in the center, history and tools on the right. Zustand manages all shared state across panels.
2. FastAPI Backend
Accepts image uploads, assigns job IDs, and runs generation in background tasks. Injects dynamic values into a saved ComfyUI workflow JSON, submits it to ComfyUI, and listens for completion via WebSocket. Serves the finished GLB back to the client.
3. ComfyUI + TRELLIS 2
A locally hosted ComfyUI instance running on a Windows GPU machine (default http://127.0.0.1:8188). TRELLIS 2 runs as custom nodes inside ComfyUI — it receives a reference image and outputs a 3D mesh as a .glb file.
┌───────────────────────────────────────┐
│ Next.js Frontend │
│ App Router · TypeScript · Tailwind │
│ │
│ Left → Image upload + settings │
│ Center → Interactive 3D GLB viewer │
│ Right → History tab + tools │
└──────────────┬────────────────────────┘
│ POST /generate
│ GET /jobs/{id} (poll 2s)
│ GET /jobs/{id}/download
┌──────▼──────────────────────┐
│ FastAPI (port 8000) │
│ Background tasks │
│ Workflow JSON patching │
│ WebSocket listener │
└──────┬──────────────────────┘
│ POST /upload/image
│ POST /prompt
│ WS /ws?clientId=…
│ GET /history/{id}
│ GET /view?filename=…
┌──────▼──────────────────────┐
│ ComfyUI (127.0.0.1:8188) │
│ TRELLIS 2 custom nodes │
│ Local Windows GPU machine │
└─────────────────────────────┘Three-Panel Layout
The UI is split into three vertical sections — a pattern common to creative tools. The left side is input, the center is output, and the right side provides history and utilities without cluttering the primary workspace.
dynamic to skip SSR. 1const ALLOWED_TYPES = new Set([
2 "image/jpeg", "image/png", "image/webp",
3 "image/gif", "image/avif", "image/tiff", "image/bmp",
4]);
5const MAX_BYTES = 20 * 1024 * 1024; // 20 MB
6
7function validateAndSetImage(file: File) {
8 if (!ALLOWED_TYPES.has(file.type)) {
9 toast.error("Unsupported file type", {
10 description: "Please upload JPEG, PNG, WebP, GIF, AVIF, TIFF or BMP",
11 });
12 return;
13 }
14 if (file.size > MAX_BYTES) {
15 toast.error("Image too large", {
16 description: `Max 20 MB — your file is ${fmt(file.size)}.`,
17 });
18 return;
19 }
20 const url = URL.createObjectURL(file);
21 setImageFile(file, url); // stored in Zustand
22}The blob URL lifecycle is tied to Zustand state — reset() calls URL.revokeObjectURL before clearing the store.
Pre-Generation Confirmation
Before submitting to the API, the user sees a confirmation dialog showing the image's filename, pixel dimensions, file size, and format. Pixel dimensions are read client-side using createImageBitmap — a browser API that decodes the image without a server round-trip.
3D generation can take several minutes. This dialog gives the user a chance to catch the wrong file before the wait begins, without requiring a server call to validate the image.
1// Read pixel dimensions without a server round-trip
2const bitmap = await createImageBitmap(imageFile);
3const info = {
4 name: imageFile.name,
5 size: imageFile.size,
6 type: imageFile.type,
7 width: bitmap.width,
8 height: bitmap.height,
9};
10bitmap.close(); // free decoded bitmap memoryTyped API Module
All API calls go through a single typed module (src/lib/api.ts). It wraps fetch in a helper that reads the base URL from an environment variable and throws on non-2xx responses. The rest of the app only imports from this module — no raw fetch calls elsewhere.
api.health()api.generate(image, name, seed?)job_id and seedapi.getJob(jobId)api.getJobDownloadUrl(jobId)api.listJobs()1const BASE_URL =
2 process.env.NEXT_PUBLIC_API_URL ?? "http://localhost:8000";
3
4async function request<T>(path: string, init?: RequestInit): Promise<T> {
5 const res = await fetch(`${BASE_URL}${path}`, init);
6 if (!res.ok) {
7 const body = await res.text().catch(() => res.statusText);
8 throw new Error(body || `HTTP ${res.status}`);
9 }
10 return res.json() as Promise<T>;
11}1// DO NOT set Content-Type manually — the browser sets the
2// boundary automatically when body is a FormData instance.
3generate: (image: File, name: string, seed?: number) => {
4 const form = new FormData();
5 form.append("image", image);
6 form.append("name", name);
7 if (seed != null) form.append("seed", String(seed));
8 return request<GenerateResponse>("/generate", {
9 method: "POST", body: form
10 });
11},Polling + Staged Status UI
3D generation is slow — it can take several minutes. Rather than holding a connection open, the frontend polls GET /jobs/{id} every 2 seconds and maps each backend status to a labeled phase shown in the UI.
The Polling Loop
The interval reference is stored in a useRef so it persists across renders without causing re-renders. Network failures during polling are treated as transient — a warn log entry is added but the interval keeps running.
1let pollCount = 0;
2pollRef.current = setInterval(async () => {
3 pollCount++;
4 addActivity("info", `GET /jobs/${jobId} (poll #${pollCount})`);
5
6 const job = await api.getJob(jobId);
7
8 if (job.status === "pending") setStatus("queued");
9 if (job.status === "processing") setStatus("generating");
10
11 if (job.status === "done") {
12 stopPolling();
13 setStatus("postprocessing");
14 const dlRes = await fetch(api.getJobDownloadUrl(jobId));
15 const blob = await dlRes.blob();
16 setAssetUrl(URL.createObjectURL(blob));
17 setStatus("done");
18 }
19
20 if (job.status === "failed") {
21 stopPolling();
22 setStatus("failed");
23 setError(job.error ?? "Generation failed");
24 }
25}, 2000);Staged Status Phases
Rather than a generic spinner, each phase maps to a distinct label shown in the bottom status bar and as a running pill overlaid on the 3D canvas. Users know whether they're waiting on a queue or on actual GPU compute.
| Status | Meaning |
|---|---|
queued |
Job submitted, waiting in the FastAPI queue |
analyzing |
Backend is uploading and processing the image |
generating |
TRELLIS 2 is running the 3D generation on GPU |
postprocessing |
Downloading the finished GLB from the API |
done |
GLB loaded and visible in the 3D viewer |
failed |
Job errored — error message surfaced in the UI |
3D Viewer
Once a GLB blob URL lands in the Zustand store, the viewer loads it with useGLTF from @react-three/drei. The canvas is lazy-loaded via Next.js dynamic to avoid running WebGL imports during server-side rendering.
1function GlbModel({ url }: { url: string }) {
2 const { scene } = useGLTF(url);
3 return <primitive object={scene} />;
4}
5
6// Lazy-loaded — skip SSR for WebGL
7const Viewer = dynamic(
8 () => import("./GlbViewer"),
9 { ssr: false }
10);Zustand State
The Zustand store is the single source of truth for anything that crosses component boundaries. The reset() action revokes both blob URLs before clearing state, preventing memory leaks.
imageFile / imagePreviewUrl— set by left panel, cleared by resetstatus— drives polling, status bar, generate buttonassetUrl— GLB blob URL consumed by the 3D vieweractivity— append-only log of every API call and polllogs— per-job summaries for the history tabThumbnail Previews in localStorage
The history tab fetches GET /jobs on mount and re-fetches after every generation. Each job appears as a card in a 3-column grid. Thumbnail previews are generated client-side at job creation time using a canvas — this avoids needing a thumbnail API endpoint and means previews survive page refreshes.
Older jobs without a saved thumbnail fall back to a placeholder icon. Clicking a card fetches and loads the GLB into the viewer. A three-dot menu provides View in 3D, Download GLB, and Copy Job ID.
1async function saveThumbnail(jobId: string, file: File) {
2 const bitmap = await createImageBitmap(file);
3 const size = 300;
4 const scale = Math.min(
5 size / bitmap.width,
6 size / bitmap.height
7 );
8 const canvas = document.createElement("canvas");
9 canvas.width = Math.round(bitmap.width * scale);
10 canvas.height = Math.round(bitmap.height * scale);
11 canvas.getContext("2d")!.drawImage(
12 bitmap, 0, 0, canvas.width, canvas.height
13 );
14 bitmap.close(); // free decoded memory
15 localStorage.setItem(
16 `thumb_${jobId}`,
17 canvas.toDataURL("image/jpeg", 0.75)
18 );
19}FastAPI Orchestration Layer
The API sits between the Next.js frontend and the ComfyUI inference server. Its job is to accept image uploads from the browser, orchestrate a 3D generation job on ComfyUI, and deliver the finished .glb file back to the client. The frontend and ComfyUI never talk directly.
| Method | Path | Description |
|---|---|---|
| GET | /health | API liveness check |
| GET | /health/comfyui | Probes the ComfyUI server |
| POST | /generate | Accepts image + name, queues a generation job — returns 202 immediately |
| GET | /jobs/{job_id} | Returns current job status (pending / processing / done / failed) |
| GET | /jobs/{job_id}/download | Streams the finished .glb — Content-Type: model/gltf-binary |
| GET | /jobs | Lists all jobs and their statuses |
Reads image bytes, assigns a UUID job ID, returns 202 Accepted immediately. Actual generation runs in a FastAPI BackgroundTask.
Re-uploads image bytes to ComfyUI's /upload/image. The image is pre-processed to 512×512 PNG on the frontend to keep generation time predictable.
Loads Trellis_2.json — a workflow exported from ComfyUI — and patches three nodes at runtime. Everything else (model weights, sampler settings, mesh resolution) is preserved exactly as configured.
Opens a persistent WebSocket to /ws?clientId=… with keepalive pings disabled to survive multi-minute generations. Listens for executing with node: null (done), execution_error, or execution_interrupted.
Queries /history/{prompt_id} and parses the output node graph to locate the .glb path. Because ComfyUI returns Windows absolute paths, the filename is extracted from the tail and fetched via /view.
File saved to outputs/{job_id}.glb. Frontend polls until status is done, then streams the GLB from /jobs/{id}/download.
1import json, copy
2
3with open("Trellis_2.json") as f:
4 base_workflow = json.load(f)
5
6def build_prompt(image_name: str, asset_name: str, seed: int) -> dict:
7 wf = copy.deepcopy(base_workflow)
8
9 # Node 6 — uploaded image filename
10 wf["6"]["inputs"]["image"] = image_name
11
12 # Node 214 — generation seed
13 wf["214"]["inputs"]["seed"] = seed
14
15 # Node 219 — output filename prefix
16 wf["219"]["inputs"]["filename_prefix"] = asset_name
17
18 return wf1payload = {
2 "prompt": build_prompt(image_name, asset_name, seed),
3 "client_id": client_id, # UUID per job
4}
5resp = await http.post("/prompt", json=payload)
6prompt_id = resp.json()["prompt_id"]ComfyUI + TRELLIS 2
TRELLIS 2 is Microsoft's image-to-3D model. Inside ComfyUI it runs as a set of custom nodes that take one or more reference images and output 3D meshes and textures. Everything runs locally — there is no cloud API.
Getting ComfyUI + TRELLIS 2 Running
Portable build or git clone. Confirm the base UI runs at http://127.0.0.1:8188 before installing any custom nodes.
Via ComfyUI Manager (easiest) or manually — git clone into ComfyUI/custom_nodes/ and pip install -r requirements.txt against the ComfyUI Python interpreter.
TRELLIS 2 nodes are picky about the torch/CUDA build. Recent guides pin torch==2.8.0 with a cu121 or cu128 wheel depending on GPU. If using the portable build, install against the embedded interpreter.
Into ComfyUI/models/trellis/ via huggingface-cli download microsoft/TRELLIS-image-large or the equivalent for TRELLIS 2 weights.
Drop a community TRELLIS 2 workflow JSON onto the canvas. Confirm a single image → 3D mesh works end-to-end in the browser UI before wiring up API calls — it isolates "ComfyUI/TRELLIS problem" from "my client problem."
--disable-pinned-memory launch flag| Endpoint | Purpose |
|---|---|
| POST /prompt | Queue a workflow. Returns prompt_id |
| WS /ws?clientId=… | Real-time execution events — progress, completion, errors |
| GET /history/{id} | Fetch results and output file paths after completion |
| GET /view | Retrieve output file by filename, subfolder, type |
| POST /upload/image | Upload input image into ComfyUI's input folder |
| GET /queue | Inspect the current execution queue |
| POST /interrupt | Cancel the currently running prompt |
| GET /object_info | Introspect node types — validate workflow JSON at startup |
executing with node: null
→ Job finished. Proceed to /history/{prompt_id} to locate the output files.
execution_error
→ Job failed. Surface the node ID and error message back to the client.
execution_interrupted
→ Job was cancelled via POST /interrupt.
Exporting the workflow directly from ComfyUI guarantees the API sends a configuration that already works. If it runs in the UI, it will run via the API.
Only three values are injected at runtime — image filename, seed, asset name. Model weights, sampler config, mesh resolution, and VRAM settings stay untouched.
Call /object_info at startup to confirm node names and parameter keys still match what's installed. Custom node updates can rename fields — this catches it early.
Technical Highlights
BackgroundTasks keeps the implementation self-contained without Redis or Celeryreset() revokes both the image preview URL and the GLB asset URL — memory is freed when state is clearedChallenges Solved
/history responses. The filename is extracted from the path tail before fetching via /viewContent-Type manually on a FormData request breaks the boundary the browser auto-generates — the server cannot parse the bodyConclusion
Forge3D connects three systems that each speak a different protocol — a browser frontend, an async Python API, and a local node-based inference server — into a single cohesive pipeline. Each layer has exactly one job: the frontend owns UX, FastAPI owns orchestration, and ComfyUI owns inference.
The workflow-as-JSON approach means the generation configuration is always known-good — tested in the ComfyUI UI before a single API call is made. The FastAPI layer's only job is to inject three values, submit it, and listen for completion. This simplicity makes the system reliable even when generation takes several minutes and network conditions are imperfect.
The Quality and Style dropdowns are currently cosmetic — the API only accepts image, name, and seed. The provider abstraction is designed to make passing generation parameters through straightforward without changing the frontend contract as the backend grows.
