A fully local AI pipeline that turns text prompts into game-ready sprites and concept art, powered by Flux running on ComfyUI.
I'm building a 2D 32-bit game in Unity and kept running into the same problems: generating consistent character sprites, creating sprite sheets, iterating quickly on concept art, and maintaining a cohesive visual style across assets.
Hosted AI APIs like DALL-E and Midjourney were too expensive for rapid experimentation and didn't give me the control I needed. So I built my own local AI asset pipeline.
The pipeline is split into four layers: a browser-based frontend, an API route for request handling, a ComfyUI client for workflow orchestration, and the local ComfyUI server running Flux.
Next.js + React + Tailwind. Handles prompt entry, model selection, and style presets.
app/api/ai-assets/2d/route.ts receives prompts and calls generateWithComfyUI().
lib/comfyui.ts builds workflow JSON, sends to /prompt, polls /history, fetches images.
Local GPU at http://10.0.0.184:8188. Flux UNet (GGUF), Dual CLIP, T5 encoder, VAE decoder.
User types prompt in browser
-> Next.js API route receives it
-> Workflow JSON is generated dynamically
-> Workflow is POSTed to ComfyUI /prompt
-> App polls /history for completion
-> Image is fetched from /view
-> Converted to base64
-> Displayed in browser
ComfyUI uses a node-based workflow system defined in JSON. Each node represents a step in the generation pipeline.
The key injection point is node 4 -- this is where the user's prompt gets inserted into the workflow at runtime. The rest of the nodes are statically defined in the template.
| Node | Type | Purpose |
|---|---|---|
| 1 | Load UNet | Loads the Flux GGUF model weights |
| 2 | Load Dual CLIP | Loads CLIP + T5 text encoders |
| 3 | Create Latent | Initializes empty latent image at target resolution |
| 4 | Encode Positive Prompt | Converts user prompt to CLIP embeddings |
| 5 | Encode Negative Prompt | Encodes what to avoid in generation |
| 6 | KSampler | Runs diffusion sampling (Euler, 20 steps, CFG 2.6) |
| 7 | Decode with VAE | Converts latent to pixel image |
| 8 | Load VAE | Loads the VAE decoder model |
| 9 | Save Image | Writes output to disk for retrieval |
The workflow is a JSON object where each key is a node ID. Node 4 receives the user prompt, and node 6 controls the sampling parameters.
1{
2 "4": {
3 "class_type": "CLIPTextEncode",
4 "inputs": {
5 "text": "{{ user prompt injected here }}",
6 "clip": ["2", 0]
7 }
8 },
9 "6": {
10 "class_type": "KSampler",
11 "inputs": {
12 "seed": {{ random seed }},
13 "steps": 20,
14 "cfg": 2.6,
15 "sampler_name": "euler",
16 "denoise": 1
17 }
18 }
19}
When a user submits a prompt, the API route injects the prompt text, a random seed, and target dimensions into the workflow JSON template.
1function buildWorkflow(prompt: string, seed: number, width: number, height: number) {
2 const workflow = structuredClone(FLUX_WORKFLOW_TEMPLATE);
3
4 // Inject user prompt into CLIP text encode node
5 workflow["4"].inputs.text = prompt;
6
7 // Set random seed for reproducibility
8 workflow["6"].inputs.seed = seed;
9
10 // Set output dimensions
11 workflow["3"].inputs.width = width;
12 workflow["3"].inputs.height = height;
13
14 return workflow;
15}
The built workflow is sent to ComfyUI's /prompt endpoint. ComfyUI returns a prompt_id that we use to track the generation job.
1const response = await fetch(`${{COMFYUI_URL}}/prompt`, {
2 method: 'POST',
3 headers: { 'Content-Type': 'application/json' },
4 body: JSON.stringify({ prompt: workflow })
5});
6
7const { prompt_id } = await response.json();
ComfyUI processes jobs asynchronously. We poll the /history endpoint until the job completes.
1async function waitForCompletion(promptId: string): Promise<any> {
2 while (true) {
3 const res = await fetch(`${{COMFYUI_URL}}/history/${{promptId}}`);
4 const data = await res.json();
5
6 if (data[promptId]) {
7 return data[promptId];
8 }
9
10 // Wait before polling again
11 await new Promise(resolve => setTimeout(resolve, 1000));
12 }
13}
Once complete, we extract the output filename from the history response and fetch the image from ComfyUI's /view endpoint.
1const outputs = history.outputs;
2const imageNode = outputs["9"]; // Save Image node
3const { filename, subfolder, type } = imageNode.images[0];
4
5const imageUrl = `${{COMFYUI_URL}}/view?filename=${{filename}}&subfolder=${{subfolder}}&type=${{type}}`;
6const imageResponse = await fetch(imageUrl);
The raw image bytes are converted to a base64 data URL for display in the browser without needing a separate static route.
1const imageBuffer = await imageResponse.arrayBuffer();
2const base64 = Buffer.from(imageBuffer).toString('base64');
3const dataUrl = `data:image/png;base64,${{base64}}`;
4
5return NextResponse.json({ image: dataUrl });
This project demonstrates more than AI image generation. It's a practical example of:
The Game Asset Creator bridges creative tooling, AI infrastructure, and full-stack engineering. It solves a real problem I face in game development -- generating consistent, high-quality 2D assets without the friction and cost of cloud APIs.
By running Flux locally through ComfyUI and wrapping it in a custom web interface, I've built a tool that fits naturally into my development workflow. It's fast to iterate with, cheap to run, and fully under my control.
More importantly, this project demonstrates how modern AI models can be integrated into practical development tools -- not as black-box APIs, but as configurable components in a larger engineering system.