GAME ASSET CREATOR

AI-Powered 2D Game Asset
Generation

A fully local AI pipeline that turns text prompts into game-ready sprites and concept art, powered by Flux running on ComfyUI.


Why I Built This

I'm building a 2D 32-bit game in Unity and kept running into the same problems: generating consistent character sprites, creating sprite sheets, iterating quickly on concept art, and maintaining a cohesive visual style across assets.

Hosted AI APIs like DALL-E and Midjourney were too expensive for rapid experimentation and didn't give me the control I needed. So I built my own local AI asset pipeline.

  • 2D character sprites and background assets
  • Concept art and visual references
  • 32-bit retro-style aesthetics
  • Rapid iteration without token costs
  • Full creative control over generation parameters
The Solution
Flux (Local)
GGUF quantized model on local GPU
ComfyUI
Node-based inference engine
Next.js Interface
Custom web frontend for prompts & results
Reusable Workflow Pipeline
Extensible JSON workflow templates

System Overview

The pipeline is split into four layers: a browser-based frontend, an API route for request handling, a ComfyUI client for workflow orchestration, and the local ComfyUI server running Flux.

1. Frontend

Next.js + React + Tailwind. Handles prompt entry, model selection, and style presets.

2. API Route

app/api/ai-assets/2d/route.ts receives prompts and calls generateWithComfyUI().

3. ComfyUI Client

lib/comfyui.ts builds workflow JSON, sends to /prompt, polls /history, fetches images.

4. ComfyUI Server

Local GPU at http://10.0.0.184:8188. Flux UNet (GGUF), Dual CLIP, T5 encoder, VAE decoder.

Request Flow
User types prompt in browser
  -> Next.js API route receives it
  -> Workflow JSON is generated dynamically
  -> Workflow is POSTed to ComfyUI /prompt
  -> App polls /history for completion
  -> Image is fetched from /view
  -> Converted to base64
  -> Displayed in browser

The Flux Workflow

ComfyUI uses a node-based workflow system defined in JSON. Each node represents a step in the generation pipeline.

The key injection point is node 4 -- this is where the user's prompt gets inserted into the workflow at runtime. The rest of the nodes are statically defined in the template.

9-Node Pipeline
Node Type Purpose
1 Load UNet Loads the Flux GGUF model weights
2 Load Dual CLIP Loads CLIP + T5 text encoders
3 Create Latent Initializes empty latent image at target resolution
4 Encode Positive Prompt Converts user prompt to CLIP embeddings
5 Encode Negative Prompt Encodes what to avoid in generation
6 KSampler Runs diffusion sampling (Euler, 20 steps, CFG 2.6)
7 Decode with VAE Converts latent to pixel image
8 Load VAE Loads the VAE decoder model
9 Save Image Writes output to disk for retrieval

Workflow JSON Structure

The workflow is a JSON object where each key is a node ID. Node 4 receives the user prompt, and node 6 controls the sampling parameters.

JSON
 1{
 2  "4": {
 3    "class_type": "CLIPTextEncode",
 4    "inputs": {
 5      "text": "{{ user prompt injected here }}",
 6      "clip": ["2", 0]
 7    }
 8  },
 9  "6": {
10    "class_type": "KSampler",
11    "inputs": {
12      "seed": {{ random seed }},
13      "steps": 20,
14      "cfg": 2.6,
15      "sampler_name": "euler",
16      "denoise": 1
17    }
18  }
19}

Step-by-Step: How the Code Works

1

Build the Workflow

When a user submits a prompt, the API route injects the prompt text, a random seed, and target dimensions into the workflow JSON template.

TypeScript
 1function buildWorkflow(prompt: string, seed: number, width: number, height: number) {
 2  const workflow = structuredClone(FLUX_WORKFLOW_TEMPLATE);
 3
 4  // Inject user prompt into CLIP text encode node
 5  workflow["4"].inputs.text = prompt;
 6
 7  // Set random seed for reproducibility
 8  workflow["6"].inputs.seed = seed;
 9
10  // Set output dimensions
11  workflow["3"].inputs.width = width;
12  workflow["3"].inputs.height = height;
13
14  return workflow;
15}
2

Queue the Prompt

The built workflow is sent to ComfyUI's /prompt endpoint. ComfyUI returns a prompt_id that we use to track the generation job.

TypeScript
1const response = await fetch(`${{COMFYUI_URL}}/prompt`, {
2  method: 'POST',
3  headers: { 'Content-Type': 'application/json' },
4  body: JSON.stringify({ prompt: workflow })
5});
6
7const { prompt_id } = await response.json();
3

Poll for Completion

ComfyUI processes jobs asynchronously. We poll the /history endpoint until the job completes.

TypeScript
 1async function waitForCompletion(promptId: string): Promise<any> {
 2  while (true) {
 3    const res = await fetch(`${{COMFYUI_URL}}/history/${{promptId}}`);
 4    const data = await res.json();
 5
 6    if (data[promptId]) {
 7      return data[promptId];
 8    }
 9
10    // Wait before polling again
11    await new Promise(resolve => setTimeout(resolve, 1000));
12  }
13}
4

Fetch the Image

Once complete, we extract the output filename from the history response and fetch the image from ComfyUI's /view endpoint.

TypeScript
1const outputs = history.outputs;
2const imageNode = outputs["9"]; // Save Image node
3const { filename, subfolder, type } = imageNode.images[0];
4
5const imageUrl = `${{COMFYUI_URL}}/view?filename=${{filename}}&subfolder=${{subfolder}}&type=${{type}}`;
6const imageResponse = await fetch(imageUrl);
5

Convert to Base64

The raw image bytes are converted to a base64 data URL for display in the browser without needing a separate static route.

TypeScript
1const imageBuffer = await imageResponse.arrayBuffer();
2const base64 = Buffer.from(imageBuffer).toString('base64');
3const dataUrl = `data:image/png;base64,${{base64}}`;
4
5return NextResponse.json({ image: dataUrl });

Technical Highlights

Fully local inference
No cloud APIs, no token costs, no data leaving the network
Modular workflow system
Swap models or add post-processing by editing JSON templates
Network-aware architecture
GPU machines serve inference for lighter client devices over LAN
Model-level control
Euler sampler, CFG 2.6, 20 steps, 768x1024, full denoise
GGUF model support
Quantized Flux UNet for efficient VRAM usage on consumer GPUs

Challenges Solved

macOS local network permissions
Handling firewall rules and binding to the correct network interface
Async generation handling
Polling-based approach with proper timeout and error handling
Parsing workflow outputs
Deeply nested output structures that vary by workflow configuration
GGUF support for Flux UNet
Specific ComfyUI custom nodes and careful model loading for VRAM limits

Why This Matters

This project demonstrates more than AI image generation. It's a practical example of:

  • Full-stack AI infrastructure -- integrating a local inference server with a modern web app
  • Local model hosting -- running production-quality AI on consumer hardware
  • Microservice architecture -- separating inference from the application layer
  • Async workflow orchestration -- managing long-running AI jobs with polling and recovery
  • Cost-efficient AI deployment -- eliminating per-request API costs

Future Improvements

Sprite sheet automation -- consistent multi-frame sprite sheets from a single prompt
Animation frame generation -- walk cycles, attack animations, idle poses
Background removal -- automatic alpha channel extraction
Model comparison -- same prompt across multiple Flux checkpoints
Seed locking -- consistent character designs across generations
Asset metadata storage -- track prompts, seeds, and parameters
Direct Unity export -- Unity-compatible formats with import settings

Conclusion

The Game Asset Creator bridges creative tooling, AI infrastructure, and full-stack engineering. It solves a real problem I face in game development -- generating consistent, high-quality 2D assets without the friction and cost of cloud APIs.

By running Flux locally through ComfyUI and wrapping it in a custom web interface, I've built a tool that fits naturally into my development workflow. It's fast to iterate with, cheap to run, and fully under my control.

More importantly, this project demonstrates how modern AI models can be integrated into practical development tools -- not as black-box APIs, but as configurable components in a larger engineering system.

Tech Stack
ComfyUI
Flux
Laravel
Vue 3
Python
Next.js
Tailwind
Local GPU
Want to learn more?
Check out my other projects or get in touch to discuss AI-powered development workflows.
Get in touch
Kevin Morales
Kevin Morales
Full-Stack Developer