Stable Diffusion Local Setup: RTX 3090 vs Apple Silicon

Let's start with the honest numbers.

Midjourney Basic: $10/month, 200 images. Standard: $30/month, unlimited relaxed. Pro: $60/month, unlimited fast. You're paying per generation, the company owns your prompts in its training data, and when they decide to raise prices or shut down, your workflow disappears with them.

Stable Diffusion locally: hardware cost up front. After that? Zero per image. Zero per month. No account, no email, no prompts logged anywhere, no content policy that decides your professional photography mock-up with a cigar violates their terms.

Break-even on a used RTX 3090 (24GB VRAM, under $500 on eBay in 2026) versus Midjourney Standard: roughly 16 months. After that you're in pure savings, running generations 24/7 if you want, with models Midjourney can't compete with for specific styles.

This guide covers the complete stack from hardware selection through professional workflows. If you want the underlying theory on how diffusion models actually work, and the math behind the denoising process, read that separately — this article is about making the thing run well.

What's Actually Running in 2026

The landscape shifted significantly from where it was 18 months ago. Here's what's actually relevant:

FLUX.1 (Black Forest Labs) is the current king of open-source image generation. FLUX.1 Dev produces Midjourney v6-quality output with significantly better text rendering and photorealism. FLUX.1 Schnell generates in 1-4 steps versus SDXL's 20-50, making it dramatically faster for iteration. Black Forest Labs is the same team that created Stable Diffusion — they left Stability AI and built FLUX with $300M in funding and a $3.25B valuation. The model quality shows it.

SDXL (Stable Diffusion XL) remains the workhorse. The ecosystem is massive — thousands of LoRAs, ControlNets, upscalers, fine-tuned checkpoints. If you need a specific art style or aesthetic that someone has already trained, it's in SDXL format on CivitAI. Juggernaut XL is the community's most popular photorealistic base. DreamShaper XL for artistic styles. RealVisXL for portraits. The ecosystem depth is something FLUX hasn't caught up to yet.

SD 1.5 — still alive, still useful for low VRAM setups, still the target format for a ton of community models. If you have 4-6GB VRAM, this is your primary option.

HunyuanVideo / Wan 2.1 — AI video generation that's genuinely impressive in 2026. Needs 60GB+ VRAM for standard quality. Not practical on consumer hardware without heavy compromise, but worth knowing it exists.

Pick your tier and we'll configure accordingly.

Do you like what you are reading so far? Take a look at our Your Own Private AI: The Complete 2026 Guide to Running a Local LLM on Your PC article next for more reading from us!

Your Own Private AI: The Complete 2026 Guide to Running a Local LLM on Your PC

Everything you need to run a capable, private, offline AI assistant or coding copilot on your own hardware — from picking your model to wiring it into VS Code — with zero cloud, zero API bills, and zero code leaving your machine.

CoderOasisTraven

Hardware Requirements

For Apple Silicon (M1 Pro, M2, M3 with 16GB+ unified memory): ComfyUI's Metal backend gives you near-discrete-GPU speed. A 16GB M3 MacBook Pro handles SDXL comfortably and FLUX FP8 with patience. The 32GB M3 Max handles everything except video models. Apple Silicon is a legitimate option and not a second-class citizen anymore.

For AMD GPUs on Linux with ROCm properly configured: near-CUDA performance. AMD on Windows with DirectML is 30-50% slower than equivalent NVIDIA. If you're on Windows with AMD, consider a Linux dual-boot for your generation setup.

The VRAM ceiling matters for FLUX. SDXL fits in 12GB. FLUX.1 Dev in BF16 wants 24GB. FLUX.1 Dev in FP8 fits in ~12GB with minor quality reduction. FLUX.1 Dev in GGUF Q4 fits in 6GB with more quality reduction. Pick your model tier based on your VRAM.

Choose Your Frontend

Three frontends dominate in 2026. Pick based on your needs:

ComfyUI — node-based workflow graph. Steeper learning curve, maximum flexibility. Most professional and advanced users run ComfyUI. Workflows are shareable as JSON files. Multi-model pipelines (inpaint → upscale → face-fix → save) are natural to express as connected nodes. This is the right choice if you're doing production work, building pipelines, or want the full capability set.

Forge (A1111 fork) — form-based web UI. Lowest barrier to entry. Better VRAM management than original A1111. Right choice if you want to get generating quickly without learning a node graph. The extension ecosystem is massive.

Fooocus — dead simple, focused on Midjourney-like UX. Hides all the technical complexity. Good for quick creative exploration, bad for reproducible production workflows.

For this guide we'll install ComfyUI because it's what you'll end up on if you take image generation seriously. We'll also cover Forge for the immediate-results path.

Install ComfyUI

Prerequisites: Python 3.10 or 3.11 (critical — 3.12 breaks several extensions), Git.

# Clone ComfyUI
git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
 
# Create and activate virtual environment
python3.10 -m venv venv
source venv/bin/activate          # Linux/macOS
# venv\Scripts\activate           # Windows
 
# Install PyTorch — pick the right variant for your hardware
# NVIDIA CUDA 12.1 (most common):
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
 
# AMD ROCm (Linux only):
# pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.7
 
# Apple Silicon / CPU:
# pip install torch torchvision torchaudio
 
# Install ComfyUI dependencies
pip install -r requirements.txt
 
# Launch
python main.py
# For NVIDIA low VRAM (under 8GB):
# python main.py --lowvram
# For Apple Silicon:
# python main.py --force-fp16

Open http://localhost:8188 in your browser. You'll see an empty node canvas. Don't panic — we'll put the workflow files in before you have to learn the node system.

Install ComfyUI Manager (do this immediately — it's essential for extension management):

cd custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Manager
cd ..

Restart ComfyUI. You'll see a "Manager" button in the top right. Use this to install additional nodes, models, and extensions rather than doing it manually.

Download Your Models

Models go in specific subdirectories of ComfyUI/models/. The structure matters.

For SDXL:

cd ComfyUI/models/checkpoints/
 
# Juggernaut XL — best photorealistic SDXL base in 2026
# Download from CivitAI: https://civitai.com/models/133005
# Or via Hugging Face:
wget https://huggingface.co/RunDiffusion/Juggernaut-XL-v9/resolve/main/Juggernaut-XL_v9_RunDiffusionPhoto_v2.safetensors
 
# SDXL VAE (required for best quality)
cd ../vae/
wget https://huggingface.co/stabilityai/sdxl-vae/resolve/main/sdxl_vae.safetensors

For FLUX.1:

FLUX has a different model structure than SDXL — it uses separate files for the transformer, text encoders, and VAE.

# Create FLUX-specific directory
mkdir -p ComfyUI/models/unet/
 
# FLUX.1 Dev transformer (24GB VRAM — full quality)
# You need Hugging Face access and to accept the license at black-forest-labs/FLUX.1-dev
huggingface-cli download black-forest-labs/FLUX.1-dev flux1-dev.safetensors \
  --local-dir ComfyUI/models/unet/
 
# FLUX.1 Dev FP8 (12GB VRAM — minimal quality loss)
wget https://huggingface.co/Kijai/flux-fp8/resolve/main/flux1-dev-fp8.safetensors \
  -O ComfyUI/models/unet/flux1-dev-fp8.safetensors
 
# FLUX.1 Schnell (faster, 1-4 step generation, same VRAM)
huggingface-cli download black-forest-labs/FLUX.1-schnell flux1-schnell.safetensors \
  --local-dir ComfyUI/models/unet/
 
# Text encoders (same for all FLUX models)
mkdir -p ComfyUI/models/clip/
huggingface-cli download comfyanonymous/flux_text_encoders \
  clip_l.safetensors t5xxl_fp8_e4m3fn.safetensors \
  --local-dir ComfyUI/models/clip/
 
# FLUX VAE
mkdir -p ComfyUI/models/vae/
wget https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/ae.safetensors \
  -O ComfyUI/models/vae/flux_ae.safetensors

Low VRAM option — FLUX GGUF (runs on 6-8GB):

# FLUX.1 Dev GGUF Q4 — 6GB VRAM minimum
wget https://huggingface.co/city96/FLUX.1-dev-gguf/resolve/main/flux1-dev-Q4_K_S.gguf \
  -O ComfyUI/models/unet/flux1-dev-Q4_K_S.gguf

For GGUF models, install the ComfyUI GGUF node via Manager, then use the "Load Diffusion Model" GGUF node instead of the standard model loader.

Your First SDXL Workflow

ComfyUI ships with example workflows. Load the default SDXL one from the menu, or build it manually. Here's the essential node structure — I'll describe connections so you can rebuild it if needed:

The core SDXL generation graph:

[Load Checkpoint] → model, clip, vae
     ↓ model
[KSampler] ← seed, steps (30), cfg (7.0), sampler (euler), scheduler (karras)
     ↓ latent
[VAE Decode] ← vae (from Load Checkpoint)
     ↓ image
[Save Image]
 
[CLIP Text Encode] (positive prompt) ← clip → [KSampler] ← positive
[CLIP Text Encode] (negative prompt) ← clip → [KSampler] ← negative
[Empty Latent Image] (width 1024, height 1024, batch 1) → [KSampler] ← latent_image

Key settings that matter:

• Steps: 25-35 for quality, 15-20 for iteration speed. SDXL doesn't improve much past 35 steps.
• CFG Scale: 6-8 is the sweet spot for SDXL. Below 5 gets weird. Above 10 causes artifacts and color burns.
• Sampler: euler or dpmpp_2m are reliable. dpmpp_2m karras is a good default.
• Negative prompt: For photorealism, (worst quality, low quality:1.4), blurry, deformed, extra limbs, text, watermark is a solid baseline. SDXL is less sensitive to negative prompts than SD 1.5 was, but it still helps.

The FLUX Workflow

FLUX's node structure is more complex because of the separate model components. Here's the connection pattern:

[Load Diffusion Model] (flux1-dev-fp8.safetensors) → model
[DualCLIPLoader] (clip_l, t5xxl_fp8) → clip
[VAELoader] (flux_ae) → vae
 
[CLIPTextEncode] ← clip → conditioning
[EmptySD3LatentImage] (width 1024, height 1024) → latent
 
[KSamplerAdvanced]
  ← model
  ← conditioning (positive)
  ← latent
  sampler: euler
  scheduler: simple
  steps: 28
  cfg: 3.5
  ↓ latent
 
[VAEDecode] ← vae → image
[SaveImage]

FLUX-specific settings that differ from SDXL:

• CFG: 3.5-4.5. FLUX doesn't need or want high CFG. The model follows prompts more precisely than SDXL, so you don't need to push it hard with high guidance.
• Steps: 25-35 for Dev. 4-8 for Schnell. Schnell is genuinely good at 4 steps for fast iteration — the adversarial distillation training means it converges much faster.
• No negative prompt. FLUX doesn't use negative prompts in the traditional way. The model's guidance mechanism is different. You can include one, but it has minimal effect.
• Scheduler: simple works well. euler is also solid.
  For the GGUF low-VRAM workflow, swap [Load Diffusion Model] for the GGUF-specific node and point it at your .gguf file. Everything else stays the same.

ControlNet — Because Random Generation Gets Old Fast

ControlNet is what separates "cool toy" from "production tool." It lets you constrain generations using reference images — pose references for character consistency, edge maps for layout control, depth maps for lighting matching, inpainting for precise area replacement.

Install ControlNet nodes:

Through ComfyUI Manager → Install Custom Nodes → search "ControlNet" → install ComfyUI's ControlNet Auxiliary Preprocessors

Download ControlNet models:

mkdir -p ComfyUI/models/controlnet/
 
# Canny (edge detection) — workhorse for layout control
wget https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0/resolve/main/diffusion_pytorch_model.fp16.safetensors \
  -O ComfyUI/models/controlnet/controlnet-canny-sdxl.safetensors
 
# Depth — for matching lighting and 3D structure
wget https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0/resolve/main/diffusion_pytorch_model.fp16.safetensors \
  -O ComfyUI/models/controlnet/controlnet-depth-sdxl.safetensors
 
# OpenPose — for pose control in character images
wget https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/resolve/main/OpenPoseXL2.safetensors \
  -O ComfyUI/models/controlnet/controlnet-pose-sdxl.safetensors
 
# For FLUX: use XLabs ControlNet (the community FLUX ControlNet implementation)
wget https://huggingface.co/XLabs-AI/flux-controlnet-canny/resolve/main/flux-canny-controlnet.safetensors \
  -O ComfyUI/models/controlnet/flux-canny-controlnet.safetensors

SDXL Canny workflow addition:

Add these nodes between your image source and KSampler:

[Load Image] (your reference image)
     ↓
[Canny Edge Preprocessor] (low_threshold: 100, high_threshold: 200)
     ↓ image
[Apply ControlNet] ← controlnet model ← strength (0.6-0.85) ← conditioning
     ↓ conditioning → [KSampler]

Start at strength 0.7. Higher = tighter adherence to the control image structure. Lower = more creative freedom. 0.5-0.85 is the useful range for most use cases.

LoRA Training

Pre-trained models are generic. LoRAs are how you teach them your specific needs — a consistent character face, your brand's art style, a product line's visual language, your photography aesthetic.

What is a LoRA technically? Low-Rank Adaptation. Instead of fine-tuning the entire model (billions of parameters), LoRA adds small adapter matrices to specific layers and trains only those adapters. A full SDXL model is 6-7GB. A LoRA fine-tune for a specific style or character is 50-300MB. Training a LoRA takes hours on a consumer GPU instead of weeks on a cluster.

Quick setup with kohya-ss:

git clone https://github.com/kohya-ss/sd-scripts
cd sd-scripts
 
python -m venv venv
source venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
 
# Prepare your training images
# - 15-30 images for a character or style
# - 50-100 images for higher quality
# - Captions in .txt files alongside each image
# - Images cropped to 1024x1024 for SDXL
 
mkdir -p training_data/my_style/
# Place your images and caption files here

Caption format matters. Each image needs a .txt file with the same name:

training_data/my_style/photo_001.jpg
training_data/my_style/photo_001.txt  → "photo of person, my_trigger_word, outdoor lighting, casual clothing"

The trigger word — my_trigger_word in this example — is what you'll use in prompts later to activate the LoRA's learned style. Pick something distinctive that the base model won't confuse with existing concepts.

Training command for SDXL LoRA:

accelerate launch train_network.py \
  --pretrained_model_name_or_path="/path/to/sd_xl_base_1.0.safetensors" \
  --train_data_dir="./training_data/my_style/" \
  --output_dir="./output/" \
  --output_name="my_style_lora" \
  --save_model_as=safetensors \
  --sdxl \
  --network_module=networks.lora \
  --network_dim=32 \
  --network_alpha=16 \
  --max_train_steps=1500 \
  --learning_rate=0.0001 \
  --unet_lr=0.0001 \
  --text_encoder_lr=0.00005 \
  --train_batch_size=1 \
  --mixed_precision=fp16 \
  --xformers \
  --gradient_checkpointing \
  --cache_latents

Key parameters to understand:

• --network_dim=32 — the rank of the LoRA adapter matrices. Higher = more capacity to learn, larger file size, more VRAM during training. Start at 32, go to 64 if quality is insufficient.
• --max_train_steps=1500 — for 20-30 images. Scale roughly with dataset size. More steps = more fitting, but too many steps causes overfitting.
• --learning_rate — if generations look burned-in and lack variation, lower this. If the style isn't sticking, raise it slightly.
  The LoRA file lands in ./output/my_style_lora.safetensors. Drop it in ComfyUI/models/loras/ and load it via the "Load LoRA" node in your workflow at strength 0.7-0.9.

The Full Production Workflow

This is what separates casual use from professional output. A generation pipeline that takes a rough draft and turns it into something printable.

Stage 1: Text-to-image — Generate your base image at 1024×1024. Don't stress about perfect quality at this stage. Iterate fast, get the composition right.

Stage 2: Img2img refinement — Feed the best result back through img2img at denoise 0.4-0.6. The model re-draws the image with the constraints of the original, fixing artifacts and tightening quality.

Stage 3: Upscale — Use Ultimate SD Upscale or ESRGAN to 2x or 4x resolution. Install via ComfyUI Manager → "UltimateSDUpscale" node. This does a tiled upscale where each tile gets refined during the upscale pass, giving you sharp detail at high resolution instead of blurry enlargement.

[Load Image] (your 1024px base)
     ↓
[Ultimate SD Upscale]
  ← model, clip, vae (from checkpoint loader)
  ← positive/negative conditioning
  upscale_by: 2
  tile_width: 512
  tile_height: 512
  tile_overlap: 32
  mode: Linear
  seam_fix_mode: Half tile
  denoise: 0.3
     ↓
[Save Image] (2048px output)

Stage 4: Face fix (for portraits) — If you're generating people, faces often need correction at high resolutions. Install the "Impact Pack" via ComfyUI Manager, then use the SEGS + Face Detailer nodes to isolate and re-generate face regions at full quality.

Performance Tuning

A few things that actually move the needle:

xformers — Memory-efficient attention for NVIDIA GPUs. Reduces VRAM usage by 15-30% and speeds generation slightly. Install: pip install xformers and launch ComfyUI with --use-xformers. Check compatibility with your PyTorch/CUDA version.

Compilation (for repeated generation) — PyTorch 2.0+ supports torch.compile() which JIT-compiles the model on first use. ComfyUI supports this via --force-upcast-attention --use-pytorch-cross-attention. The first generation is slow (30-90 seconds for compilation), every subsequent generation in the session is faster.

SDXL turbo / LCM LoRAs — Latent Consistency Model LoRAs let you run SDXL at 4-8 steps instead of 25-35 with acceptable quality. Good for rapid iteration. Quality ceiling is lower than full-step generation, but the speed is 4-5x faster.

Queue depth — ComfyUI queues generations asynchronously. If you're doing batch work (generating 100 variations for a project), queue them all and let ComfyUI chew through the batch. The GPU utilization between queued generations is near zero overhead.

The Local vs. Cloud Math in 2026

I said it at the start and I'll say it again with the actual numbers:

If you're using Midjourney Standard ($30/month), you hit break-even on a used RTX 3090 ($350-450 on eBay) in 12-15 months. Every month after that is free. You're generating unlimited images, at higher quality than Midjourney for specific use cases, with no prompts logged, no content filters, and no dependency on a company's pricing decisions.

If you're on Midjourney Pro ($60/month), break-even on a new RTX 4080 is 10-12 months.

The math matters less if you're a casual user who generates 20 images a month. It matters a lot if you're a designer, photographer, or developer who generates hundreds or thousands.

We covered the GPU shortage and CPU bottleneck hitting AI infrastructure — understanding that context explains why cloud AI image generation prices are not decreasing as fast as the models are improving. The hardware scarcity is real. Owning your compute locks in your cost basis regardless of what happens to GPU availability.

And for the developers who've been following the Claude Code source leak situation — the same principle applies here. Your prompts and generations are your IP. Local generation means they stay local.

Troubleshooting Common Failures

"CUDA out of memory" — You need --lowvram or --medvram flag at launch. Or drop to a lower quantization of your model. For FLUX specifically, switch from Dev FP16 to FP8, or to the GGUF Q4 variant.

Python version errors on launch — ComfyUI specifically needs 3.10 or 3.11. Use pyenv to manage: pyenv install 3.10.14 && pyenv local 3.10.14 then re-run install.

"Error loading model" — File is in the wrong directory, or corrupted download. Verify the model file size matches the expected size on Hugging Face. Re-download if there's a mismatch.

Generations look washed out — Missing or wrong VAE. SDXL and FLUX both have specific VAEs that need to match the model. Load the correct VAE via the VAELoader node.

FLUX images are greenish/wrong colors — Using the wrong VAE. FLUX requires flux_ae.safetensors, not the SDXL VAE. This is the most common FLUX setup mistake.

Slow generation on Apple Silicon — You need --force-fp16 to use the Metal backend efficiently. Default precision on Apple Silicon is FP32 which is much slower.

Related reading from CoderOasis: Running local LLMs on your PC — if you're setting up a local AI stack, combine this with a local coding assistant for a fully offline AI setup. The CPU shortage hitting AI infrastructure provides the hardware market context for why owning your GPU makes economic sense right now.