Flux, LoRa & IPAdapter: Text2Img or Text+Img2Img

Introduction

This workflow empowers creators to achieve detailed style customization and artistic refinement. Ideal for advanced style exploration and creative inspiration, it’s perfect for generating high-quality, unique visuals. You can choose to use LoRA for text-to-image creation or combine it with IPAdapter to incorporate an image as a style reference alongside text for enhanced outputs.

IP-Adapter

IP-Adapter is a compact yet powerful tool enabling image prompts for pre-trained text-to-image diffusion models. With just 22M parameters, it matches or surpasses fine-tuned models, supports cross-model generalization, integrates with controllable generation tools, and combines image and text prompts for advanced multimodal image creation.

Read more: https://github.com/tencent-ailab/IP-Adapter

Download: https://github.com/tencent-ailab/IP-Adapter.git

Workflow overview

How to use this workflow?

Step 1: Upload an image as a style reference.

A reference image should be provided to guide the generation task (maintaining the colors or structure of the reference).

The IPAdapter module is used to adapt the generation style based on the input image.

clip_vision: Extracts visual features from the reference image for structural guidance during generation.

weight: Sets the influence of the reference image in the generation process (e.g., 0.4).

Step 2: Input Prompt

Use the CLIPTextEncodeFlux module to input descriptive text prompts that guide the artistic direction.

Step 3: Image Generation

After entering a prompt and upload a reference image, then click “Queue” to start generating. You’ll see a preview of the output image.

Step 4: Refinement and Upscaling

We’ve integrated Ultimate SD Upscale into this workflow to enhance image clarity and refine generated visuals.

Step 5: Export and Save

Preview the generated image and save the output in the desired resolution and format.