Cog VideoX-1.5: Img2Video

Introduction:

This workflow is designed to create dynamic video outputs and upscale images using the CogVideoX pipeline in ComfyUI. By leveraging advanced tools like LoRA and specialized samplers, it ensures professional-grade results for creators focused on motion graphics or high-quality still image generation. Perfect for MimicPC users, this workflow allows customization while maintaining simplicity.

What is CogVideoX1.5 5B I2V

The CogVideoX1.5 5B I2V is an open-source model developed by Qingying which can generate vidoes from a single image using text prompts. Specifically, CogVideoX V1.5 features a 5-billion-parameter video model capable of producing 10-second, 768p resolution videos at 16 frames per second. This model excels in generating complex visuals with improved quality and supports integrated sound effects.

For more information:

CogVideoX1.5-5B-SAT

CogVideoX1.5-5B diffusers

To use CogVideoX 1.5 with ComfyUI, simply update ComfyUI and install the ComfyUI-CogVideoXWrapper plugin via the Git plugin manager. The model will be automatically downloaded on the first run. For manual installation, download all necessary files and place them in the ComfyUI/models/CogVideo directory:

Plugin repository:

ComfyUI-CogVideoXWrapper

Kijai/CogVideoX-5b-1.5 Model

Capabilitis

• Multimodal Understanding: The model can understand and process both images and text, allowing it to create coherent and contextually rich video narratives tha align with the given prompts.
• Flexible Creative Control: Users can provide a range of inputs, from simple descriptions to detailed visual cues, giving them control over the video's themes, style, and narrative direction.
• Time-Based Context: The model effectively understands temporal progression, ensuring that the generated video has a logical flow, from scene transitions to character actions, making it more natural and engaging.

Limitations

• Potential of Artifacts: In some cases, the video generation may result in visual artifacts, such as unnatural movements or inconsistencies between frames, particularly in more dynamic scenes.
• Dependence on Clear Prompts: The model's effectiveness is highly dependent on the clarity and detail of the input. Vague or ambiguous text prompts may lead to less desirable or unexpected video outcomes.

Workflow

How to Run This Workflow on MimicPC

1.Set Up the Workflow

Open the ComfyUI interface on MimicPC. Load the provided workflow and ensure the necessary models (CogVideoX-5B-1.5 , LoRA, and other dependencies) are downloaded.

2.Upload the Image or Video

Use the Load Image node for uploading still images, or prepare your initial video files if necessary. Then, set the input parameters, such as resolution, in the Resize Image node (e.g., 512x512 pixels).

3.Configure the CogVideoX Pipeline

In the CogVideo Sampler node:

• Samples: Adjust the sample frames for video creation.
• Scheduler: Use CogVideoXDPMScheduler for stable outputs.
• Denoise Strength: Set to 1.0 for sharper results.

Input prompts for camera movement and specify negative prompts for elements that you don't want in the final output video using the CogVideo TextEncode nodes:

Example prompts:

• Upper Column for Camera Movement Prompts: The camera slowly pans from left to right, starting at a desk cluttered with papers and ending on the two men sitting in front of the computer.
• Bottom Column for Negative Prompts: low quality, artifacts, glitchy frames, inconsistencies

4.Save the Output

To download the final video, simply set save_output to true, and the final outcome will be saved in the output file as shown below.