Detailed explanation of the lighting artifact IC-Light! Can control light direction and color atmosphere, free to use

llyasviel, the author of Controlnet, SD WebUI Forge and LayerDiffusion, recently launched a new project IC-Light, which can control the lighting direction and tone atmosphere of an image. Today we will learn about its characteristics and usage.

1. Introduction to IC-Light

IC-Light project homepage: https://github.com/lllyasviel/IC-Light

IC-Light is the abbreviation of "Imposing Consistent Light". It is a lighting control project that currently supports lighting control of foreground content through text and background images, so that the foreground subject is consistent with the background environment lighting, thereby allowing The two merge into one.

As can be seen from the official screenshots, IC-Light can control the tone atmosphere and lighting direction. If the prompt word says "sunshine outside the window", then the characters in the processed image will have obvious light and shadow, and the overall tone will turn into a warm orange; if the prompt word says neon lights, the picture will become Dark red and blue light. The lighting direction can be controlled through the Lighting Preference option, supporting left light, right light, bottom light and top light. Even if the light and shadow in the original image are very complex, IC-Light can re-light it.

2. How to use

1. Online experience

Direct URL: https://huggingface.co/spaces/lllyasviel/IC-Light

Huggingface already has an online demo of IC-Light, so you can directly experience its lighting control function. The usage is also very simple. Upload a character image, use text to describe the new background content and tone atmosphere, then set the direction of lighting in Lighting Preference, and finally click Relight to generate.

2.ComfyUI

IC-Light native plug-in: https://github.com/huagetai/ComfyUI-Gaffer

Model download: https://huggingface.co/lllyasviel/ic-light/tree/main

ComfyUI currently has multiple plug-ins that implement IC-Light. Here I use the native plug-in ComfyUI-Gaffer. We can install it through git clone or comfyui manager.

After the plug-in is installed successfully, you need to download 3 IC-Light models (resource packages are at the end of the article) and install them in the ComfyUI/models/iclight folder. The three models are currently only applicable to SD 1.5 models, and their functions are:

1. iclight_sd15_fc.safetensors: The default relight mode, based on text and foreground. You can use the initial latent image to affect the relight effect. Use it in your FG workflow.

2. iclight_sd15_fcon.safetensors: Same as "iclight_sd15_fc.safetensors", but uses offset noise for training. Use it in your FG workflow.

3. iclight_sd15_fbc.safetensors: Relighting model conditioned on text, foreground and background. Use it in your BG workflow.

After the above steps are completed, restart ComfyUI, and then import the 6 official workflow files from the example folder in the root directory of the ComfyUI Gaffer plug-in, and then you can use it normally. The relevant parameters and usage of the IC-Light node are described in detail on the Github homepage of ComfyUI Gaffer. You can read it carefully before using it. The functions of the 6 workflows are:

① workflow-gaffer-iclight-fc-text: Text generates background and modifies the lighting of the given foreground to blend it into one.

② workflow-gaffer-iclight-fc: Text generation background + manually set 8 lighting directions, modify the lighting of the given foreground, and make it blend into one.

③ workflow-gaffer-iclight-fbc: Upload a given background and modify the lighting of the given foreground to blend it into one.

④ workflow-gaffer-iclight-fc-normal: Generate different lighting environments in batches.

⑤ workflow-gaffer-iclight-fc-animated: Generate dynamic lighting animation

⑥ workflow-gaffer-iclight-all: Upload background + set light direction + batch generation function collection

3. SD WebUI Forge

Plug-in link: https://github.com/huchenlei/sd-forge-ic-light

Model download: https://huggingface.co/huchenlei/IC-Light-ldm/tree/main

In addition to ComfyUI, lllyasviel's self-developed SD WebUI Forge also has an IC-Light plug-in, which can be installed through the URL in "extension". Then download the two models clight_sd15_fc_unet_ldm (for foreground) and iclight_sd15_fbc_unet_ldm (for background) and install them to the root directory of stable-diffusion-webui-forge/models/unet (create one manually if there is no unet folder).

SD WebUI Forge also supports generating backgrounds from text, controlling lighting direction, and uploading background images to affect a given foreground. have to be aware of is:

1. If you upload the background image yourself, you need to convert the lighting model from FC to FBC;

2. The CFG value should be lowered, usually 2;

3. When controlling the lighting direction, the plug-in automatically generates a grayscale image in the image section. The CFG here also needs to be adjusted to 2, and the redraw amplitude needs to be increased to 1.

The above is the relevant introduction to the lighting control plug-in IC-Light in this issue. I hope it can help everyone. I will introduce more AI content for you in the future.