Flaming coiling serpent - Diablo IV Inferno spell [Flux] [Concept]版本v0.1 - 800 steps (ID: 1292099)

This LoRA was inspired by a contest, related to the upcoming Year of the Snake.

(More details about the contest here: “Call for Submissions: Year of the Snake Resources!”)

Inspirations and main idea

I was inspired by a flaming snake, Inferno spell from Diablo 4 game. I decided to create this concept based on some online publicly available images. Since the topic can be related to serpents of the Lunar New Year, I decided the first.

Dataset

My dataset is based on original images (6) that I downloaded from publicly available sources like Google and Bing image search. First, all images were augmented by adding horizontal flip. Then, I used a more advanced technique to create color augmentation and variations of the images, which utilized latent image encoding via VAE, with canny control net used to keep the correct shape.

More details on dataset augmentation

In order to do so, I used Control net (canny) - with Ksampler and SD 1.5 canny control net (v.1.1). The checkpoint - majicMIX realistic 麦橘写实 (v.7) got a latent input from VAE encode of the source image, which was used to add color augmentation to images. This allowed me to make more variations in color and expand the dataset from 6 to 16 images.

The workflow to create latent color augmentation used Ksampler (efficient) with following parameters:

• Sampler: heun

• Scheduler: Karras

• Steps: 10

• CFG: 1.0

• ControlNet strength — 1.2

The image I used to transfer the style via VAE encode in latents was not the same as the image used in control net (different one, the same images won't work).

I used LoRA tagging workflow with Florence 2 tagger, and resized images to 512×512 (WxH) and 256×256 as well. The final dataset consisted of 16×2=32 images, including flip and color augmentation.

Training workflow

Now to the training workflow. I used the official workflow from Kijai (GitHub - kijai/ComfyUI-FluxTrainer), based on Kohya script. I trained the LoRA with such settings — 64 images (including buckets), number of steps — 1000 (I found that the best result was at 200 and 800 steps, all others were seem to give less impressive results. So, based on my observations, these values translate to 3 and 15 epochs.

Now regarding the checkpoints used. I used Atomix FLUX Unet (v.1.0) for training, well, because it was only checkpoint in photorealistic style, I had in Unet format and FP8, so they may not fit training for photorealistic style. Regarding the training parameters — I used fp8 training format without offloading and b. More details are provided in the training workflow.

I generated the LoRA in few intervals — 200, 400, 500, 600, 800, 1000 steps. The 200 steps as well as 800 looks promising because of capturing the style I wanted.

LoRA deployment and testing

Now to deployment of the model. I tested it (and still testing to check any issues) using same Unet and Text encoder I used during training:

• clip-L from Hugging Face from Flux-dev repository: black-forest-labs/FLUX.1-dev at main

• T5xxl fp8 encoder FLUX.1 T5 Text Encoder

The best results so far I got with the following parameters:

• Lora model weight — 1.0

• Lora CLIP weight — 1.0

• Steps — 15

• CFG — 1.5

• Sampler: Euler

• Scheduler: simple

Since the LoRA was trained with tags from the initial training images, instead of trigger words you may use the tags section from the example prompt:

"Flaming coiling snake, fire, dark fantasy, Diablo IV, Inferno spell, magic, glowing, sorcerer, scales, large snake, neutral background, dark background, A digital illustration shoot from a bird's eye view about a fiery, serpent-like structure in the middle of a dark, rocky landscape. the structure is intricately designed with a textured surface, resembling a snake's skin, and is surrounded by a glowing, orange glow. in the center of the image, a small, humanoid figure appears to be a warrior, with a muscular build and a determined expression. the figure is positioned at the top of the structure, with the snake winding around it, creating a sense of movement and energy. the background is filled with a mix of dark and light tones, with hints of greenery and ruins, adding to the dramatic and intense atmosphere of the scene."

Credits

Thanks to the developers of mentioned models and ComfyUI nodes, for inspiration in prompting and workflows. All credits for used models and workflows left for the respective authors (AlexLai, Merjic, kijai). Thanks to authors of other awesome nodes, models and tools not mentioned here, but which were essential to create this image.

Disclaimer on content

Since the checkpoint is in early beta stage, it can generate some content, that is not for all audiences, if used alongside a checkpoint (e.g., dedistilled), if prompted. The LoRA does not depict a real person and serves only for testing purposes only.

Disclaimer on fair usage of training data

The training data (64 images) was created from 6 publicly available images, crawled from online search platforms as Google and Bing image search. The resulted outputs are not intended to replicate or imitate the Diablo IV video game footage or its content, except for artistic purposes like fan art and illustrations dedicated to the Diablo IV theme. The resulted images do not represent actual game footage and do not give any impressions about actual game.

Transformative work, including color augmentation, control net, distilling, filtering, resizing was made to make the output of the model less look-alike to the original images, downloaded from online image search platforms such as Bing and Google. The resulted model is intended to use for research purposes and has non-commercial license to distribute, create or recreate any content. All credits of the original images and footage from the Diablo IV video game are given to the authors of the images and to the authors of the Diablo IV video game (Blizzard inc.) respectively.

License

The LoRA inherits the license from Atomix Flux (used in training workflow as Unet):

FLUX.1 [dev] Non-Commercial License .

The FLUX.1 [dev] Model is licensed by Black Forest Labs. Inc. under the FLUX.1 [dev] Non-Commercial License. Copyright Black Forest Labs. Inc.

IN NO EVENT SHALL BLACK FOREST LABS, INC. BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH USE OF THIS MODEL.

描述:

Version with 800 steps. More details in the description.

训练词语: