Illustrious × Pony Mi×

v3 is the best version of the merge by far! Improvements:

There is now a default style, which looks good even if you don't use pony score tags, aesthetic prompts, LoRAs or artist tags! The first preview image uses none of these. Just a basic positive prompt and negative prompt.
Because of the above LoRA compatibility is much better.
Even less noise and latent garbling than v2.
Better prompt-adherence, quality and anatomy.
The model now has a NAI3-like clarity and details.
Backgrounds still need further improvement.

Previous updates:

v2 is a significant improvement over the original model, with less latent noise and better anatomy, including hands and fingers.

This is a merge of my fine-tune of illustrious, with my fine-tune of a Pony-base model.

Works with pony scoretags, rating tags and Illustrious artist tags. Pony score tags, which are optional and activate Pony-mode, should be at the head of your prompt, followed by the artist.

Effect of

score_9,score_8_up,score_7_up

at different strengths:

If you do not specify the artist, ~~the default style looks like crap because I did not use caption dropout in the final adjustment fine-tune~~ the results look great!

Some kind words are appreciated, it was a massive pain to merge the two models. This output isn't perfect but does complex scenes really well and to me, has a certain charm, just like the base Illustrious model.

General method:

First step was to use train difference and comparative interpolation to merge the models. These two models are then merged normally. The result is noisy and greyish but actually contains the properties and knowledge of both models. This is where I fine tuned the model on a dataset of 400,000 images for one epoch to stabilise it. I then merged a set of special LoRAs which bring out features muted by the merge. This is followed by fine tuning another model on the same data for 2 epochs - this model when converted into a LoRA and applied at negative strength significantly improves anatomy/fingers/noise. This was then merged to make the v2 model linked.

The result is that pony score tags and rating tags work, and so do the illustrious artist tags. The detail of the original Illustrious model is also boosted. Using pony prompts recalled the same kinds of images I used with the Pony fine-tune I merged in, confirming the concepts transferred through.

Tools:

https://github.com/silveroxides/sd-webui-untitledmerger

https://github.com/hako-mikan/sd-webui-supermerger/issues/408

https://github.com/Linaqruf/kohya-trainer/blob/main/Kohya%20Trainer%20XL%20Runpod.ipynb