Escape XL (Pony)版本v0.1-lightning_beta (ID: 461404)

I hope I'm not too late to the party for the scenery contest...

Introducing my new EscapeXL model! This checkpoint is a fine-tuning of PonyXL designed to restore its ability to create stunning scenery and detailed landscapes as well as integrate well with your characters. It was originally trained as a LoRA, but a standard fine tuning with Lightning weights delivered way better results. It has been trained on roughly 200 pictures.

It's also designed to work with your favorite PonyXL LoRAS!

This model is currently in beta. It's still not perfect but available as early release for the landscape contest as it works well for scenery but just okay for characters and NSFW. If you guys like it, I'll train a v1.0 with more than 1000 images, new concepts and probably better natural language support to describe your best landscapes and scenery.

To-do list for v1.0:

More biomes: space, underwater, interiors, cities...
More concepts not involving humans: vehicles, architecture, objects...
Possibly better natural language support for describing your scenery as well as Booru tags.
Fix the issues with anatomy and "forgotten knowledge" from PonyXL. Final goal is to make this model able to handle both characters and scenery in a good composition.
Feel free to suggest!

How to use this model

Just download the checkpoint and put it in your checkpoints folder (Stable Diffusion).

Here are the recommended parameters for inference (image generation) :

Clip Skip: 2

Sampler: DPM++ SDE

Steps: ~15

CFG Scale: ~2.0

Positive prompt:

score_9, score_8_up, score_7_up, score_6_up, <your prompt>

Negative prompt:

score_6, score_5, score_4

Trigger words for scenery:

scenery
landscape
no humans

This model can, to a certain extent, understand natural language prompts.

For human and character pictures, you may want to lower the CFG and steps.

Caveats

Beware, this checkpoint hasn't been trained much on images of humans and characters, so it may have the same "wild style" issue as the base Pony XL model! It's best to use it with your LoRA for stability. Also, it's not a beast at doing NSFW stuff or very complex poses unlike other Pony derivatives, this will potentially fixed in v1.

Technical specs

Trained on 1xA40

More info about the project and motivations

PonyXL is an incredible checkpoint that excels in depicting complex characters and situations. His understanding of anatomy makes it probably one of the best models for NSFW scenes. However, it seems that most of the dataset was oriented towards the representation of humanoid characters. The extensive fine-tuning seems to have partly "erased" or replaced the concepts initially assimilated by the base model. It is also very likely that the initial training of text encoders has been modified to a point they're only capable of understanding Booru tags (or at least 50% oriented towards this type of descriptive syntax).

Because of this, it seems sometimes difficult to represent anything other than characters with Pony XL without resorting to LoRAs or other extensions. Using the base model by itself, it's even particularly difficult to achieve consistent, aesthetically pleasing landscape scenes.

The aim of this project is therefore to give Pony XL the ability to represent certain concepts such as landscapes, biomes, night scenes, vehicles, objects, etc. in addition to what it already knows how to do, and to give more importance to the scene in general than to characters alone. This obviously raises very big challenges:

- How can we avoid compromising the model's current capabilities?

- How can we ensure that the model is close enough to the original to ensure that LoRAs made for Pony still work?

I'll try to answer these questions with this model. So far, it's not perfect: the model is better at representing environments, but sometimes suffers from overfitting and has become less good than Pony at handling anatomy. The next version will be much better.

This model comes with no warranty. Do not use this model for inappropriate purposes.

Tips:

Most potential small issues, e.g. with eyes can be very easily solved using inpaint/hires fix.