Can you write an extension for Forge or A1111 ?

[wardensc2]

Hi ethan

Can you support us write an extension for A1111 or Forge. Your plugin not only improve render speed, it also reduce about 30% or more Vram when upscale and really necessary for us to create high resolution image

Thank you so much

[ethansmith2000]

Hi there, I'm currently looking into it, the closest I see is that ToMe is applied here although its not clear where it imports from.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/dev/modules/sd_models.py#L866

[wardensc2]

I think that is token merge in Cross attention optimization in Settings but it can't get much speed or reduce Vram like your node, I already try but the result is bad, the image is uglier

[wardensc2]

People already update your node into Forge but still no one write an extension for A1111 yet sadly

[AG-w]

forge added model patcher similar to comfyUI, can't just copy them directly

[AG-w]

People already update your node into Forge but still no one write an extension for A1111 yet sadly

I'm not sure if it's work 100% correctly, but you can try
https://github.com/AG-w/sd-webui-todo/

[wardensc2]

People already update your node into Forge but still no one write an extension for A1111 yet sadly

I'm not sure if it's work 100% correctly, but you can try https://github.com/AG-w/sd-webui-todo/

Thank you so much

[ethansmith2000]

we're also exploring it over here ethansmith2000/ImprovedTokenMerge#2
the current method im using lets us circumvent having to use the setup a1111 uses for ToMe but also still unsure on some other parts

[wardensc2]

People already update your node into Forge but still no one write an extension for A1111 yet sadly

I'm not sure if it's work 100% correctly, but you can try https://github.com/AG-w/sd-webui-todo/

Hi AG-w I try to run it on A1111 so far it only work in some resolution, I try to render 1920x1080 A1111 will stop running even when I disable this extension. Thank for your effort it seem that we need to wait for ethansmith2000 to solve this problem

[feffy380]

I was implementing ToDo for kohya/sd-scripts (I found it greatly speeds up training).
The code was self-contained enough that I was able to turn it into an extension.
https://github.com/feffy380/sd-webui-token-downsampling

Instead of individual downsample factors for different depths, I've opted to add a single max_depth slider which determines which layers are affected (basically tomesd's max_downsample argument but renamed to avoid confusion). I found the quality/speed tradeoff terrible at deeper layers so I didn't feel the granularity was necessary.

There's also the option to disable ToDo after a certain percentage of steps. I found this helps with details, though you sacrifice some speed. You're essentially using the original model as a refiner for the ToDo model.

[ethansmith2000]

Awesome to hear! The reason why I set multiple depths is because for 2048x2048 I found using 4 and 2 for the top layer and one below it respectively works pretty well.

[feffy380]

Ah fair enough, I hadn't gone that high. I'll see about adding sliders to override the downsample factor for individual layers.