Asymmetric tiling

[alejobrainz]

Hello. I'm trying to achieve tiling asymmetrically using Diffusers, in a similar fashion to the asymmetric tiling in Automatic1111's extension https://github.com/tjm35/asymmetric-tiling-sd-webui.

My understanding is that I must traverse all layers to alter the padding, in my case circular in X and constant in Y, but I would love to get advice on how to make a such change to the conv2d system in DIffusers.

Your advice is highly appreciated, as it may also help others down the road facing the same need.

[patrickvonplaten]

Hey @alejobrainz,

Sorry I currently won't have time to look into it, but maybe someone in the community is interested!

[alejobrainz]

Actually solved it. Sharing for the community. Basically I a look at @shirayu's great solution for overriding tiling to circular, and adapted it based on the Asymmetric tiling extension from Automatic 1111 WebUI.

Essentially, you collect layers from the vae, text encoder and unet within the pipe, and override their conv2D Forward method, setting a different padding mode for x and y.

Here is my code for making the result tilable on X only:

#...create pipe here

from typing import Optional
from torch import Tensor
from torch.nn import functional as F
from torch.nn import Conv2d
from torch.nn.modules.utils import _pair

def asymmetricConv2DConvForward(self, input: Tensor, weight: Tensor, bias: Optional[Tensor]):
    self.paddingX = (self._reversed_padding_repeated_twice[0], self._reversed_padding_repeated_twice[1], 0, 0)
    self.paddingY = (0, 0, self._reversed_padding_repeated_twice[2], self._reversed_padding_repeated_twice[3])
    working = F.pad(input, self.paddingX, mode='circular')
    working = F.pad(working, self.paddingY, mode='constant')
    return F.conv2d(working, weight, bias, self.stride, _pair(0), self.dilation, self.groups)

targets = [pipe.vae, pipe.text_encoder, pipe.unet,]
conv_layers = []
for target in targets:
    for module in target.modules():
        if isinstance(module, torch.nn.Conv2d):
            conv_layers.append(module)

for cl in conv_layers:
    cl._conv_forward = asymmetricConv2DConvForward.__get__(cl, Conv2d)

# ...use pipe here

[alejobrainz]

Solved. See above.

[yijinsheng]

it works for me in diffusers==0.19.1, but fails in diffusers==0.19.2
maybe tthe official made some upgrades that caused it failed

[yijinsheng]

Actually solved it. Sharing for the community. Basically I a look at @shirayu's great solution for overriding tiling to circular, and adapted it based on the Asymmetric tiling extension from Automatic 1111 WebUI.

Essentially, you collect layers from the vae, text encoder and unet within the pipe, and override their conv2D Forward method, setting a different padding mode for x and y.

Here is my code for making the result tilable on X only:

#...create pipe here

from typing import Optional
from torch import Tensor
from torch.nn import functional as F
from torch.nn import Conv2d
from torch.nn.modules.utils import _pair

def asymmetricConv2DConvForward(self, input: Tensor, weight: Tensor, bias: Optional[Tensor]):
    self.paddingX = (self._reversed_padding_repeated_twice[0], self._reversed_padding_repeated_twice[1], 0, 0)
    self.paddingY = (0, 0, self._reversed_padding_repeated_twice[2], self._reversed_padding_repeated_twice[3])
    working = F.pad(input, self.paddingX, mode='circular')
    working = F.pad(working, self.paddingY, mode='constant')
    return F.conv2d(working, weight, bias, self.stride, _pair(0), self.dilation, self.groups)

targets = [pipe.vae, pipe.text_encoder, pipe.unet,]
conv_layers = []
for target in targets:
    for module in target.modules():
        if isinstance(module, torch.nn.Conv2d):
            conv_layers.append(module)

for cl in conv_layers:
    cl._conv_forward = asymmetricConv2DConvForward.__get__(cl, Conv2d)

# ...use pipe here

it works for me in diffusers==0.19.1,but not working in diffusers==0.19.2
maybe the official made some upgrades that caused it failed

[cmdr2]

Yep, this changed with the recent change in diffusers 0.19.2 - b1e5279#diff-528ec2d3100f053b2b75ba808cb0e8b983545b721b1c42dc1fd47d76522d7faeR130

This workaround works for me. Change the last for-loop to:

for cl in conv_layers:
    if isinstance(cl, diffusers.models.lora.LoRACompatibleConv) and cl.lora_layer is None:
        cl.lora_layer = lambda *x: 0

    cl._conv_forward = asymmetricConv2DConvForward.__get__(cl, torch.nn.Conv2d)

This forces it to use the _conv_forward wrapper when a lora isn't set. This workaround should work with loras as well, although I haven't tested it.

[yijinsheng]

Yep, this changed with the recent change in diffusers 0.19.2 - b1e5279#diff-528ec2d3100f053b2b75ba808cb0e8b983545b721b1c42dc1fd47d76522d7faeR130

This workaround works for me. Change the last for-loop to:

for cl in conv_layers:
    if isinstance(cl, diffusers.models.lora.LoRACompatibleConv) and cl.lora_layer is None:
        cl.lora_layer = lambda *x: 0

    cl._conv_forward = asymmetricConv2DConvForward.__get__(cl, torch.nn.Conv2d)

This forces it to use the _conv_forward wrapper when a lora isn't set. This workaround should work with loras as well, although I haven't tested it.
yes ,it worked.but i need to set a lora model in my case, so use 0.19.1 before anyone can solve it ,i won't do an upgrade.

[cmdr2]

@yijinsheng The workaround should work with lora as well. Does it fail if you set a lora?

[yijinsheng]

@yijinsheng The workaround should work with lora as well. Does it fail if you set a lora?

@cmdr2 sorry late to reply .i have test it and it work with lora as well,👍

• [ ]

[cmdr2]

As a tip, disable VAE Tiling before using this. That is, set pipe.vae.use_tiling to False before running this code.