self conditioning not working (size mismatch)

[inspirit]

When enabling self_cond there is size mismatch when trying to sum embeds:

soundstorm-pytorch/soundstorm_pytorch/soundstorm.py

Line 582 in 08a5e20

x = x + sum_embeds

this happens because 'sum_embeds' tensor is not reduced as it was projected to:

nn.Linear(dim, dim * num_effective_quantizers),
Rearrange('b n (h d) -> b (n h) d', h = num_effective_quantizers)

so 'sum_embeds' is actually 'b (n h) d' while 'x' is 'b n d'

[lucidrains]

@inspirit ah, thanks for uncovering this

how about 0.4.4?

[inspirit]

you forgot to put 'sum' argument to 'reduce' call :)

[inspirit]

ohh, and btw there is a special case when 'generate' is called because you will get 'null_embed' at the first iteration which is just [dim] shape/size

[inspirit]

probably smth like:

if sum_embeds.ndim == 3:
    sum_embeds = reduce(sum_embeds, 'b (n h) d -> b n d', 'sum', h = self.num_effective_quantizers)

should do the trick

[lucidrains]

@inspirit ok done

does self conditioning work for you? haven't heard of many people using it

[inspirit]

i just started digging it for one of my projects with Residual Quantizers, so too early to comment, was validating that all works after incorporating it to my pipeline and found this issue, so might be completely useless in the end :)