DCGAN loss does not comply with literature

[ben-schindler]

Hello,

I wanted to run experiments with the “historical” DCGAN architecture on CIFAR10, noting that, unlike the original paper by Radfort et al. (2015), the discrimination is not bounded to a [0,1]-classification, nor is it trained with the BinaryCrossEntropy loss. Instead, the sigmoid activation is replaced by the softplus function.

Investigating the code history revealed that older versions comply more with the original DCGAN, but were adapted with commit 3f875a9.

     def d_vanilla(d_logit_real, d_logit_fake, DDP):
 -      device = d_logit_real.get_device()
 -      ones = torch.ones_like(d_logit_real, device=device, requires_grad=False)
 -      d_loss = -torch.mean(nn.LogSigmoid()(d_logit_real) + nn.LogSigmoid()(ones - d_logit_fake))
 +      d_loss = torch.mean(F.softplus(-d_logit_real)) + torch.mean(F.softplus(d_logit_fake))
        return d_loss


     def g_vanilla(d_logit_fake, DDP):
 -      return -torch.mean(nn.LogSigmoid()(d_logit_fake))
 +      return torch.mean(F.softplus(-d_logit_fake))

Maybe you could explain your design decision. Why was the original Binary-Classification with BCE-Loss replaced? Is there some work backing the current implementation?

I am very much in favor of a solution where the “vanilla” version actually contains the original loss to preserve consistency with the literature. I therefore suggest rolling back the change of the quoted commit and adding a new loss that contains the unbounded variant.

Nevertheless, many thanks for the code provided!