In the previous article, we covered progress in image generation using denoising diffusion probabilistic models (DDPMs), potentially built with U-Nets and Transformers. However, we only discussed unconditional image generation, which doesn's allow us to guide the generation to obtain the kinds of images we want and is impractical. Hence, in this article, we will discuss ways to perform conditional image generation.

Classifier Guidance

Dhariwal, P. et al. (2021), inspired by GANs, proposed classifier guidance, a method that uses a pretrained image classifier during inference to guide the unconditional model towards the desired class. The method is based on the observation that the conditional generation $p_{\phi, \theta}(x_t | x_{t+1}, y)$ can be expressed as $Zp_{\theta}(x_t|x_{t+1})p_{\phi}(y|x_t)$, where $Z$ is the normalizing constant, $p_{\theta}(x_t|x_{t+1})$ is the unconditional generation, and $p_{\phi}(y|x_t)$ is the classification, implying that conditional generation can be achieved by the unconditional generator $\theta$ and classifier $\phi$.

After mathematical derivations (available in the cited paper, which is out of scope for this article), they found that subtracting $\sqrt{1-\bar{\alpha}_t}\nabla_{x_t} log(p_{\phi}(y | x_t))$, the classifier gradient representing the changes needed to classify $x_t$ as $y$, from the noise $\epsilon_{\theta}(x_t)$ can nudge the model towards denoising the latents in the direction of images with the provided class. By scaling the gradient by a hyperparameter $s$, we can control how much the classifier gradient nudges the denoising.

The model behaves like the unconditional generation when $s=0$, and interpolates between unconditional and normal conditional generation when $0 < s < 1$. Though classifier guidance allows us to perform varying degrees of conditional generation during inference without modifying DDPMs, the signals from the classifier on the initial noisy images might be unhelpful or even adversarial. It also adds complexity to the training, as the classifier needs to be trained on those noisy images.

Classifier Free Guidance

In addition to the drawbacks of classifier guidance mentioned previously, the classifier guidance might only be contributing to "deceiving" Inception-v3 to achieve high IS and FID scores by using the gradients from a classifier pretrained on ImageNet, rather than generating realistic images. To consider alternatives, we can analyze the mathematics behind classifier guidance.

In classifier guidance, we parameterized $\nabla_x log(p_{\phi}(y|x_t))$ with a classifier $\phi$. However, we can derive it differently to apply classifier guidance without a classifier, or using classifier-free guidance (CFG). Using Bayes' theorem, we can express $p(y|x_t)$ as $\frac{p(x_t|x_{t+1}, y)p(y)}{p(x_t | x_{t+1})}$. Taking the derivatives of the log with respect to $x$, we can substitute $\nabla_x log(p_{\phi}(y|x_t))$ as $\nabla_x log(p(x_t|x_{t+1}, y)) - \nabla_x log(p(x_t| x_{t+1}))$. With the hyperparameter $s$, we can rewrite the guidance as follows.

This approach results in a gradient $\nabla_x log(p(x_t | x_{t+1}, y))$ comprising only of conditional generator, unconditional generator, and guidance weight, acting as a hyperparameter controlling the implicit classifier gradient or strength of conditioning. After mathematical derivations, we arrive at the new noise sampler $\hat{\epsilon}_{\theta}(x_t, y) = (1+s)\epsilon_{\theta}(x_t, y) - s\epsilon_{\theta}(x_t)$. Though we need to train the new conditional model on top of the unconditional model, unlike the previous formulation, we can avoid the classifier, which was a major source of the drawbacks. Furthermore, creating a conditional model and training it jointly with the unconditional model is not overly complicated, as we can simply attach a class embedding to the input and occasionally drop it out (typically 10–20%) during training.

We want great control over the class of the generated images, so we generally prefer training the model with high guidance weights. (We also want strong conditioning for models pretrained on large datasets to achieve coherent image generation.) However, it is empirically found that high guidance weights often result in less diversity and highly-saturated images due to excessive guidance that exceeds the normalized pixel value range of [-1, 1]. For the latter, we typically resort to dynamic thresholding, where a dynamic threshold $s$, representing the current intensity percentile (typically 99.5%), is chosen and applied to $x_t$ after each sampling step.

CADS

Though dynamic thresholding is effective in addressing high saturation to some degree, the original problem of diversity still needs to be addressed. Analyzing the problem, we suspect that the diversity issue could be caused by excessively high guidance weights for earlier stages. While we want to maintain high guidance weights for images with less noise to ensure high quality and validity, applying high guidance weights for images with more noise can overly constrain the generation, leading it to produce the easiest-to-classify image and hurting diversity.

Hence, condition-annealing diffusion sampling (CADS) addresses this by applying scheduled Gaussian noise to the class embedding, from strong noise to weaker noise. This allows the model to be less class-dependent initially for diverse generation and then gradually recover class dependence for high generation quality.

Here, $\tau_1$ and $\tau_2$ are user-defined thresholds. Since $t$ goes backward during denoising, this approach applies more noise at the early stages and then gradually decreases the noise level. However, adding Gaussian noise results in $\tilde{y}$ having a different mean than the original $y$ that the model was trained to see, potentially impacting performance. To mitigate this, we can rescale it back to $y$ and control the extent of rescaling to ensure some level of diversity in image generation, using the following formula.

Here, $\psi$ is a hyperparameter controlling the extent of rescaling, and $\hat{y}$ is passed as the noised class embeddings for the conditional model. With this approach and hyperparameter tuning (which you can check in the original paper cited at the bottom of the article), CADS managed to improve the diversity of image generation for virtually all the pretrained conditional models while maintaining image quality and FID. It also compared CADS with dynamic CFG, which simply uses annealed weights to the conditional noise during inference, and confirmed its superiority regarding both diversity and quality. This superiority could be caused by the change in directions of the class embedding by the noise, which scalar weights cannot achieve.

Conclusion

In this article, we discussed classifier guidance, CFG, and CADS, which are noise sampling methods during inference (that use a pretrained classifier or conditional DDPM on top of an unconditional DDPM) that allow us to control the strength of conditioning and achieve diverse and quality conditional image generation based on class labels. Though we assumed that the condition $y$ is a class embedding throughout the article, these three methods are not necessarily constrained by using class labels. For example, we can use various conditions, including image embeddings of rough doodles and text embeddings of more detailed captions, for more controlled and fine-grained generation. Therefore, in the next article, we will discuss how we can generate text embeddings that best tie the text and visual information for more practical and performant conditional image generation.

Resources

• Adaloglou, N. & Kaiser, T. 2024. An overview of classifier-free guidance for diffusion models. AI Summer.
• Dhariwal, P. et al. 2021. Diffusion Models Beat GANs on Image Synthesis. ArXiv.
• Ho, J. & Salimans, T. 2022. Classifier-Free Diffusion Guidance. ArXiv.
• Sadat, S. et al. 2024. CADS: Unleashing The Diversity Of Diffusion Models Through Condition-Annealed Sampling. ICLR 2024.
• sasgawy. 2024. 最新の高性能 Diffusion Models （2024年）. Qiita.