About the tT_loss

[zzc681]

Hi, Thanks for your excellent work, but I have a small question about the loss function. When I was reading the code, I found that tT_loss calculates the loss between X_t and 0. Is there any meaning to doing this?
The code in the gaussian_diffusion.py, the function training_losses_e2e in class GaussianDiffusion
out_mean, _, _ = self.q_mean_variance(x_start, torch.LongTensor([self.num_timesteps - 1]).to(x_start.device))
tT_loss = mean_flat(out_mean ** 2)

[ryuliuxiaodong]

Same question for me.

The other loss terms written in training_losses_e2e are clear, which are also described in the paper (Equation 2). But I don't quite understand this tT_loss: why loss is calculated on each timestep of this forward diffusion process?

[K0ntact]

Seems like tT_loss is to calculate how well the input are diffused into Normal Gaussian noise, since it is calculated by diffusing x_start to the last diffusion step via q_mean_variance.

Diffusion-LM/improved-diffusion/improved_diffusion/gaussian_diffusion.py

Lines 252 to 259 in 759889d

	def q_mean_variance(self, x_start, t):
	"""
	Get the distribution q(x_t | x_0).
	:param x_start: the [N x C x ...] tensor of noiseless inputs.
	:param t: the number of diffusion steps (minus 1). Here, 0 means one step.
	:return: A tuple (mean, variance, log_variance), all of x_start's shape.
	"""

The squared difference between out_mean and normal distribution mean (0) is tT_loss, which explains why it is written as out_mean ** 2.

Diffusion-LM/improved-diffusion/improved_diffusion/gaussian_diffusion.py

Lines 1566 to 1567 in 759889d

	out_mean, _, _ = self.q_mean_variance(x_start, th.LongTensor([self.num_timesteps - 1]).to(x_start.device))
	tT_loss = mean_flat(out_mean ** 2)