gradient clipping

[wangjiawen2013]

Hi,
This is how I clip the gradients, but it seems that it will not work.

let loss = MseLoss::new().forward(output.clone(), batch.targets, Mean);
let grads = loss.backward();
let grads = GradientClippingConfig::Norm(1.0).init().clip_gradient(grads);

So could you give an example on how to clip the gradients ?

[wangjiawen2013]

Besides, how to display and get the values of the grads ?

[laggui]

With the GradientsParams, you can retrieve the gradients for a given ParamId (associated to a parameter tensor) with grads.get(id).

And then you can display / check the values of the grad tensor. If you want to check all values of a module, you could use a ModuleVisitor and do something similar to the optimizer adaptor I linked in my other response.

[laggui]

You can set this via the optimizer config .with_gradient_clipping.

You can see how it's applied in the optimizer:

burn/crates/burn-core/src/optim/simple/adaptor.rs

Lines 120 to 131 in b33bd24

	let grad = self.grads.remove(id);
	if let Some(grad) = grad {
	let device = grad.device();
	let is_require_grad = tensor.is_require_grad();
	let (key, record) = self.records.remove_entry(&id).unzip();
	let clipped_grad = if let Some(g_clipping) = self.grad_clipping {
	g_clipping.clip_gradient(grad)
	} else {
	grad
	};

[wangjiawen2013]

Thanks, I got access to the grads and clipped them using the following code:

let tensor = Tensor::<Autodiff<NdArray>, 2>::from_floats([[1,2,3], [4,5,6]], &device);
let mut model: Linear<Autodiff<NdArray>> = LinearConfig::new(3, 1).init(&device);
let output = model.forward(tensor);
let loss = MseLoss::new().forward(output.clone(), Tensor::<Autodiff<NdArray>,2>::from_floats([[4], [7]], &device), Mean);
let grads = loss.backward();
let grads = GradientsParams::from_grads(grads, &model);

// Get access to the grads, NdArray must be used here.
println!("{}", grads.get::<NdArray, 2>(model.weight.id).unwrap());

let mut optim: OptimizerAdaptor<Adam, Linear<Autodiff<NdArray>>, Autodiff<NdArray>> = AdamConfig::new()
    .with_grad_clipping(Some(GradientClippingConfig::Norm(1.0)))
    .init();
model = optim.step(0.001, model, grads);

I can run the code, and I found we must use NdArray when getting the grads, Autodiff<NdArray> will cause an panic.