requires_grad_pre_hook on Llava

[Erland366]

There's an issue when training Llava model on latest transformers (4.49.0.dev0) where requires_grad_pre_hook failed on the CLIPEncoder -> specifically this module model.base_model.model.vision_tower.vision_model.encoder

Upon inspection, this is because the inputs on the hook is empty. This means this is a larger issue because CLIPEncoder never receive any input.

I tested bypassing the hook error, and the training result is very bad