MTL-LoRA

[FrLdy]

🌟 New adapter setup

Model description

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

LoRA has been widely used for parameter-efficient fine-tuning (PEFT), especially in domain adaptation. However, in multi-task learning (MTL) scenarios, LoRA struggles with task interference due to the projection of high-dimensional task-specific features into the same low-dimensional space. This results in suboptimal performance.

To address this challenge, MTL-LoRA is proposed, an enhancement to LoRA that preserves its efficiency while improving multi-task adaptation. MTL-LoRA introduces task-adaptive parameters to better differentiate between tasks while capturing shared knowledge. This leads to improved performance across various MTL benchmarks with a comparable or even reduced number of trainable parameters.

Open source status

• the model implementation is available: No
• the model weights are available: No
• who are the authors: Yaming Yang, Dilxat Muhtar, Yelong Shen, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Denvy Deng, Feng Sun, Qi Zhang, Weizhu Chen, Yunhai Tong

Tasks

• ForwardContext manage task_ids forward parameter as discussed in Support for Passing `task_ids` to `forward_context` for Multi-Task Learning #783, Support for Passing task_ids to forward_context for Multi-Task Learning #784
• add MTLLoRA Adapter Add MTL-LoRA Adapter #792