Is MLA supported for deepseek models?

[ymcki]

It seems like the MLA related PRs are not merged, so it is not supported yet? If it is supported, from which release does it support?

[fairydreaming]

This PR (#11446) is not merged yet. The main reason for this is that the ambitious large-scale KV cache refactoring PR #11213 that would enable creation of custom KV cache implementations was not merged and instead only a limited subset of these changes was merged in #12181. There is some future work planned in #12181 (listed in the Next section), but the timeline for this is currently unknown. Perhaps @ggerganov will be able to give some estimate.

However, there is a llama.cpp fork that already has #11446 PR (and many other changes that speed up DeepSeek R1/V3 inference) merged, you can try it if you want: https://github.com/ikawrakow/ik_llama.cpp

[fairydreaming]

@ymcki Also there is a new PR #12725, it doesn't require any substantial changes in KV cache implementation so maybe it will be merged soon.

[ymcki]

@ymcki Also there is a new PR #12725, it doesn't require any substantial changes in KV cache implementation so maybe it will be merged soon.

Good to hear that. Hopefully we will see it soon.

[ymcki]

#12181

Seems merged. But #12725 is closed. Does that mean we have MLA or not?

[Dampfinchen]

#12181

Seems merged. But #12725 is closed. Does that mean we have MLA or not?

#12801 They are working on it currently.