Add Qwen3 Moe #2260
Add Qwen3 Moe #2260
Conversation
| self, | ||
| num_query_heads, | ||
| num_key_value_heads, | ||
| layer_index, |
There was a problem hiding this comment.
This layer index is gross, let's remove it. Handle the args properly in the backbone and pass the correct sliding_window_size to this layer and the decoder layer above it.
There was a problem hiding this comment.
since it's an Moe, layer index is not just used for sliding window but for experts
There was a problem hiding this comment.
I replaced this passing of layer_index, decoder_sparse_step and mlp_only_layers with a single boolean switch:
| model(input_data) | ||
| """ | ||
|
|
||
| def __init__( |
There was a problem hiding this comment.
In general, let's make sure we prune this list down just to the config options we need.
| sliding_window_size=32768, | ||
| output_router_logits=False, | ||
| router_aux_loss_coefficient=0.001, | ||
| mlp_only_layers=[], |
There was a problem hiding this comment.
Fine to have something like this for the toplevel, but let's pass something more direct to each decoder layer (so we don't need to pass the index down). Make sure to document if we keep it.
There was a problem hiding this comment.
but let's pass something more direct to each decoder layer
what do you suggest?
No description provided.