MoE Clowncar with Mistral Nemo Models?

[icsy7867]

Is this possible? I tried to use 3 Mistral Nemo models. But the final output marks this as: Qwen2MoeForCausalLM

I tried setting the architecture to "Mixtral", but the output told me that the models were not compatible. I am assuming this means I cannot make an MOE using Mistral Nemo models? Thanks for reading and answering.

[icsy7867]

Hmm it looks like I cannot set a shared_expert with mistral models? Does that only work with QWEN models?