exllamav2 and IBM Granite models

[RaRasputinRGLM]

I wanted to work with these because the sizes are appealing for local LLMs, they all use the starcoder tokenizer, however the model.embed_tokens is not found in any of the safe tensors, I was thinking, that I could hack a solution out with extracting the safetensor piece from the starcoder, but I think there might be a faster way to go that I am missing due to inexperience.

I don't think this is really a exllamav2 issue but worth discussing. Here is a similar issue with IBM granite that can give some insight about how they structured their models, over at llama.cpp ggml-org/llama.cpp#7116

In short what is the right route with exllamav2 and missing embed_tokens if you know the tokenizer it uses?

[turboderp]

I'm unsure what the issue is? I added support for Granite two weeks ago.

It indeed doesn't have an embed_tokens tensor, but that's just because GPTBigCode calls it wte instead. It has a number of other features that set it apart from Llama, like learned embeddings, non-gated MLP and bias everywhere, but that should all be accounted for.

[RaRasputinRGLM]

I tracked it down to the init for the STfile in fastensors.py, the header key is transformer.wte.weight
instead of just wte.weight, so it never sees it since it isn't the beginning of the header key

[turboderp]

That appears top be a new bug that's crept in. I did convert that model successfully here: https://huggingface.co/turboderp/granite-20b-code-instruct-exl2

I'll fix it in a moment.

[turboderp]

There, fixed it. It crept in while I was refactoring and adding GPT2 support. If you pull the latest master it should allow you to quantize. Inference on quantized models should work even without the fix (with a prebuilt wheel of 0.0.21 for instance) since the tensors are remapped as part of the conversion.

[RaRasputinRGLM]

Thanks for the fix, learned more about these models doing this, it was a much easier fix than the route I was going down.

[turboderp]

Thanks for pointing it out. Probably wouldn't have caught it for a while.