Exllamav2 installed but alway got "AssertionError: Please install ExllamaV2 or LLama CPP Python as backend"

[vuminhquang]

Hi,
I installed Exllamav2 via (successful)

git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .

However, always get
File "/home/user/micromamba/envs/genv/lib/python3.12/site-packages/gallama/backend/model.py", line 39, in
assert ExLlamaV2 or Llama, "Please install ExllamaV2 or LLama CPP Python as backend"

Please help, thank you.

[remichu-ai]

Hi, I recalled have fixed this bug previously.

Can you try to update to the latest by running pip install -U gallama to see if the problem still persists

[vuminhquang]

Hi,
I do the installation for gallama from code too with git clone and "pip install .".
The os using is fresh ubuntu noble (wsl).
So gallama should be latest, though.

[remichu-ai]

Can you try the following:
1/ Double check if you activate the environement correctly. Seem like you are using mamba, i am not sure how it goes for mamba but i am using conda and i need to activate the environement before using e.g. conda activate genv

2/ Try running the following script, you can also use jupyter notebook to run it line by line if it helps.

This is the part of the script that is showing error. As you can see from the script, it just import ExllamaV2 and then check if it is imported.
It wont go into this error "Please install ExllamaV2 or LLama CPP Python as backend" if the import for ExllamaV2 working

# import transformers
import torch
from typing import List, Dict
from gallama.logger.logger import logger
from gallama.data_classes.data_class import ModelParser

try:
    from exllamav2 import (
        ExLlamaV2,
        ExLlamaV2Tokenizer,
        ExLlamaV2Cache,
        ExLlamaV2Cache_Q4,
        ExLlamaV2Cache_Q6,
        ExLlamaV2Cache_Q8,
        ExLlamaV2Config
    )
except:
    ExLlamaV2 = None
    ExLlamaV2Tokenizer = None
    ExLlamaV2Cache = None
    ExLlamaV2Cache_Q4 = None
    ExLlamaV2Cache_Q6 = None
    ExLlamaV2Cache_Q8 = None
    ExLlamaV2Config = None

try:
    from llama_cpp import Llama
except:
    # optional dependency
    Llama = None

# experimental feature: tensor parallel
try:
    from exllamav2 import ExLlamaV2Cache_TP
except:
    # optional dependency
    ExLlamaV2Cache_TP = None

assert ExLlamaV2 or Llama, "Please install ExllamaV2 or LLama CPP Python as backend"

[vuminhquang]

Hi,
I still get
"Please install ExllamaV2 or LLama CPP Python as backend"
For more information:

• Sure, genv is activated using: "micromamba activate genv"
• Tried install exllamav2 via whl too (after activating genv) "pip install https://github.com/turboderp/exllamav2/releases/download/v0.1.8/exllamav2-0.1.8+cu121.torch2.3.1-cp311-cp311-linux_x86_64.whl"
  But nothing different. Do I missed set some env variables? Thanks

[remichu-ai]

For some strange reason, install clean from source doesnt work for version 0.2.3 for me as well, and i am using prebuilt wheel. Look like your prebuilt url was of outdated version v0.1.8 instead of v0.2.3

You can look through all the version here. Ideally match both the python version and cuda version
https://github.com/turboderp/exllamav2/releases/tag/v0.2.3

Another thing you can try is, still build from source, but install v0.2.0 instead of v0.2.3

Inside the exllamav2 folder that you clone from github. You can run the following command to check out a specific version:

git checkout tags/v0.2.0

v0.2.1 and v0.2.2 have known bugs hence we skip it. I will check the exllamav2 install again later and open an issue on exllamav2 repo if it is confirmed to be issue with their install script

[vuminhquang]

Hi,
This is my updating:

• Tried v0.2.0 of exllamav2 -> still not work
• Tried tabbyAPI with "update_scripts/update_deps.sh" -> the server can run

I think I will wait for a couple of time before continuing to try.
Thanks.

[remichu-ai]

Hi Quang,

I have clarified with turboderp himself and couldnt figure out the issue. I have the exact issue with you where i can only use the wheel to install instead of build from source.

I resolved it by creating a new python env in conda and reinstall everything from scatch. I believe that other library that i was testing messed up with my environment.

Hope that you managed to sort out the issue on your end.

Thanks