Question about the option llm_int8_has_fp16_weight in BitsAndBytesConfig

[peterjc123]

As can be seen here, axolotl uses llm_int8_has_fp16_weight=False and it aligns with the example code in the QLora repo. But another training framework xllm defaults to llm_int8_has_fp16_weight=True (code). Also, according to the FAQ of bitsandbytes, it says

There are two modes:
Mixed 8-bit training with 16-bit main weights. Pass the argument has_fp16_weights=True (default)
Int8 inference. Pass the argument has_fp16_weights=False

So, which one should I use? Or maybe it should be exposed as an option to the yaml configuration?

[NanoCode012]

Hey, this has been exposed recently. You can pass bnb_config_kwargs.

https://github.com/OpenAccess-AI-Collective/axolotl/blob/a3595793712000921a94440e8272723a733d6f57/src/axolotl/utils/models.py#L452-L453