[Feature Suggestion] SmoothQuant (W8A8) leads to ~50% better throughput

[DreamGenX]

Hey there,

Thank you for making Exllama!

Most quants tend to have worse throughput and latency compared to fp16 inference. SmoothQuant avoids dequantization step and allows for much higher throughput and better latency than fp16.

There's a repo for SmootQuant that lets you create SmoothQuant models and to do inference:
https://github.com/AniZpZ/AutoSmoothQuant

There's also a PR for vLLM by @AniZpZ, but given the complexity of vLLM, is unlikely to get merged any time soon:
vllm-project/vllm#1508

Exllama is known for its speed, so I think this could fit really well here.