How to improve the performance of gemm_half_q_half_gptq_kernel operator?

[lovelynight]

On the A100 graphics card, gptq's custom operator has been tested and is slower than pytorch's linear layer when batch > 10, thus affecting the throughput of high concurrency during model inference. I tried to modify the hyperparameters in q_gemm, but there was no obvious improvement. I would like to ask if there are any ideas to improve the speed of high concurrency? Thanks

[turboderp]

I am working on some improvements, yes, specifically for batch sizes around 5-30, but the improvements should also make it easier to adjust kernel parameters to optimize for other architectures. I'm not sure when I'll have anything ready, though.

(Basically the issue is that the kernel becomes compute-bound at higher batch sizes, and the solution is likely going to be tensor cores.)