What do the different quantize types actually do?

[halfburnttoast]

I’m fairly new to AI, but I’ve been playing around with llama for a week or so.

I noticed in the code for quantize there are four different types implemented but the readme only shows using type 2. I’m having trouble following what the four modes are for.

What do the other 3 quantize types actually do to the model when run? They produce different outputs, but I’m not following what the purpose of it is.

Thanks!

[sw]

See #1121

[halfburnttoast]

Thanks. Weird, I searched for the topic but missed that one.