New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wrong size of iq1s_grid #4

Open

zinccat opened this issue Feb 21, 2025 · 0 comments

zinccat commented Feb 21, 2025

seems that llama.cpp's iq1s_grid is now of size 2048, https://github.com/ggml-org/llama.cpp/blob/ee02ad02c56ff36a5edd22d8617ab3f9546ce7fe/ggml/src/ggml-common.h#L1075

while it is still 512 in this project, causing wrong dequantization and vecmatmul result to be wrong

llama-cpp-torch/llamacpp_kernel.cu

Line 743 in f1da76e

static const __device__ uint64_t iq1s_grid[512] = {

A correct implementation is shown in https://github.com/Isotr0py/ggml-libtorch/blob/main/ggml-cuda/gguf_kernel.cu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment