Step 4: Evaluating Models with knn：Incorrect perplexity (ppl)

[Binn0]

Why is it that when I reproduce Step 4: Evaluating Models, the perplexity (ppl) I get from running knn-LM is around 17? Could you please explain why this is the case? I would greatly appreciate it if you could provide a response.

[urialon]

Hi Rubin, Thank you for your interest in our work. Does it still happen when you use our datastore and our index? Best, Uri

…

On Wed, Dec 4, 2024 at 2:41 PM Rubin ***@***.***> wrote: Why is it that when I reproduce Step 4: Evaluating Models, the perplexity (ppl) I get from running knn-LM is around 17? Could you please explain why this is the case? I would greatly appreciate it if you could provide a response. — Reply to this email directly, view it on GitHub <#18>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXMBTE24EH4CJMZOL3GT2D5LGHAVCNFSM6AAAAABTA5C5N6VHI2DSMVQWIX3LMV43ASLTON2WKOZSG4YTQNRUGEYTGMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Binn0]

Dear author,

Thank you very much for your response! I am using the neulab/gpt2-finetuned-wikitext103 model, and the dataset is Wikitext-103. The index and vals files I am using are gpt2/index_gpt2_116988150_768.indexed and gpt2/dstore_gpt2_116988150_768_vals.npy, respectively, from the link https://knn-transformers.s3.amazonaws.com/index.html.

However, when using the --knn option, the perplexity (PPL) of GPT2 is 17.34, which is significantly higher than the 12.57 you provided. I was wondering if you know what might be causing this discrepancy?

Another question is that in your article, the method RetoMaton is compared using Foss, and according to your image, a smaller Foss value indicates a lower PPL and better performance. However, for the knn-LM, it seems there is no hyperparameter related to Foss in the code.

If I could receive your reply, it would be greatly appreciated.