Why the chat mode starts looping so often?

[FezVrasta]

I was trying to write a RPG bot using the LLaMA 13B model but after just a few exchanges the AI starts repeating the same sentences in loop or recaps what happened so far for no reason.

Am I doing something wrong or it's a known problem?

[hungerf3]

What do you have repeat_last_n set to?

[FezVrasta]

I tried with the default chat example value and I also tried to set it to twice and 3x it. I experience always the same issue 🤔

[SlyEcho]

Try --memory_f32, I've found that helps with long contexts (2048)

[FezVrasta]

Thanks, unfortunately it isn't helping, it starts repeating the same sentences after just a couple of messages. 580 openAI tokens or 2600 characters.

These are my settings:n_predict 4096, ctx_size 4096, temp 0.7, top_k 40, top_p 0.5, repeat_last_n 512, batch_size 1024, repeat penality 1.17647, memory_f32

[Green-Sky]

context size only goes up to 2048.
you could try to set the temperature higher, so less likely tokens become more likely.

[Green-Sky]

interactive explanation on how temperature works: https://lukesalamone.github.io/posts/what-is-temperature/

[FezVrasta]

Thanks that seems to have helped, I set it to 0.9. Unfortunately now it sometimes complete my sentences before proceeding with a reply 🤷 no big deal but a bit surprising.