Performance implications of context size

[k-b-1]

I tested a fixed sequence of prompts that needs a context size of almost 4k on a mistral 7b instruct model. Only one slot, no de-fragmentated kv cache.
When using a 4k context size, the sequence lasts 3 minutes. When using 32k context size, the same sequence lasts 11minutes.
Why does an increased but not used context size has such an impact on the performance?