Avoid duplicate special tokens in chat formats

[CISC]

Having multiple BOS can ruin generation, this would occur in several ways, usually through user adding them unnecessarily, in this case output a warning if we detect two in a row.

However, several formats/templates also have BOS in them, and we would add another one later in completion, therefore tokenize format/template prompts before completion to avoid this.

Added a new added_special property to ChatFormatterResponse to detect when BOS/EOS has already been added to prompt.

Also added missing token_cls, token_sep, add_bos_token and add_eos_token to _LlamaModel because I thought I needed them (turns out I didn't).

Fixes #1501

[abetlen]

@CISC thank you (also for the other PRs), this looks good. I may update the warning to use the python warnings library (would be useful for deprecations as well) so it can be silenced in a standard way.

[BigCatGit]

me too

[Rocketknight1]

Hi, Matt from Hugging Face here. Just a note that we expect chat templates to contain all the special tokens the model expects, and therefore in Transformers, whenever a chat has been formatted with a chat template, it should be tokenized without adding additional special tokens.

We really had no choice but to do it this way - many base models add EOS at the end of sequences, but this will always cause problems if we want to generate a chat completion following a user message, as user messages often don't end with EOS in many chat formats.

[CISC]

@Rocketknight1 Thanks for checking in. :)

Yeah, I perfectly understand the reasoning, it just wasn't obvious until recently that this and some built in chat formats were causing duplicate tokens due to this.