Designating Chat-Completion API roles to text blocks

[IS2511]

How would one go about marking some text as assistant? Or as user? Mainly using OpenAI Compatible API.

Steps to reproduce: In Adventure mode, send a second action.

> Action 1

Story 1

> Action 2

Story 2 (to be generated)

Expected request:

[
{ "role": "user", "content": "Action 1" }
{ "role": "assistant", "content": "Story 1" }
{ "role": "user", "content": "Action 2" }
{ "role": "assistant", "content": "Story 2" }
]

Observed request:

[{ "role": "user", "content": "> Action 1\n\nStory 1\n\n> Action 2\n\nStory 2" }]

(user is the Main Message Role)

And yes, I do want it to work with at least multi-line assistant text blocks.

I tried using Chat mode and doing something like this.

User: Action 1

Assistant: Story 1

User: Action 2

Assistant: Story 2 (to be generated)

But the results were the same - a single user block.

How can I make the request be a conversation between Action as user and Story as assistant?

Suggested issue tags: question, enhancement

[LostRuins]

Is it possible to use the completions api instead? The chat completions api allows you to add prefix and postfix blocks but the main block is grouped together

[IS2511]

Is it possible to use the completions api instead?

The raw one that simply continues the text input? Hmm. Honestly, I don't think DeepSeek has one for the R1... Only a Chat Completions one. (But don't quote me on this)

It's usually not a problem to simply have a system Prefix with the direct instructions to the LLM like writing style and formatting, and everything else in Memory and World Info. But I'm experimenting more and more with DeepSeek-R1, and it apparently requires a strict user - assistant - user - assistant - ... chain of chats. From my brief testing, having a single big user block yields worse output and "confuses" the model. Plus it frequently tries to generate improper formatting and Actions (since it has example of both Story and Action blocks and format in input), making KoboldAI Lite fail the generation entirely (maybe due to an early > token or something, idk).

I'd really like to have more granular control of the Chat Completion blocks...

Last time I wanted to do something like this, I added some terrible hacks into the code, parsing all KoboldAI text blocks in sequence, concatenating all regular blocks into big assistant blocks, splitting at any block that begins with > , making those blocks user (my Actions are usually a single line, no line breaks). But that's terrible and I'd rather not make stuff like that hardcoded.

Maybe something like making Chat mode support per-character Chat Completions role selection? Then I could make the "You" character be user blocks and the "KoboldAI" character be assistant blocks, similar to my second "Chat mode" example from above. I'm fine with manually marking text via "Name: " prefixes to assign Chat Completion roles. And I kinda think it could work for Adventure mode too, if the Actions are restricted to no line breaks...

[LostRuins]

Yeah, that was the main limitation since turns don't work with story mode. I might be able to make it work with assistant mode, but like you said, Kobold offers too much freedom. Users can put turns in any order and even multiple user turns in sequence, which is often not allowed in chat completions.

[IS2511]

Last time I wanted to do something like this, I added some terrible hacks into the code,

I did it again, wanted to see how it would work with DeepSeek-R1 (still not sure of the results). I hardcoded (on top of 920117d) a bunch of prefixes with some less than ideal code, catered more to my personal preferences than a general workable approach. But it should demonstrate what I was aiming for more clearly.

Notes: saved_oai_role ID selected to not intersect with any future IDs ("experimental"). The approach I'm using is not scalable at all, so you probably shouldn't do it this way.

Patch (spoiler)

diff --git a/index.html b/index.html
index 4b00226..3e3d952 100644
--- a/index.html
+++ b/index.html
@@ -3079,7 +3079,7 @@ Current version indicated by LITEVER below.
 		saved_cohere_preamble: "", //cohere preamble
 		saved_palm_jailbreak:"", //gemini system prompt
 		saved_oai_custommodel: "", //customized oai custom model
-		saved_oai_role: 0, //0=user,1=assistant,2=system
+		saved_oai_role: 0, //0=user,1=assistant,2=system,1003=dynamic
 		saved_a1111_url: default_a1111_base,
 		saved_comfy_url: default_comfy_base,
 		saved_xtts_url: default_xtts_base,
@@ -14414,6 +14414,18 @@ Current version indicated by LITEVER below.
 						}
 					}
 					let myrole = (localsettings.saved_oai_role==2)?"system":(localsettings.saved_oai_role==1?"assistant":"user");
+					switch (localsettings.saved_oai_role) {
+						case 2:
+							myrole = "system";
+							break;
+						case 1:
+							myrole = "assistant";
+							break;
+						case 0:
+						default:
+							myrole = "user";
+							break;
+					}
 					oai_payload.messages = [];
 					targetep = (custom_oai_endpoint + oai_submit_endpoint_turbo);
 					if (document.getElementById("jailbreakprompt") && document.getElementById("jailbreakprompt").checked && document.getElementById("jailbreakprompttext").value!="") {
@@ -14421,7 +14433,64 @@ Current version indicated by LITEVER below.
 						addrole = ((addrole==2)?"system":(addrole==1?"assistant":"user"));
 						oai_payload.messages.push({ "role": addrole, "content": document.getElementById("jailbreakprompttext").value });
 					}
-					oai_payload.messages.push({ "role": myrole, "content": mainoaibody });
+					if (localsettings.saved_oai_role == 1003) {
+						let paragraphs = mainoaibody.split("\n\n");
+						let lastRole = null;
+						let currentContentBlock = "";
+						const rolePrefixes = {
+							"user": ["User: ", "Action: ", "> "],
+							"assistant": ["Assistant: ", "Story: "],
+							"system": ["System: "]
+						};
+						let inMemoryBlock = false;
+						let inInstructionsBlock = false;
+						for (const paragraph of paragraphs) {
+							let role = "assistant";
+							if (rolePrefixes["user"].some(prefix => paragraph.startsWith(prefix))) {
+								role = "user";
+							} else if (rolePrefixes["assistant"].some(prefix => paragraph.startsWith(prefix))) {
+								role = "assistant";
+							} else if (rolePrefixes["system"].some(prefix => paragraph.startsWith(prefix))) {
+								role = "system";
+							}
+							if (paragraph.startsWith("<memory>")) {
+								inMemoryBlock = true;
+							} else if (paragraph.startsWith("<instructions>")) {
+								inInstructionsBlock = true;
+							}
+							if (inMemoryBlock || inInstructionsBlock) {
+								role = "user";
+							}
+							if (role !== lastRole) {
+								if (lastRole !== null) {
+									oai_payload.messages.push({ "role": lastRole, "content": currentContentBlock });
+								}
+								currentContentBlock = paragraph;
+							} else {
+								currentContentBlock += "\n\n" + paragraph;
+							}
+							if (paragraph.endsWith("</memory>")) {
+								inMemoryBlock = false;
+							} else if (paragraph.endsWith("</instructions>")) {
+								inInstructionsBlock = false;
+							}
+							lastRole = role;
+						}
+						// Concat consequetive messages with the same role into single message
+						lastRole = null;
+						for (const message of oai_payload.messages) {
+							let role = message.role;
+							if (lastRole !== null) {
+								if (role === lastRole) {
+									oai_payload.messages[oai_payload.messages.indexOf(message) - 1].content += "\n\n" + message.content;
+									oai_payload.messages.splice(oai_payload.messages.indexOf(message), 1);
+								}
+							}
+							lastRole = role;
+						}
+					} else {
+						oai_payload.messages.push({ "role": myrole, "content": mainoaibody });
+					}
 					if (document.getElementById("jailbreakprompt2") && document.getElementById("jailbreakprompt2").checked && document.getElementById("jailbreakprompttext2").value!="") {
 						let addrole = document.getElementById("jailbreakprompttext2role").value;
 						addrole = ((addrole==2)?"system":(addrole==1?"assistant":"user"));
@@ -20707,6 +20776,7 @@ Current version indicated by LITEVER below.
 						<option value="0" selected>User</option>
 						<option value="1">Assistant</option>
 						<option value="2">System</option>
+						<option value="1003">Dynamic</option>
 					</select>
 					<input type="checkbox" id="jailbreakprompt" onchange="togglejailbreak()">
 					<div class="box-label" title="Adds extra text at the start to improve AI response">Add Prefix</div>

[LostRuins]

If people really want chat completions with proper turns I can add it in. Just be aware that it will only work in instruct mode, and might not work correctly for all formats or work with story/chat

[IS2511]

If people really want chat completions with proper turns

I have no idea about the demand for this feature. I think people seeing sequences of "user submitted text" followed by "LLM response" and assume those are user and assistant text blocks when Chat Completions are on... So maybe it's a good idea to at least better convey what "Main Message Role" is and how it works 🤷 I personally would welcome this feature.

it will only work in instruct mode

Doesn't cover my use case, which is kinda sad. But as shown above, I can get by with some local patching, even if it's a bit annoying.

[LostRuins]

Alright, this has been added. Please select AutoRole as the main text body, and it will attempt to select correct roles for each message chunk.