Skip to content
This repository has been archived by the owner on May 28, 2024. It is now read-only.

Add examples to the prompt format docs #128

Merged
merged 1 commit into from
Jan 26, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,40 @@ A prompt format is used to convert a chat completions API input into a prompt to

The string template should include the `{instruction}` keyword, which will be replaced with message content from the ChatCompletions API.

For example, if a user sends the following message for llama2-7b-chat-hf ([prompt format](continuous_batching/meta-llama--Llama-2-7b-chat-hf.yaml#L27-L33)):
```json
{
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is the capital of France?"
},
{
"role": "assistant",
"content": "The capital of France is Paris."
},
{
"role": "user",
"content": "What about Germany?"
}
]
}
```
The generated prompt that is sent to the LLM engine will be:
```
[INST] <<SYS>>
You are a helpful assistant.
<</SYS>>

What is the capital of France? [/INST] The capital of France is Paris. </s><s>[INST] What about Germany? [/INST]
```

##### Schema

The following keys are supported:
* `system` - The system message. This is a message inserted at the beginning of the prompt to provide instructions for the LLM.
* `assistant` - The assistant message. These messages are from the past turns of the assistant as defined in the list of messages provided in the ChatCompletions API.
Expand All @@ -87,7 +121,7 @@ In addition, there some configurations to control the prompt formatting behavior
* `strip_whitespace` - Whether to automatically strip whitespace from left and right of the content for the messages provided in the ChatCompletions API.


You can see an example in the [Adding a new model](#adding-a-new-model) section below.
You can see config in the [Adding a new model](#adding-a-new-model) section below.

### Scaling config

Expand Down
Loading