Has anyone tested watt-tool-8B with tool calling?

[edmcman]

cc @ochafik

Has anyone tested watt-tool-8B with tool calling? watt-tool-8B doesn't seem to have a proper chat template. watt-tool-70B does, but it is the old "JSON" style llama prompt. The how to use section of the README suggests that it needs pythonic style function calls. Is there an official jinja around for that?

I tried watt-tool-8b with the watt-tool-70B jinja, and it generated what looked like valid llama JSON calls to me:

{"name": get_decompiled_function, "parameters": {"binary_hash": "12a6d908a68ccf6f9f3d799705577c28763f5deef6eddcff7643d6d8a6de543d", "function_addr": "0x1101b0"}}

But it doesn't match the following regex:

srv  params_from_: Chat format: Llama 3.x
srv  params_from_: Grammar trigger pattern full: `(\{\s*(?:"type"\s*:\s*"function"\s*,\s*)?"name"\s*:\s*")[\s\S]*`

Is this regex correct? It doesn't look right to me. It doesn't look like it would match this:

{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}

[edmcman]

The regex looks OK in the code?

llama.cpp/common/chat.cpp

Line 1096 in ea57e47

"\\s*\\{\\s*(?:\"type\"\\s*:\\s*\"function\"\\s*,\\s*)?\"name\"\\s*:\\s*\"([^\"]+)\"\\s*,\\s*\"parameters\"\\s*: ");

I'm confused.

[edmcman]

Oh, the problem is the function name isn't quoted. So anyway, the real question is whether there's a jinja for Llama pythonic syntax?

[edmcman]

Here's the example prompt for watt-tool:

In [8]: print(tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=Fa
   ...: lse))
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

You are an expert in composing functions. You are given a question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the function can be used, point it out. If the given question lacks the parameters required by the function, also point it out.
You should only return the function call in tools call sections.

If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]
You SHOULD NOT include any other text in the response.
Here is a list of functions in JSON format that you can invoke.
[{'name': 'financial_ratios.interest_coverage', 'description': "Calculate a company's interest coverage ratio given the company name and duration", 'arguments': {'type': 'dict', 'properties': {'company_name': {'type': 'string', 'description': 'The name of the company.'}, 'years': {'type': 'integer', 'description': 'Number of past years to calculate the ratio.'}}, 'required': ['company_name', 'years']}}, {'name': 'sales_growth.calculate', 'description': "Calculate a company's sales growth rate given the company name and duration", 'arguments': {'type': 'dict', 'properties': {'company': {'type': 'string', 'description': 'The company that you want to get the sales growth rate for.'}, 'years': {'type': 'integer', 'description': 'Number of past years for which to calculate the sales growth rate.'}}, 'required': ['company', 'years']}}, {'name': 'weather_forecast', 'description': 'Retrieve a weather forecast for a specific location and time frame.', 'arguments': {'type': 'dict', 'properties': {'location': {'type': 'string', 'description': 'The city that you want to get the weather for.'}, 'days': {'type': 'integer', 'description': 'Number of days for the forecast.'}}, 'required': ['location', 'days']}}]<|eot_id|><|start_header_id|>user<|end_header_id|>

Find me the sales growth rate for company XYZ for the last 3 years and also the interest coverage ratio for the same duration.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Really weird. They use but bypass most of the chat template they have. I guess it will require a custom jinja.

[edmcman]

This is the Llama 3.3 format from here: https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/

[ochafik]

Hey @edmcman, thanks for looking into this! It seems the model likes to wrap its tool calls inside wiki block syntax

```json
{"name": ..., "parameters": ...}
```

That's a syntax commonly used in other models's styles (DeepSeek R1, Qwen2.5 Coder) and actually helps disambiguate some outputs, should be straightforward to support. Adding to my todo list, might not get to it before pushing some of the tool diff work tho.

(and then we'll be able to just use --chat-template-file <( ./scripts/get_chat_template.py watt-ai/watt-tool-70B ))

[edmcman]

Supporting the markdown format seems like a good thing to support in general.

About the watt models in particular

The model page has an example using the Llama 3.3 pythonic syntax. watt-ai/watt-tool-70B's chat template is identical to the Llama 3.3 HF chat template, which uses the Llama JSON function calling syntax. I hacked up a template here for the pythonic syntax, but llama.cpp did not seem to be able to parse any of the returned calls either. I'm guessing that isn't supported, which isn't surprising since it seems like all of Meta's jinja files use the JSON format.

I don't know if the json vs. pythonic format matters, but I'm thinking it might since the watt developers seem to have gone out of their way to make their example use the pythonic format.