Thanks for reporting the regression and the details report.

In principle, I'm OK with reverting the change to restore the tool_call_id in the short term. For future releases, do you have any suggestions on how we might better expose or preserve the tool call ID in a more robust or consistent way? Would be great to make this behavior more intentional and less prone to accidental breakage.

Hey Albert, I think the main issue is that currently, the smolagents API assumes that models will want to consume tool calls and responses as raw text. We can see that here and here where messages are transformed into a message with content type: text.

However, for both the OpenAI and Anthropic APIs, you are supposed to return tool calls/responses using a special content type (function_call/function_call_output for OpenAI and tool_use/tool_result for Anthropic) which encodes information about the tool call in structured JSON.

I think this problem of needing to transform the raw text messages returned from smolagents to this structured format is the key point which is making the current API difficult to work with.

Ideally, ActionStep.to_messages should serialize tool calls/observations into a structured format where the tool calls, tool results, and associated IDs can be accessed as individual fields in the resulting dictionary. e.g. in to_messages, you could handle self.tool_calls and self.observations like this:

if self.tool_calls is not None:
    messages.append(
        Message(
            role=MessageRole.TOOL_CALL,
            content=[
                {
                    "type": "tool_use",
                    "id": tool_call.id,
                    "name": tool_call.name,
                    "arguments": make_json_serializable(tool_call.arguments)
                }
                for tool_call in self.tool_calls
            ],
        )
    )

if self.observations is not None:
    messages.append(
        Message(
            role=MessageRole.TOOL_RESPONSE,
            content=[
                # ideally, we should have one content block per tool response to avoid hardcoding self.tool_calls[0].id
                # as-is, this will break if there is more than one tool response in a single message
                # unfortunately, since `self.observations` is just `str` instead of `list[str]` it is probably the best we can do
                {
                    "type": "tool_result",
                    "tool_use_id": self.tool_calls[0].id,
                    "content": self.observations,
                }
            ],
        )
    )

The exact format of these dictionaries is somewhat arbitrary as long as they contain all of the required information.

I used something close to Anthropic's format here, but you could easily model them after OpenAPI/HuggingFace/LiteLLM/whatever instead. The idea is that consuming models could then adapt this dictionary to the exact format expected by their provider.

Unfortunately, this would be a major breaking change, so I'm not sure if it would be feasible for you to implement.

Maybe one alternative would be to pass AgentMemory.steps directly the model in addition to the serialized messages? Then the model could use ActionStep directly to construct a message in the desired format. The downside of this is that it would drastically increase your API surface though.