Instructor appears to be overwriting tool calls

[Terrevue]

It appears that Instructor is overwriting the tool array and replacing it with BaseAgentOutputSchema only. How are you guys passing tool functions to, in this case, the OpenAI API, so that they don't get overwritten?

[KennyVaneetvelde]

Heya,

It is important to note perhaps that, underlying everything, an LLM never actually calls a tool in any library or framework but rather it is always the developer (you or the library dev) that implements some logic to do so.

That being said, Instructor & Atomic Agents makes this fact much more explicit and takes a more developer-first approach.

Please have a look at some of the examples, but I think this example will help you along the most:
https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/deep-research

Here you can see an agent generating a "tool call" schema for the search tool, for example..

Some more examples that can help you:

• https://github.com/KennyVaneetvelde/persistent-memory-agent-example
• https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/orchestration-agent

[Terrevue]

Thanks for the quick response. I understand how "tool" calling works. What I am attempting to do is pass in the json function definition in the model_api_parameters:

{'temperature': 0.2, 'max_tokens': 4096, 'tools': [{'type': 'function', 'function': {'name': 'enhanced_file_reader', 'description': 'Enhanced tool for reading various file formats.', 'parameters': {'type': 'object', 'properties': {'file_path': {'type': 'string', 'description': 'Path to the file to read'}, 'extract_images': {'type': 'string', 'description': 'Whether to extract and describe images (for formats that support it)'}, 'start_page': {'type': 'string', 'description': 'Starting page for multi-page documents (1-indexed)'}, 'end_page': {'type': 'string', 'description': 'Ending page for multi-page documents (inclusive)'}}, 'required': []}}}], 'tool_choice': 'auto'}

However, once Inspector processes the params, it overwrites the tools, leaving only BaseAgentOutputSchema:

'model': 'gpt-4o-mini', 'temperature': 0.2, 'max_tokens': 4096, 'tools': [{'type': 'function', 'function': {'name': 'BaseAgentOutputSchema', 'description': 'This schema represents the response generated by the chat agent.', 'parameters': {'properties': {'chat_message': {'description': 'The chat message exchanged between the user and the chat agent. This contains the markdown-enabled response generated by the chat agent.', 'title': 'Chat Message', 'type': 'string'}}, 'required': ['chat_message'], 'type': 'object'}}}], 'tool_choice': {'type': 'function', 'function': {'name': 'BaseAgentOutputSchema'}}}

While I realize that I can call instructor directly:

client = instructor.from_openai( openai.OpenAI(), mode=instructor.Mode.FUNCTIONS # This is the key change )

I would prefer to use a non-breaking approach using native Atomic Agent friendly functionality, if possible. Does that make sense?

[Terrevue]

@KennyVaneetvelde

Unless I am missing something, it appears that the tool calls from those examples are managed externally rather than delegated to an agent directly through function calling. What I am attempting to do is create a framework that allows the agent to directly choose which tools to use and the params to pass to those tools. I am intentionally making it generic to allow agents the latitude to take actions as they deem necessary. The examples I have seen (again, if I am missing something, please let me know) appear to make the decisions and run the tools outside the scope of the agent and simply provide the results back to the agent.

Hopefully that adds some clarity to my original question.

Thanks!

[KennyVaneetvelde]

hi @Terrevue , I do get your original question, but it is coming from a bit of a wrong paradigm.

Atomic Agents chooses to lay bare how things work under the hood, in order to give more control to the developer, which is a thing we have found is highly valued in enterprise - And also to demystify things that are the result of certain abstractions (i.e: You say "chat messages vs tool calls", I say "it's all just JSON, man"; You say: "The agent used the search tool", I say: "The agent returned some JSON that signifies that we should execute some code rather than print a message") - In the end, LLMs are only capable of one single thing, and that is data input & output.

An agent is no different, really, it is the exact same LLM, what changes is the orchestration around it. But all of that is traditional code.

To that end, everything is self-consistent. Agents & Tools within Atomic Agents follow the exact same structure for a reason, because both agents and tools are just input -> processing -> output

And so, what instructor does, and Atomic Agents embraces, is that everything follows the same path, so, everything is a tool call, or everything is a message, or everything is just JSON.

Perhaps conceptually it helps if you think of normal chat message being a tool as well, a tool that prints something to your console or sends some text over an API...

Either way, this way you have full control, full debuggability, ...)

I hope this kind of answers your question

Note: This doesn't mean you can't do what you want to do, though, you can certainly give agents full and complete autonomy, you just have to stop looking at agents as if they are this special magical thing but just look at them as normal, albeit smart, IPO (Input->processing->output) functions. In fact someone else has already done a lot of work around that as well, you might want to go have a look at his repository: https://github.com/mrseanryan/gpt-multi-atomic-agents