feat: Add Toolset to tooling architecture

[vblagoje]

Why:

Enhances the tooling infrastructure by introducing a new abstraction, the Toolset class, to facilitate the grouping and management of related tool functionalities, supporting dynamic tool loading and registration.

What:

• Added a new Toolset class to manage collections of tools and enable dynamic loading of tool configurations.
• Modified __init__.py to include the Toolset in the exposed API.
• Added comprehensive tests for the Toolset class to ensure its functionality.

How can it be used:

The Toolset class can be used to group related tools into a cohesive unit, allowing for easier management and potential dynamic loading from external sources:

def add_numbers(a: int, b: int) -> int:
    return a + b

def subtract_numbers(a: int, b: int) -> int:
    return a - b

# Create the tools
add_tool = Tool(
    name="add",
    description="Add two numbers",
    parameters={
        "type": "object",
        "properties": {"a": {"type": "integer"}, "b": {"type": "integer"}},
        "required": ["a", "b"],
    },
    function=add_numbers,
)

subtract_tool = Tool(
    name="subtract",
    description="Subtract two numbers",
    parameters={
        "type": "object",
        "properties": {"a": {"type": "integer"}, "b": {"type": "integer"}},
        "required": ["a", "b"],
    },
    function=subtract_numbers,
)

# Create a toolset
math_toolset = Toolset([add_tool, subtract_tool])

# Create a complete pipeline
pipeline = Pipeline()
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini", tools=math_toolset))
pipeline.add_component("tool_invoker", ToolInvoker(tools=math_toolset))
pipeline.add_component(
    "adapter",
    OutputAdapter(
        template="{{ initial_msg + initial_tool_messages + tool_messages }}",
        output_type=list[ChatMessage],
        unsafe=True,
    ),
)
pipeline.add_component("response_llm", OpenAIChatGenerator(model="gpt-4o-mini"))

# Connect the components
pipeline.connect("llm.replies", "tool_invoker.messages")
pipeline.connect("llm.replies", "adapter.initial_tool_messages")
pipeline.connect("tool_invoker.tool_messages", "adapter.tool_messages")
pipeline.connect("adapter.output", "response_llm.messages")

# Test addition through the pipeline
user_input = "What is 5 plus 3?"
user_input_msg = ChatMessage.from_user(text=user_input)

result = pipeline.run({"llm": {"messages": [user_input_msg]}, "adapter": {"initial_msg": [user_input_msg]}})

# Verify the complete flow worked
pipe_result = result["response_llm"]["replies"][0].text
assert "8" in pipe_result or "eight" in pipe_result

How did you test it:

Tests were written to validate the Toolset functionality, including registration of tools, serialization/deserialization, and integration within a pipeline. Specific methods like ToolInvoker were used to test tool invocation. Tests confirm that tools can be dynamically loaded, executed, and serialized correctly.

Notes for the reviewer:

Please pay special attention to the dynamic loading capabilities and serialization logic within the Toolset. Ensure there are no name conflicts when registering tools and that the integration tests cover all intended use cases.

[coveralls]

Pull Request Test Coverage Report for Build 14266761775

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 69 unchanged lines in 5 files lost coverage.
• Overall coverage decreased (-0.2%) to 90.114%

---

Files with Coverage Reduction	New Missed Lines	%
tools/tool.py	5	92.45%
utils/misc.py	6	72.41%
components/generators/chat/openai.py	7	96.07%
components/tools/tool_invoker.py	13	90.51%
components/embedders/openai_document_embedder.py	38	61.61%

Totals
Change from base Build 14240019342:	-0.2%
Covered Lines:	10637
Relevant Lines:	11804

---

💛 - Coveralls

[anakin87]

• I don't particularly like the fact that we would have to change all ChatGenerators, but right now I don't see any other way to do Pipeline/YAML support
• We also talked about giving users the ability to filter tools e.g. from MCP server. How do you envision this? How can this be supported in Pipeline/YAML?

[vblagoje]

• I don't particularly like the fact that we would have to change all ChatGenerators, but right now I don't see any other way to do Pipeline/YAML support

Yeah - neither do I but as you say - there doesn't seem to be another option. Perhaps we can zero-in on the changes required - as exemplified - by OpenAIChatGenerator - so that the changes are minimal, clear and easy to apply in one swoop over all chat generators, e.g. in this early cycle and targeting for 2.13 release

• We also talked about giving users the ability to filter tools e.g. from MCP server. How do you envision this? How can this be supported in Pipeline/YAML?

I would defer it to custom impls of Toolset subclasses as they are all free to do whatever filtering the see the most suitable. For example via init param as a list of tool names.

[anakin87]

• We also talked about giving users the ability to filter tools e.g. from MCP server. How do you envision this? How can this be supported in Pipeline/YAML?

I would defer it to custom impls of Toolset subclasses as they are all free to do whatever filtering the see the most suitable. For example via init param as a list of tool names.

I understand. It would be great if you can show a pseudocode example for this use case.

Thinking of MCP, this seems a common use case: you have several tools available, but you want to restrict your Pipeline to only use some of them.

[vblagoje]

Ok, yes roughly something like this

[anakin87]

The overall design looks good.

I found some details to be refined.

[vblagoje]

@bilgeyucel have a look now please

[vblagoje]

Have another look @anakin87 at the pydoc of Toolset - much better now IMHO, thanks for single-ing this out

[anakin87]

This PR is getting a little hard to follow and review.

Here's what we could try:

• Go through my comments and close the related GitHub conversations (linking the relevant commits if possible).
• Tidy up the tests: remove duplicates or less important ones, and consolidate most of them in test_toolset.py to keep things readable.
• Keep in mind that integration tests don't contribute to the coverage - let's try to avoid them because they don't give benefits in this case and increase time spent by the CI.
• Ping me when you need another review (or use the Re-request review button).

[vblagoje]

@anakin87:

• Minimised number of integration tests with 6622e27
• Toolset with Tool init change
• Remove some openai + toolset test that are tested elsewhere with 128d16a
• Closed all relevant conversations with comment if applicable

The PR should be as compact as possible now. Have a look 🙏

[anakin87]

It's much better.

I have some suggestions for tests removal.

[vblagoje]

@anakin87 I was a bit hesitant to commit the suggested changes above - but check the latest commit - I think it does what you suggested

[anakin87]

Ok for me.

Feel free to ask @bilgeyucel for another review.

[vblagoje]

@bilgeyucel @dfokina I'll integrate this now so we can update all ChatGenerators to use Toolset - if something minor comes up we can adjust it, still lots of time till next release.