Skip to content

Add total token counts to models #1282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

gotDaijobu
Copy link

Description
This PR adds total token count tracking functionality to the models in the smolagents package. While the existing implementation already tracks last_input_token_count and last_output_token_count for individual calls, this enhancement adds cumulative tracking with total_input_token_count, total_output_token_count, and total_token_count properties to provide users with better visibility into token usage across an entire agent session.

Motivation
Token usage is a critical metric for LLM applications, especially in production environments where costs need to be monitored and optimized. Having access to cumulative token counts allows developers to:

  • Monitor the total cost of an agent run
  • Set token budgets and limits for production deployments
  • Benchmark different prompt strategies for efficiency
  • Provide transparency to end users about resource usage

This also enable the use of several models combined (more support of this maybe in future works) to reduce costs depending on the tasks.

Changes

Added total_input_token_count, total_output_token_count properties to base model classes
Initialized these counters to zero in constructors
Added support in the serialization methods
Added update_token_counts method in the base model class
Updated the token counting logic to increment these totals whenever new tokens are processed in all child classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant