Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

instructor not working with litellm. AttributeError: 'dict' object has no attribute 'audio_tokens' #1339

Open
brycedrennan opened this issue Feb 6, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@brycedrennan
Copy link

Describe the bug
instructor doesn't work with litellm.

Traceback (most recent call last):
  File ".pyenv/versions/3.10.13/envs/job-listing-monitor-3.10.13/lib/python3.10/site-packages/instructor/retry.py", line 158, in retry_sync
    response = update_total_usage(
  File ".pyenv/versions/3.10.13/envs/job-listing-monitor-3.10.13/lib/python3.10/site-packages/instructor/utils.py", line 151, in update_total_usage
    ttd.audio_tokens = (ttd.audio_tokens or 0) + (rtd.audio_tokens or 0)
AttributeError: 'dict' object has no attribute 'audio_tokens'

model: gpt-4o-mini-2024-07-18
versions:

instructor==1.7.2
litellm==1.44.10
openai==1.60.2

To Reproduce

from litellm import completion
from pydantic import BaseModel
import instructor

class ChatClassification(BaseModel):
    relevant: bool
    message: str

client = instructor.from_litellm(completion)

resp = client.chat.completions.create(
    model="gpt-4o-mini-2024-07-18",
    messages=[{"role": "user", "content": "Is a secret answer relevant?"}],
    response_model=ChatClassification,
)

Expected behavior
The I can make llm calls without errors.

Hints
Looks like litellm doesn't fully convert the usage information to objects.
https://github.com/BerriAI/litellm/blob/c17342ac5bf0e9b281ffe2dbf6380411856eaf96/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py#L511-L514

See also #1332

@github-actions github-actions bot added the bug Something isn't working label Feb 6, 2025
@villqrd
Copy link

villqrd commented Feb 21, 2025

Confirm - does not work with litellm. @brycedrennan have you found a work around or another router library that works with Instructor?
Best

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants