[Bug] lm.history[-1].usage shows nothing

[NumberChiffre]

What happened?

Hey guys,

I use langchain for part of my workflow, but mostly with dspy, so I'm calling lm.history to get token count. I want to also add the token cost, which unfortunately returns nothing. I believe this is a bug as there are no other fields in lm.history for this purpose.

Thanks in advance for any help, much appreciated!

Steps to reproduce

This is a snippet function I use, but I removed the usage part as there's nothing:

def get_total_token_usage(lms: list[dspy.LM], openai_callback_handler: OpenAICallbackHandler) -> dict[str, float]:
    dspy_tokens = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
    for current_lm in lms:
        for entry in current_lm.history:
            usage = entry["usage"]
            dspy_tokens["prompt_tokens"] += usage.get("prompt_tokens", 0)
            dspy_tokens["completion_tokens"] += usage.get("completion_tokens", 0)
            dspy_tokens["total_tokens"] += usage.get("total_tokens", 0)

    langchain_tokens = {
        "prompt_tokens": openai_callback_handler.prompt_tokens,
        "completion_tokens": openai_callback_handler.completion_tokens,
        "total_tokens": openai_callback_handler.total_tokens,
    }
    return {
        "langchain_prompt_tokens": langchain_tokens["prompt_tokens"],
        "dspy_prompt_tokens": dspy_tokens["prompt_tokens"],
        "langchain_completion_tokens": langchain_tokens["completion_tokens"],
        "dspy_completion_tokens": dspy_tokens["completion_tokens"],
        "langchain_total_tokens": langchain_tokens["total_tokens"],
        "dspy_total_tokens": dspy_tokens["total_tokens"],
        "langchain_successful_requests": openai_callback_handler.successful_requests,
        "langchain_total_cost": openai_callback_handler.total_cost,
    }

DSPy version

2.6.0rc8

[okhat]

Hey! I can't understand the role of langchain in this bug report. Can you get the same issue if you eliminate that factor?

[NumberChiffre]

@okhat I'm getting the same issue with or without langchain, I added langchain in this code because I could only get a portion of the total cost whereas most of my cost comes from dspy. So the problem is dspy's lm.history[-1].usage isn't providing a cost, which leads me to think it is a bug rather than a feature request.