Skip to content

Commit fe0fd9d

Browse files
authored
openai[patch]: upgrade tiktoken and fix test (#30621)
Related to #30344 #30542 introduced an erroneous test for token counts for o-series models. tiktoken==0.8 does not support o-series models in `tiktoken.encoding_for_model(model_name)`, and this is the version of tiktoken we had in the lock file. So we would default to `cl100k_base` for o-series, which is the wrong encoding model. The test tested against this wrong encoding (so it passed with tiktoken 0.8). Here we update tiktoken to 0.9 in the lock file, and fix the expected counts in the test. Verified that we are pulling [o200k_base](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py#L8), as expected.
1 parent 3880787 commit fe0fd9d

File tree

2 files changed

+34
-34
lines changed

2 files changed

+34
-34
lines changed

libs/partners/openai/tests/unit_tests/test_token_counts.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@
1010
"gpt-4": 12,
1111
"gpt-4-32k": 12,
1212
"gpt-3.5-turbo": 12,
13-
"o1": 12,
14-
"o3": 12,
13+
"o1": 11,
14+
"o3": 11,
1515
"gpt-4o": 11,
1616
}
1717

libs/partners/openai/uv.lock

Lines changed: 32 additions & 32 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)