Skip to content

feat: add vllm v1 engine tracing #17992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1,354 commits into from
Closed

feat: add vllm v1 engine tracing #17992

wants to merge 1,354 commits into from

Conversation

RichardoMrMu
Copy link

following this issue #17794, and #10582 this issue ([P2] Make it work with tracing too (there's some request level stats that tracing needs, like queue time, ttft). These request level metric should be possible to be surfaced in v1 too.). I make engine v1 to generate model tracing span.

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added documentation Improvements or additions to documentation v1 labels May 12, 2025
gmarinho2 and others added 21 commits May 12, 2025 19:19
@gmarinho2 @maxdebayser
Truncation control for embedding models (vllm-project#14776)
5d41d27 
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@huydhn
Update PyTorch to 2.7.0 (vllm-project#16859)
6fa7a6f 
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@hmellor
Improve configs - ModelConfig (vllm-project#17130)
f6417b0 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@hmellor
Fix call to logger.info_once (vllm-project#17416)
6d2f2fb 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@huydhn
Fix some speculative decode tests with tl.dot (vllm-project#17371)
149bc42 
Signed-off-by: Huy Do <huydhn@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@mgoin
Support LoRA for Mistral3 (vllm-project#17428)
7a56465 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@jikunshang
[Intel GPU] [CI]Fix XPU ci, setuptools >=80.0 have build issue (vllm-…
2d3d317 
…project#17298)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@jikunshang @mayuyuace
[Hardware][Intel GPU] Upgrade to torch 2.7 (vllm-project#17444)
0436cfb 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@chaunceyjiang
[Bugfix] Fix AttributeError: 'State' object has no attribute 'engine_…
365ddd9 
…client' (vllm-project#17434)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@mlinmg @Isotr0py
[MODEL ADDITION] Ovis2 Model Addition (vllm-project#15826)
ba29574 
Signed-off-by: Marco <121761685+mlinmg@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@houseroad
Make the _apply_rotary_emb compatible with dynamo (vllm-project#17435)
82279b6 
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@chaunceyjiang
[Misc] Remove deprecated files (vllm-project#17447)
54bf2ee 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@lengrongfu
[V1][Bugfix]: vllm v1 verison metric num_gpu_blocks is None (vllm-pro…
8f9e430 
…ject#15755)

Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@NickLucche
[TPU][V1][CI] Update regression test baseline for v6 CI (vllm-project…
298341f 
…#17064)

Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@dr75
[Core] Prevent side-channel attacks via cache salting (vllm-project#1…
3713d2e 
…7045)

Signed-off-by: Marko Rosenmueller <5467316+dr75@users.noreply.github.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@alec-flowers @markmc
[V1][Metrics] add support for kv event publishing (vllm-project#16750)
474a5af 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@chaunceyjiang
[Feature] The Qwen3 reasoning parser supports guided decoding (vllm-p…
8940d38 
…roject#17466)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@russellb
[Docs] Add command for running mypy tests from CI (vllm-project#17475)
d1db451 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@aarnphm
[Fix] Support passing args to logger (vllm-project#17425)
f603f84 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@psav
[Bugfix] Fixed mistral tokenizer path when pointing to file (vllm-pro…
a72cd88 
…ject#17457)

Signed-off-by: Pete Savage <psavage@redhat.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@russellb @DarkLight1337
[V1] Allow turning off pickle fallback in vllm.v1.serial_utils (vllm-…
efcaaae 
…project#17427)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Potabk and others added 6 commits May 12, 2025 19:19
@Potabk
[Bugfix] Fix pydantic.errors.PydanticUserError (vllm-project#17962)
c0eaf9b 
Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@lsy323
[Bugfix][TPU] Use np array when updating cache slot_mapping (vllm-pro…
303e6e4 
…ject#17971)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@b8zhong
[Fix] Benchmark "EngineClient" has no attribute "model_config" (vll…
9063f3d 
…m-project#17976)

Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@Xu-Wenqing
[Feature] Support DeepSeekV3 Function Call (vllm-project#17784)
0d0279c 
Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>
Signed-off-by: Xu Wenqing <xuwq1993@qq.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@AaruniAggarwal
Correcting testcases in builkite job for IBM Power (vllm-project#17675)
8dbe604 
Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@jeejeelee
[Misc] Improve modelscope import error (vllm-project#17983)
bc6f4dd 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@mergify mergify bot added the tpu Related to Google TPUs label May 12, 2025
@mergify Mergify
Copy link

mergify bot commented May 12, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @RichardoMrMu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@russellb russellb Awaiting requested review from russellb russellb is a code owner

@LiuXiaoxuanPKU LiuXiaoxuanPKU Awaiting requested review from LiuXiaoxuanPKU LiuXiaoxuanPKU is a code owner

@KuntaiDu KuntaiDu Awaiting requested review from KuntaiDu KuntaiDu is a code owner

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123 zhuohan123 is a code owner

Assignees
No one assigned
Labels
ci/build documentation Improvements or additions to documentation frontend multi-modality Related to multi-modality (#4194) needs-rebase speculative-decoding structured-output tool-calling tpu Related to Google TPUs v1
Projects
Structured Output
Status: Done
Tool Calling
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.