[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

unaidedelf8777 · 2025-04-03T00:44:53Z

Adds outlines as a guided decoding backend for V1, and updates the integration for V0.

The aim of this is three fold:

Remove the dependency on outlines, and only use outlines_core
performance gains for V0 using the write_mask_into method on Guide to write a bitmask in-place for use in logits masking.
outlines backend for V1

Because the dependency on outlines will be removed, support for grammar based decoding with the outlines backend will also be removed (CFG classes reside in the outlines package)

cc @aarnphm

github-actions · 2025-04-03T00:45:05Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

unaidedelf8777 · 2025-04-03T00:49:54Z

NOTE: Can't be merged until next version of outlines_core is released.

russellb · 2025-04-07T20:04:36Z

Thank you for the PR! I will review it this week.

aarnphm

I reviewed the v0 code path. One ask is to add tests for this for disabling cache path.

And we should update the requirements/common.txt to the lowest version of outlines-core supported.

vllm/model_executor/guided_decoding/outlines_logits_processors.py

mergify · 2025-04-11T01:24:31Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @unaidedelf8777.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

unaidedelf8777 · 2025-04-13T17:54:51Z

@russellb @aarnphm All code is done. If you guys approve of it I’ll go ahead and clean up all the linter complaints, and then it should be ready.

Also outlines-core update has been pushed to pypi and pinned here (v0.2.9)

aarnphm

First round of review. A few things needs to be addressed here. but great progress so far.

vllm/model_executor/guided_decoding/outlines_decoding.py

vllm/model_executor/guided_decoding/outlines_logits_processors.py

vllm/v1/structured_output/backend_outlines.py

mergify · 2025-04-18T23:44:25Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @unaidedelf8777.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2025-05-15T00:05:47Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @unaidedelf8777.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

aarnphm · 2025-05-15T00:10:44Z

Ok, that one got merged, let's proceed with this one and get this in for 0.9.0

commit f861a35 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Thu May 1 18:39:16 2025 -0500 Update vllm/v1/structured_output/backend_outlines.py Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com> commit 50886e0 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Thu May 1 13:22:46 2025 -0500 Fix typo in merge conflict resolution commit cf92ccf Merge: 7bc8167 7423cf0 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Thu May 1 11:51:47 2025 -0500 Merge branch 'main' into update-outlines-integration commit 7bc8167 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Thu Apr 24 00:33:40 2025 +0000 Make pre-commit happy Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com> commit 5f2855d Merge: 0bf4bdf b07d741 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Wed Apr 23 19:07:34 2025 -0500 Merge branch 'main' into update-outlines-integration commit 0bf4bdf Merge: 2d67e88 5536b30 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Tue Apr 22 11:38:21 2025 -0500 Merge branch 'main' into update-outlines-integration commit 2d67e88 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Mon Apr 21 21:43:08 2025 +0000 format test last commit Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com> commit d5bdab7 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Mon Apr 21 16:44:32 2025 +0000 fix part of failing test which I can fix. Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com> commit 058ab4b Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Sun Apr 20 05:35:45 2025 +0000 Fix bugs from outlines -> outlines_core behavior differences Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com> commit 04928c8 Author: Nathan Hoos <thwackyy.y@gmail.com> Date: Sat Apr 19 17:32:36 2025 +0000 Add outlines backend to V1 Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com> Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 · 2025-05-16T01:41:18Z

Merge conflicts with your last PR are resolved. The test failure also seems to be unrelated to any changes in this PR.

aarnphm · 2025-05-16T07:49:12Z

hmm maybe tried to merge from main ?

aarnphm · 2025-05-17T00:50:48Z

@unaidedelf8777 I think only the logit processor tests in v0 is related to this PR here (at least for the outlines case)

PTAL

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 requested review from mgoin, russellb, WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners April 3, 2025 00:44

unaidedelf8777 changed the title ~~[V0][V1][Core] Add outlines integration for V1, and update V0 integration.~~ [V0][V1][Core] Add outlines integration for V1, and update V0 integration. [DO NOT MERGE] Apr 3, 2025

mergify bot added the v1 label Apr 3, 2025

unaidedelf8777 marked this pull request as draft April 3, 2025 00:46

unaidedelf8777 changed the title ~~[V0][V1][Core] Add outlines integration for V1, and update V0 integration. [DO NOT MERGE]~~ [V0][V1][Core] Add outlines integration for V1, and update V0 integration. Apr 3, 2025

hmellor mentioned this pull request Apr 3, 2025

Removes duplicate outlines processors #6900

Closed

mergify bot added the structured-output label Apr 4, 2025

aarnphm approved these changes Apr 7, 2025

View reviewed changes

vllm/model_executor/guided_decoding/outlines_logits_processors.py Outdated Show resolved Hide resolved

mergify bot added tpu Related to Google TPUs ci/build and removed tpu Related to Google TPUs labels Apr 9, 2025

mergify bot added needs-rebase and removed needs-rebase labels Apr 11, 2025

unaidedelf8777 marked this pull request as ready for review April 13, 2025 17:55

aarnphm suggested changes Apr 14, 2025

View reviewed changes

mergify bot added needs-rebase and removed needs-rebase labels Apr 18, 2025

mergify bot added the needs-rebase label May 15, 2025

unaidedelf8777 added 10 commits May 15, 2025 22:08

Add rollback and validate_tokens methods. Final touches.

fa24b70

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

bump outlines version

98ba45d

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

explain to pre-commit that its wrong

de09732

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pin newest outlines version

fd1a994

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix interface to be compliant with vllm-project#16577

5388bdc

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

make pre-commit happy

865d8a9

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

artifact from a prior merge

309c968

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix failing test

126328c

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

make pre-commit happy again

8022c7a

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 force-pushed the update-outlines-integration branch from db324d3 to 8022c7a Compare May 15, 2025 22:10

mergify bot removed the needs-rebase label May 15, 2025

simon-mo added the ready ONLY add when PR is ready to merge/full CI is needed label May 16, 2025

simon-mo enabled auto-merge (squash) May 16, 2025 05:30

aarnphm mentioned this pull request May 17, 2025

[V0][Bugfix] Fix parallel sampling performance regression when guided decoding is enabled #17731

Open

aarnphm removed this from the v0.9.0 milestone May 17, 2025

auto-merge was automatically disabled May 17, 2025 01:52
Head branch was pushed to by a user without write access

fix test_guided_json_object

3d4c13d

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 force-pushed the update-outlines-integration branch from 64c0086 to 3d4c13d Compare May 17, 2025 01:52

unaidedelf8777 added 3 commits May 17, 2025 16:06

Fix test_guided_processors to be compatible with new integration

6ad15dc

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit

53be045

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix

05023f6

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 force-pushed the update-outlines-integration branch from e70ee76 to 05023f6 Compare May 17, 2025 17:42

fix tests

8f12c49

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

unaidedelf8777 commented Apr 3, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Apr 3, 2025

unaidedelf8777 commented Apr 3, 2025

russellb commented Apr 7, 2025

aarnphm left a comment

mergify bot commented Apr 11, 2025

unaidedelf8777 commented Apr 13, 2025

aarnphm left a comment

mergify bot commented Apr 18, 2025

mergify bot commented May 15, 2025

aarnphm commented May 15, 2025

unaidedelf8777 commented May 16, 2025 •

edited

Loading

aarnphm commented May 16, 2025

aarnphm commented May 17, 2025

[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

Are you sure you want to change the base?

[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

Conversation

unaidedelf8777 commented Apr 3, 2025 • edited by github-actions bot Loading

github-actions bot commented Apr 3, 2025

unaidedelf8777 commented Apr 3, 2025

russellb commented Apr 7, 2025

aarnphm left a comment

Choose a reason for hiding this comment

mergify bot commented Apr 11, 2025

unaidedelf8777 commented Apr 13, 2025

aarnphm left a comment

Choose a reason for hiding this comment

mergify bot commented Apr 18, 2025

mergify bot commented May 15, 2025

aarnphm commented May 15, 2025

unaidedelf8777 commented May 16, 2025 • edited Loading

aarnphm commented May 16, 2025

aarnphm commented May 17, 2025

unaidedelf8777 commented Apr 3, 2025 •

edited by github-actions bot

Loading

unaidedelf8777 commented May 16, 2025 •

edited

Loading