Add NeuronxDistributedInference support, Speculative Decoding, Dynamic on-device sampling#16357

Merged

simon-mo merged 44 commits intovllm-project:mainfrom

aws-neuron:upstream-neuron-vllm-04-08

May 7, 2025

Commits on Apr 9, 2025

[NxDI upstream foundation] set up NxDI model runner

Aaron Dou
authored and
aws-satyajith
committed
Support speculation with transformers-neuronx

sssrijan-amazon
authored and
aws-satyajith
committed
Add support for eagle speculation using transformers-neuronx

sssrijan-amazon
authored and
aws-satyajith
committed
Support speculation with neuronx-distributed-inference for batch 1

sssrijan-amazon
authored and
aws-satyajith
committed
[Tnx] Fix streaming flow for speculation

sssrijan-amazon
authored and
aws-satyajith
committed
[NxdI] Support eagle speculation

sssrijan-amazon
authored and
aws-satyajith
committed
[TNx+EAGLE] Use FusedSpeculativeDecoder for EAGLE + Linear token tree.

chongmni-aws
authored and
aws-satyajith
committed
Fix the termination check for accepted speculative tokens.

chongmni-aws
authored and
aws-satyajith
committed
[TNx][Bug fix] fix the incorrect speculation output check.

chongmni-aws
authored and
aws-satyajith
committed
Add continuous batching with eagle

aws-amulyaab
authored and
aws-satyajith
committed
[NxDI] Fix masking of padding in speculative output

aws-patlange
authored and
aws-satyajith
committed
Remove assertion on bs=1 when using speculation now that we support bs>1

elaineyz
authored and
aws-satyajith
committed
Modify NxDI and TNx multi step model runners (used for speculation) to conform to the new VllmConfig construct

elaineyz
authored and
aws-satyajith
committed
Add multi-step NxD model runner

aws-tailinpa
authored and
aws-satyajith
committed
Add Framework selection logic
aws-satyajith
committed
Fix no free blocks error

aws-patlange
authored and
aws-satyajith
committed
Refactor and add basic docstrings

elaineyz
authored and
aws-satyajith
committed
Modification to enable Vllm-neuronx instead of Vllm for KTF

aws-navyadhara
authored and
aws-satyajith
committed
Fix global_top_k to be aligned with NxDI default

aws-yishanm
authored and
aws-satyajith
committed
Add neuron model runner tests for updating sampling param

chongmni-aws
authored and
aws-satyajith
committed
Updating requirements-neuron.txt

aws-navyadhara
authored and
aws-satyajith
committed
Updating requirements-neuron.txt

aws-navyadhara
authored and
aws-satyajith
committed
multi-node TP support

Aaron Dou
authored and
aws-satyajith
committed
Removing codenames that fail IP Scanning

aws-navyadhara
authored and
aws-satyajith
committed
Format auto-check and formatting changes
aws-satyajith
committed
Bug fix: Missing NxDI model runner addressed
aws-satyajith
committed
set world_size default value as 1

Aaron Dou
authored and
aws-satyajith
committed
Formatting changes to satisfy all pre-commit hooks
aws-satyajith
committed
Revert "Removing codenames that fail IP Scanning"

aws-navyadhara
authored and
aws-satyajith
committed
Fix issues identified by mypy checks
aws-satyajith
committed
Fix logging strings
aws-satyajith
committed
add example offline script for EAGLE spec

elaineyz
authored and
aws-satyajith
committed
Merge branch 'upstreaming_main' into upstream-neuron-vllm-04-08
aws-satyajith
committed
Add speculative token tree and skip EAGLEConfig creation for Neuron
aws-satyajith
committed
Fix imports and satisfy pre-commit hooks
aws-satyajith
committed
Remove deprecated files and .gitignore additions
aws-satyajith
committed
Merge branch 'upstreaming_main' into upstream-neuron-vllm-04-08
aws-satyajith
committed

Commits on Apr 22, 2025

Commits on Apr 28, 2025

Merge branch 'upstreaming_main' into upstream-neuron-vllm-04-08
aws-satyajith
committed

Commits on May 2, 2025

Commits on May 3, 2025