Docs: add tutorials on speculative decoding main page and EAGLE sub page #131
Conversation
ziqif-nv
requested review from
harryskim,
krishung5,
oandreeva-nv and
statiraju
ziqif-nv
changed the title
| --input-file /data/converted_humaneval.jsonl \ | ||
| --tokenizer /hf-models/vicuna-7b-v1.3/ \ | ||
| --concurrency 1 \ | ||
| --measurement-interval 4000 \ |
There was a problem hiding this comment.
Would recommend to use request-rate option rather measurement interval as for long tests these are often requires longer stabalization window.
We are recommending GenAI-perf customers to start using request-rate as default option.
|
|
||
| 2. Get Gen-AI Perf Tool | ||
|
|
||
| Gen-AI Perf is available in the SDK container as shown in the [Send an Inference Request](#send-an-inference-request) section. The only difference is that you need to mount the converted dataset to the container: |
There was a problem hiding this comment.
This link to me look likes is broken can you please confirm?
There was a problem hiding this comment.
I verified that the link works for me. maybe it is PR render issue on github
| 2. Get Gen-AI Perf Tool | ||
|
|
||
| Gen-AI Perf is available in the SDK container as shown in the [Send an Inference Request](#send-an-inference-request) section. The only difference is that you need to mount the converted dataset to the container: | ||
|
|
There was a problem hiding this comment.
I am not sure if you need to add installation section for GenAI-Perf here.
I created a smaple PR here for your reference: https://github.com/sgl-project/sglang/pull/3552/files
There was a problem hiding this comment.
updated the PR by utilizing the README you pointed. thanks!
| If the first assumption holds true, the latency of speculative decoding will no worse than the standard approach. If the second holds, output token generation advances by statistically more than one token per forward pass. | ||
| The combination of both these allows speculative decoding to result in reduced latency. | ||
|
|
||
| ## Performance Improvements |
There was a problem hiding this comment.
i woud like to see under what specific tasks would it be efficient to use. what kind of models should be used. draft models and the target model examples if any. I would like to see such recommendations.
Send out for a quick review within the team.
Will keep polishing