Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Math-500 and AIME24 evals #4

Merged
merged 3 commits into from
Jan 24, 2025
Merged

Adds Math-500 and AIME24 evals #4

merged 3 commits into from
Jan 24, 2025

Conversation

edbeeching
Copy link
Collaborator

@edbeeching edbeeching commented Jan 24, 2025

Evaluating models (internal)

For small models use --data_parallel=$NUM_GPUS, for large models shard with --tensor_parallel=$NUM_GPUS
Example for evaluating deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

NUM_GPUS=1
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
MODEL_ARGS="pretrained=$MODEL_ID,dtype=bfloat16,data_parallel=$NUM_GPUS,max_model_length=4096,gpu_memory_utilisation=0.8"
TASK=aime24 # or math
OUTPUT_DIR=evals/$MODEL

lighteval $MODEL_ARGS $TASK --use-chat-template --custom-tasks src/open_r1/eval/$TASK.py --output-dir $OUTPUT_DIR --system-prompt="Please reason step by step, and put your final answer within \boxed{}."

Copy link
Member

@lewtun lewtun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@lewtun lewtun merged commit 9c39897 into main Jan 24, 2025
0 of 2 checks passed
@lewtun lewtun deleted the ed-evals branch January 24, 2025 22:09
GitMonkey0 pushed a commit to GitMonkey0/open-r1 that referenced this pull request Feb 24, 2025
* adds evals

* up max model len

---------

Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants