Skip to content

Latest commit

 

History

History
181 lines (127 loc) · 10.5 KB

README.md

File metadata and controls

181 lines (127 loc) · 10.5 KB
MM-Eureka logo

MM-EUREKA

MM-EUREKA: EXPLORING THE FRONTIERS OF MULTIMODAL REASONING WITH RULE-BASED REINFORCEMENT LEARNING

🎯Overview

We present MM-Eureka-Qwen-7B and MM-Eureka-Qwen-32B, both are powerful multimodal reasoning model that successfully extends large-scale rule-based reinforcement learning (RL) to multimodal reasoning. Compared to the previous version of MM-EUREKA based on InternVL, we have made improvements in model architecture, algorithms, and data. For instance, MM-EUREKA-7B achieves 66.1 on MMK12 evaluation sets, only 0.2 points below Intern2.5-VL-78B. On MathVista, it reaches 73.0, even surpassing InternVL2.5-VL-78B. MM-EUREKA-32B demonstrates stronger performance, scoring 72.3 on MMK12 evaluation sets, which exceeds both Qwen2.5-VL-72B's 70.3 and closed-source models like Gemini2-Flash, ranking second only to o1's 73.9. On commonly used multimodal mathematical reasoning benchmarks, MM-EUREKA-32B achieves 73.4 on WeMath, outperforming all open-source models and most closed-source models including Claude3.7 Sonnet. On MathVista, it reaches 74.8, surpassing all open-source and closed-source models. Both variants demonstrate significant improvements in multidisciplinary K12 and mathematical reasoning performance, outperforming most open-source models of similar sizes.

Core Improvements:

  1. We further iterate the codebase to support algorithms including Online Filter, ADORA, and DAPO.
  2. We open-source self-collected MMK12, which has 15k diverse and high-quality samples and 2k MCQs for Math, Physics, Chemistry, Biology for evaluation.
  3. We train the MM-Eureka-Qwen-7B and MM-Eureka-Qwen-32B, which are the almost top performancer in multimodal reasoning within similar size open-source models. Especially for Multidisciplinary K12 tasks.

🔥We open-source our complete pipeline to foster further research in this area. We release all our codes, models, data, etc. at MM-EUREKA-Qwen.

🗞️ News

🚀 Features

This repository is built upon OpenRLHF, introducing several key enhancements:

  • Multimodal RFT Support: Extends OpenRLHF to incorporate vision-language models (VLMs), currently supporting InternVL, enabling multimodal reasoning capabilities.
    • Currently support RLOO, REINFORCE++, GRPO training using Ray.
    • vLLM integration and distributed training.
    • Support hybrid engine (--colocate_all_models, --vllm_enable_sleep).
  • Better Rule-based Reward support: Better training visualization for Rule-based Rewards (i.g. Format Reward, Accuracy Reward, Repetition Penalty)
  • Enhanced Online Filtering: Filtering out experiences based on Accuracy Reward during training as in PRIME
    • Use --enable_accuracy_filter, --freezing_filter_steps, --accuracy_lower_bound, --accuracy_upper_bound to control the behavior of online accuracy filter.
  • ADORA: Enable Adaptive Online Rollout Adjustment by using --use_adora and --adora_lamda as in ADORA.
  • DAPO: You can use --use_dapo to enable DAPO loss during training as in DAPO.

🤖 Models

Based on the key factors identified by https://github.com/ModalMinds/MM-EUREKA for achieving stable training, we enhanced the model, dataset, and algorithmic modules. Specifically, we maintained the strategy of omitting the KL divergence term and applying data filtering, while implementing the following critical modifications:

  • The base model was upgraded from InternVL2.5-8B-Instruct to the more powerful Qwen2.5-VL-7B-Instruct.
  • The Vision Transformer (ViT) module was frozen during training.
  • The underlying RL algorithm was replaced with GRPO, instead of the previously used RLOO.
  • The data filtering strategy was transitioned from an offline approach to an online approach.
  • Additional data from the K12 dataset was collected, expanding the total dataset size to 15,000 samples.

Finally, MM-EUREKA-Qwen achieves 73.0 on MathVista, surpassing the original Qwen2.5-VL-7B-Instruct by 4.8%.

Base Model MathVista MathVerse MathVision OlympidBench K12
Qwen2.5-VL-7B-Instruct - 68.2 47.9 25.4 15.3 36.0
InternVL2.5-VL-8B-Instruct - 64.4 39.5 19.7 8.0 24.8
InternVL2.5-VL-38B-Instruct - 71.9 49.4 31.8 29.3 37.2
MM-EUREKA-InternVL-8B InternVL2.5-VL-7B-Instruct 67.1 40.4 22.2 8.6 27.0
MM-EUREKA-Qwen-7B Qwen2.5-VL-7B-Instruct 73.0 (+4.8) 50.3 (+2.4) 26.9 (+1.5) 25.3（+10.0） 48.6 (+12.6)

🏁 Getting Started

📦 Installation

git clone https://github.com/ModalMinds/MM-EUREKA.git
git checkout qwen
cd MM-EUREKA
pip install -e .[vllm]
pip install flash_attn --no-build-isolation

📂 Data Preparation

You can download our training data from MM-Eureka-Dataset

Once downloaded, refer to the section below for additional data formation.

Custom dataset

For custom dataset, format your data in to a JSONL file, where each entry is a dictionary organized in the following format.

{
  "id": "0",
  "message": "[{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": \"file:///path/to/your/image.jpg\"}, {\"type\": \"text\", \"text\": \"How many cats in the image?\"}]}]",
  "answer": "gt that could be parsed and verified by math_verify"
}

🌐 Start Training

Before starting your own training, ensure that the paths in the provided training scripts are correctly set and that environment variables like $MASTER_ADDR and $NODE_RANK are properly configured.

start MM-Eureka-Qwen-7B training

  • for single node

    sh examples/scripts/train_mm_eureka_qwen_7b_single_node.sh

  • for multiple node

    sh examples/scripts/train_mm_eureka_qwen_7b_multi_node.sh

⭐ Starchart

Star History Chart

🤝 Contribution

MM-Eureka is stil under active development, if you want to contribute, please feel free to make a pull request or create an issue.

Please refer to CONTRIBUTING.md before you dive in！

📬 Contact

If you have any questions or would like to engage with our community, feel free to scan the QR code below to join our WeChat group.

MM-Eureka logo

🎓 Acknowledgements

We acknowledge the outstanding open-source contributions from OpenRLHF, LMM-R1 and vLLM. We also extend our gratitude to DeepSeek-R1, InternVL and QwenVL for their open-source techniques and base models, which have enabled us to further our exploration.

📜 Citation

@article{meng2025mm,
  title={MM-Eureka: Exploring Visual Aha Moment with Rule-based Large-scale Reinforcement Learning},
  author={Meng, Fanqing and Du, Lingxiao and Liu, Zongkai and Zhou, Zhixiang and Lu, Quanfeng and Fu, Daocheng and Shi, Botian and Wang, Wenhai and He, Junjun and Zhang, Kaipeng and others},
  journal={arXiv preprint arXiv:2503.07365},
  year={2025}
}