Skip to content

How to calculate the Avg. Value of SoPR, SoWR in the paper? #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
OPilgrim opened this issue Apr 7, 2025 · 1 comment
Open

How to calculate the Avg. Value of SoPR, SoWR in the paper? #8

OPilgrim opened this issue Apr 7, 2025 · 1 comment

Comments

@OPilgrim
Copy link

OPilgrim commented Apr 7, 2025

I have already tried simple summation and averaging methods, as well as weighted averaging based on the data volume of each sub-dataset, but none of these methods can reproduce the 'Avg.' value in the paper. How exactly should this value be calculated? Also, is the ToolGen model related to In-domain not yet open-sourced?

@Reason-Wang
Copy link
Owner

Reason-Wang commented Apr 12, 2025

Hi, it is weighted average. The sample number for I1, I2, and I3 Instruction is 163, 106, 61 respectively and for I1 Tool, I1 Cat, I2 Cat is 153, 158, 124 respectively.

Also, is the ToolGen model related to In-domain not yet open-sourced?

Sorry that they are actually open-sourced, but I forgot to add them to the collection, they are now at https://huggingface.co/collections/reasonwang/toolgen-668a46a4959745ec8e9891f6, ending with G1, G2, and G3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants