Deepseek-r1测试效果不好（Deepseek-r1's testing results weren't great）

[lissettecarlr]

我简单的测试了下中文试题，发现效果非常差，大家有没有跑过呢

dataset	version	metric	mode	Deepseek-r1
ceval-high_school_mathematics	1dc5bf	accuracy	gen	66.67
ceval-high_school_physics	adf25f	accuracy	gen	57.89
ceval-high_school_chemistry	2ed27f	accuracy	gen	68.42
ceval-high_school_biology	8e2b9a	accuracy	gen	68.42
ceval-middle_school_mathematics	bee8d5	accuracy	gen	73.68
ceval-middle_school_biology	86817c	accuracy	gen	76.19
ceval-middle_school_physics	8accf6	accuracy	gen	94.74
ceval-middle_school_chemistry	167a15	accuracy	gen	90.00

dataset	version	metric	mode	Deepseek-r1
GaokaoBench_2010-2022_Math_II_MCQs	5b0b29	score	gen	62.39
GaokaoBench_2010-2022_Math_I_MCQs	5b0b29	score	gen	54.21

[carllisicau]

情况一样，用DeepSeek32B版本测评aime2024数据集，仅得3.33分

[lissettecarlr]

我专门选择的是选择题，然后看了输出结果，虽然是一大段话，但是给出的结果基本都是正确的。不知道open-compass是用什么方法来确定回答是否正确的。

[MaiziXiao]

可以尝试把 results 中的结果 remove然后重新 evaluate (i.e. 加入--dump-eval-details --mode eval)，在输出结果中会将 details 打印出来

[lissettecarlr]

感谢，看出来了，错误的都是：

        "1": {
            "pred": [
                "Z"
            ],
            "refr": [
                "A"
            ],
            "is_correct": false
        }

对比详细，发现是没有回答，估计还是deepseek-R1接口的网络原因：

    "1": {
        "origin_prompt": [
            {
                "role": "HUMAN",
                "prompt": "请你做一道数学选择题\n请你一步一步思考并将思考过程写在【解析】和<eoe>之间。你将从A，B，C，D中选出正确的答案，并写在【答案】和<eoa>之间。\n例如：【答案】: A <eoa>\n完整的题目回答的格式如下：\n【解析】 ... <eoe>\n【答案】 ... <eoa>\n请你严格按照上述格式作答。\n题目如下：2. (5 分) 已知复数 $z=\\frac{\\sqrt{3}+i}{(1-\\sqrt{3} i)^{2}}, \\bar{z}$ 是 $z$ 的共轭复数, 则 $z=\\bar{z}=(\\quad)$\nA. $\\frac{1}{4}$\nB. $\\frac{1}{2}$\nC. 1\nD. 2\n"
            }
        ],
        "prediction": "",
        "gold": [
            "A"
        ]
    }

[carllisicau]

我的错误是prediction里面有输出，但没有输出完整，没有推理出最终的答案
"29": { "origin_prompt": [ { "role": "HUMAN", "prompt": "\nFind the number of rectangles that can be formed inside a fixed regular dodecagon ($12$-gon) where each side of the rectangle lies on either a side or a diagonal of the dodecagon. The diagram below shows three of those rectangles. [asy] unitsize(0.6 inch); for(int i=0; i<360; i+=30) { dot(dir(i), 4+black); draw(dir(i)--dir(i+30)); } draw(dir(120)--dir(330)); filldraw(dir(210)--dir(240)--dir(30)--dir(60)--cycle, mediumgray, linewidth(1.5)); draw((0,0.366)--(0.366,0), linewidth(1.5)); [/asy]\n\nRemember to put your final answer within \\boxed{}.\n" } ], "prediction": "<think>\nOkay, so I have this problem where I need to find the number of rectangles that can be formed inside a regular dodecagon, which is a 12-sided polygon. Each side of the rectangle has to lie on either a side or a diagonal of the dodecagon. The Asymptote diagram shows three examples of such rectangles, so I know they can be of different sizes and orientations.\n\nFirst, I should recall that a regular dodecagon has all sides equal and all internal angles equal. Each internal angle in a regular dodecagon is 150 degrees because the formula for internal angles is \\((n-2) \\times 180^\\circ / n\\), which for \\(n=12\\) gives \\((12-2) \\times 180 / 12 = 150^\\circ\\).\n\nNow, the problem is about finding rectangles. So, I need to figure out how rectangles can be formed by connecting sides or diagonals of the dodecagon. Since a rectangle has four sides with opposite sides equal and all angles 90 degrees, the key is to find sets of four points (vertices of the dodecagon) that form such a shape.\n\nLet me think about how the sides and diagonals of the dodecagon can form rectangles. In a regular polygon, the sides are the edges connecting consecutive vertices, and diagonals are the lines connecting non-consecutive vertices. In a dodecagon, each vertex is connected to 11 others, so there are a lot of diagonals.\n\nBut for a rectangle, we need four vertices such that the sides are either sides or diagonals of the dodecagon. So, maybe the sides of the rectangle correspond to sides or diagonals of the dodecagon.\n\nI remember that in regular polygons, certain diagonals can form rectangles or other shapes. For example, in a regular octagon, you can form rectangles by connecting every other vertex.\n\nSimilarly, in a regular dodecagon, perhaps connecting every third vertex or some other interval can form rectangles. Let me think about how the sides and diagonals can form right angles.\n\nWait, in a regular polygon, the central angles between consecutive vertices are equal. For a dodecagon, each central angle is \\(360^\\circ / 12 = 30^\\circ\\). So, the angle between two adjacent vertices from the center is 30 degrees.\n\nTo form a rectangle, the sides must meet at 90-degree angles. So, the vectors corresponding to the sides of the rectangle must be perpendicular. Since the polygon is regular, the vectors can be represented in terms of the central angles.\n\nSo, if I can find two vectors that are perpendicular and lie on the sides or diagonals of the dodecagon, then I can form a rectangle by combining them.\n\nLet me denote the vertices of the dodecagon as \\(V_0, V_1, V_2, \\ldots, V_{11}\\) going around the polygon. Each vertex \\(V_k\\) can be represented in the complex plane as \\(e^{i \\theta_k}\\) where \\(\\theta_k = 30^\\circ \\times k\\).\n\nIf I can find four points \\(V_a, V_b, V_c, V_d\\) such that the vectors \\(V_b - V_a\\) and \\(V_d - V_a\\) are perpendicular, and similarly for other sides, then they form a rectangle.\n\nBut maybe there's a simpler way. Since the dodecagon is regular and symmetric, maybe I can count the number of rectangles based on the number of pairs of parallel sides.\n\nWait, rectangles have opposite sides equal and parallel. So, in the dodecagon, if I can find two pairs of parallel chords (sides or diagonals) that are perpendicular to each other, they can form a rectangle.\n\nSo, perhaps I should figure out how many pairs of parallel chords exist in the dodecagon and then see how many of these pairs are perpendicular.\n\nBut first, how many pairs of parallel sides or diagonals are there in a regular dodecagon?\n\nIn a regular polygon with \\(n\\) sides, the number of pairs of parallel sides is \\(n/2\\) if \\(n\\) is even. Wait, for a dodecagon, which has 12 sides, the number of pairs of parallel sides is 6, since each side has one parallel side opposite to it.\n\nBut in addition to sides, there are diagonals that can also be parallel. So, the number of pairs of parallel diagonals is more complicated.\n\nWait, in a regular polygon, the number of directions for parallel chords depends on the number of sides. For a dodecagon, each chord can be defined by the number of vertices it skips. For example, a side skips 0 vertices, a diagonal that skips 1 vertex is another type, skips 2, skips 3, etc., up to skipping 5 vertices (since beyond that, it's the same as skipping fewer in the other direction).\n\nSo, for a regular dodecagon, chords can skip \\(k = 0, 1, 2, 3, 4, 5\\) vertices. Each \\(k\\) gives a set of parallel chords.\n\nTherefore, each direction corresponds to a step size \\(k\\), and each step size \\(k\\) from 1 to 5 (since \\(k=0\\) is the sides themselves) gives a set of parallel diagonals. So, for each \\(k = 1\\) to \\(5\\), there are 12 diagonals each, but they are grouped into parallel sets.\n\nWait, actually, for each \\(k\\), the number of distinct directions is 6 because of symmetry. Hmm, maybe I need to think differently.\n\nWait, perhaps each step size \\(k\\) and \\(12 - k\\) gives the same direction but in opposite orientation. So, for \\(k = 1\\) and \\(k = 11\\), they are the same direction but opposite; similarly for \\(k=2\\) and \\(k=10\\), etc. So, for step sizes, we can consider \\(k = 1\\) to \\(6\\), but beyond \\(k=6\\), it's the same as smaller \\(k\\) in the opposite direction.\n\nBut in our case, since we have 12 sides, each step size \\(k\\) from 1 to 5 gives a unique direction, and \\(k=6\\) is the diameter, which is its own opposite.\n\nWait, actually, in a regular dodecagon, the diameters (which connect opposite vertices) are the only chords that are their own opposites. So, for each \\(k\\) from 1 to 5, there are two directions (clockwise and counterclockwise), but in terms of parallelism, they are the same direction.\n\nWait, maybe it's better to think in terms of slopes. Each chord with step size \\(k\\) will have a certain slope, and chords with step size \\(k\\) and \\(12 - k\\) will have slopes that are negatives of each other, hence not parallel. So, actually, for each \\(k\\) from 1 to 5, there is a unique set of parallel chords.\n\nTherefore, in total, there are 6 different directions for chords: step sizes \\(k = 0\\) (sides), \\(1\\), \\(2\\), \\(3\\), \\(4\\), \\(5\\), and \\(6\\) (diameters). But step size \\(6\\) is just the diameter, which is a single direction.\n\nWait, no, step size \\(k\\) and \\(12 - k\\) are different directions because they go in opposite directions around the polygon. So, for each \\(k = 1\\) to \\(5\\), we have two directions, but they are not parallel. So, actually, each step size \\(k\\) corresponds to a unique direction. So, for step sizes \\(k = 0\\) (sides), \\(1\\), \\(2\\), \\(3\\), \\(4\\), \\(5\\), and \\(6\\), each gives a unique direction.\n\nWait, but for step size \\(k=6\\), it's the diameter, so it's only one direction because going \\(6\\) steps in either direction from a vertex gets you to the same opposite vertex.\n\nSo, in total, there are 7 different directions for chords: sides (k=0), diameters (k=6), and for \\(k=1\\) to \\(5\\), each gives two directions, but they are not parallel. Wait, no, actually, for each \\(k\\), the chords are parallel if they have the same step size. So, chords with the same \\(k\\) are parallel, regardless of starting point.\n\nTherefore, for each \\(k\\) from 0 to 6, the chords with step size \\(k\\) are all parallel to each other. So, in a dodecagon, how many unique directions do we have? For each \\(k = 0, 1, 2, 3, 4, 5, 6\\), we have a unique direction.\n\nBut wait, for \\(k = 1\\) and \\(k = 11\\), are they the same? Because stepping 1 forward or 11 backward is the same direction.\n\nWait, actually, in a regular polygon, stepping \\(k\\) forward is equivalent to stepping \\(n - k\\) backward. So, for direction purposes, stepping \\(k\\) or \\(n - k\\) gives the same direction. So, in a 12-gon, stepping 1 and stepping 11 give the same direction but in opposite orientations.\n\nBut when considering parallelism,", "gold": "315" }