Skip to content

Commit 21cbddd

Browse files
MistEOMistEO
authored
feat: order by and index of pipeline (#145)
1 parent 4a61cf8 commit 21cbddd

23 files changed

+607
-56
lines changed

‎docs/en_us/3.1-PipelineProtocol.md

+54
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,15 @@ This task property requires additional fields:
224224
Template matching threshold. Optional, default is 0.7.
225225
If it's an array, its length should match the length of the `template` array.
226226
227+
- `order_by`: *string*
228+
How the results are sorted. Optional, default is `Horizontal`
229+
Possible values: `Horizontal` | `Vertical` | `Score` | `Area` | `Random`
230+
You can use it with the `index` field.
231+
232+
- `index`: *int*
233+
Index to hit. Optional, default is `0`.
234+
If there are N results in total, the value range of `index` is [-N, N - 1], where negative numbers are converted to N - index using Python-like rules. If it exceeds the range, it is considered that there is no result in the current identification.
235+
227236
- `method`: *int*
228237
Template matching algorithm, equivalent to cv::TemplateMatchModes. Optional, default is 5.
229238
Only supports 1, 3, and 5, with higher values providing greater accuracy but also taking more time.
@@ -250,6 +259,15 @@ This task property requires additional fields:
250259
- `count`: *int*
251260
The number of required matching feature points (threshold), default is 4.
252261

262+
- `order_by`: *string*
263+
How the results are sorted. Optional, default is `Horizontal`
264+
Possible values: `Horizontal` | `Vertical` | `Score` | `Area` | `Random`
265+
You can use it with the `index` field.
266+
267+
- `index`: *int*
268+
Index to hit. Optional, default is `0`.
269+
If there are N results in total, the value range of `index` is [-N, N - 1], where negative numbers are converted to N - index using Python-like rules. If it exceeds the range, it is considered that there is no result in the current identification.
270+
253271
- `green_mask`: *bool*
254272
Whether to apply a green mask. Optional, default is false.
255273
If set to true, you can paint the unwanted parts in the image green with RGB: (0, 255, 0), and those green parts won't be matched.
@@ -298,6 +316,15 @@ This task property requires additional fields:
298316
- `count`: *int*
299317
The threshold for the number of matching points required. Optional, default is 1.
300318
319+
- `order_by`: *string*
320+
How the results are sorted. Optional, default is `Horizontal`
321+
Possible values: `Horizontal` | `Vertical` | `Score` | `Area` | `Random`
322+
You can use it with the `index` field.
323+
324+
- `index`: *int*
325+
Index to hit. Optional, default is `0`.
326+
If there are N results in total, the value range of `index` is [-N, N - 1], where negative numbers are converted to N - index using Python-like rules. If it exceeds the range, it is considered that there is no result in the current identification.
327+
301328
- `connected`: *bool*
302329
Whether to count only connected points. Optional, default is false.
303330
If set to true, after applying color filtering, it will only count the maximum connected block of pixels.
@@ -318,6 +345,15 @@ This task property requires additional fields:
318345
- `replace`: *array<string, 2>* | *list<array<string, 2>>*
319346
Some text recognition results may not be accurate, so replacements are performed. Optional.
320347

348+
- `order_by`: *string*
349+
How the results are sorted. Optional, default is `Horizontal`
350+
Possible values: `Horizontal` | `Vertical` | `Area` | `Length` | `Random`
351+
You can use it with the `index` field.
352+
353+
- `index`: *int*
354+
Index to hit. Optional, default is `0`.
355+
If there are N results in total, the value range of `index` is [-N, N - 1], where negative numbers are converted to N - index using Python-like rules. If it exceeds the range, it is considered that there is no result in the current identification.
356+
321357
- `only_rec`: *bool*
322358
Whether to recognize only (without detection, requires precise `roi`). Optional, default is false.
323359

@@ -350,6 +386,15 @@ This task property requires additional fields:
350386
- `expected`: *int* | *list<int, >*
351387
The expected category index.
352388

389+
- `order_by`: *string*
390+
How the results are sorted. Optional, default is `Horizontal`
391+
Possible values: `Horizontal` | `Vertical` | `Area` | `Random`
392+
You can use it with the `index` field.
393+
394+
- `index`: *int*
395+
Index to hit. Optional, default is `0`.
396+
If there are N results in total, the value range of `index` is [-N, N - 1], where negative numbers are converted to N - index using Python-like rules. If it exceeds the range, it is considered that there is no result in the current identification.
397+
353398
For example, if you want to recognize whether a cat or a mouse appears in a **fixed position** in the image, and you've trained a model that supports this three-category classification, and you want to click when it recognizes a cat or a mouse but not when it recognizes a dog, the relevant fields would be:
354399
355400
```jsonc
@@ -391,6 +436,15 @@ This task property requires additional fields:
391436
Model confidence threshold. Optional, default is 0.3.
392437
If it's an array, its length should match the length of the `expected` array.
393438

439+
- `order_by`: *string*
440+
How the results are sorted. Optional, default is `Horizontal`
441+
Possible values: `Horizontal` | `Vertical` | `Area` | `Random`
442+
You can use it with the `index` field.
443+
444+
- `index`: *int*
445+
Index to hit. Optional, default is `0`.
446+
If there are N results in total, the value range of `index` is [-N, N - 1], where negative numbers are converted to N - index using Python-like rules. If it exceeds the range, it is considered that there is no result in the current identification.
447+
394448
For example, if you want to detect cats, dogs, and mice in an image and only click when a cat or a mouse is detected but not when a dog is detected, the relevant fields would be:
395449

396450
```jsonc

‎docs/zh_cn/3.1-任务流水线协议.md

+54
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,15 @@ graph LR;
229229
模板匹配阈值。可选，默认 0.7 。
230230
若为数组，长度需和 `template` 数组长度相同。
231231
232+
- `order_by`: *string*
233+
结果排序方式。可选，默认 `Horizontal`
234+
可选的值：`Horizontal` | `Vertical` | `Score` | `Area` | `Random`
235+
可结合 `index` 字段使用。
236+
237+
- `index`: *int*
238+
命中第几个结果。可选，默认 0。
239+
假设共有 N 个结果，则 `index` 的取值范围为 [-N, N - 1]，其中负数使用类 Python 的规则转换为 N - index。若超出范围，则视为当前识别无结果。
240+
232241
- `method`: *int*
233242
模板匹配算法，即 cv::TemplateMatchModes。可选，默认 5 。
234243
仅支持 1、3、5，可简单理解为越大的越精确，但也会更慢。
@@ -255,6 +264,15 @@ graph LR;
255264
- `count`: *int*
256265
匹配的特征点的数量要求（阈值），默认 4.
257266
267+
- `order_by`: *string*
268+
结果排序方式。可选，默认 `Horizontal`
269+
可选的值：`Horizontal` | `Vertical` | `Score` | `Area` | `Random`
270+
可结合 `index` 字段使用。
271+
272+
- `index`: *int*
273+
命中第几个结果。可选，默认 0。
274+
假设共有 N 个结果，则 `index` 的取值范围为 [-N, N - 1]，其中负数使用类 Python 的规则转换为 N - index。若超出范围，则视为当前识别无结果。
275+
258276
- `green_mask`: *bool*
259277
是否进行绿色掩码。可选，默认 false。
260278
若为 true，可以将图片中不希望匹配的部分涂绿 RGB: (0, 255, 0)，则不对绿色部分进行匹配。
@@ -303,6 +321,15 @@ graph LR;
303321
- `count`: *int*
304322
符合的点的数量要求（阈值）。可选，默认 1。
305323
324+
- `order_by`: *string*
325+
结果排序方式。可选，默认 `Horizontal`
326+
可选的值：`Horizontal` | `Vertical` | `Score` | `Area` | `Random`
327+
可结合 `index` 字段使用。
328+
329+
- `index`: *int*
330+
命中第几个结果。可选，默认 0。
331+
假设共有 N 个结果，则 `index` 的取值范围为 [-N, N - 1]，其中负数使用类 Python 的规则转换为 N - index。若超出范围，则视为当前识别无结果。
332+
306333
- `connected`: *bool*
307334
是否是相连的点才会被计数。可选，默认否。
308335
若为是，在完成颜色过滤后，则只会计数像素点 **全部相连** 的最大块。
@@ -323,6 +350,15 @@ graph LR;
323350
- `replace`: *array<string, 2>* | *list<array<string, 2>>*
324351
部分文字识别结果不准确，进行替换。可选。
325352
353+
- `order_by`: *string*
354+
结果排序方式。可选，默认 `Horizontal`
355+
可选的值：`Horizontal` | `Vertical` | `Area` | `Length` | `Random`
356+
可结合 `index` 字段使用。
357+
358+
- `index`: *int*
359+
命中第几个结果。可选，默认 0。
360+
假设共有 N 个结果，则 `index` 的取值范围为 [-N, N - 1]，其中负数使用类 Python 的规则转换为 N - index。若超出范围，则视为当前识别无结果。
361+
326362
- `only_rec`: *bool*
327363
是否仅识别（不进行检测，需要精确设置 `roi`）。可选，默认 false。
328364
@@ -354,6 +390,15 @@ graph LR;
354390
- `expected`: *int* | *list<int, >*
355391
期望的分类下标。
356392
393+
- `order_by`: *string*
394+
结果排序方式。可选，默认 `Horizontal`
395+
可选的值：`Horizontal` | `Vertical` | `Score` | `Area` | `Random`
396+
可结合 `index` 字段使用。
397+
398+
- `index`: *int*
399+
命中第几个结果。可选，默认 0。
400+
假设共有 N 个结果，则 `index` 的取值范围为 [-N, N - 1]，其中负数使用类 Python 的规则转换为 N - index。若超出范围，则视为当前识别无结果。
401+
357402
举例：例如画面中 **固定位置** 可能出现 猫、狗、老鼠，我们训练了支持该三分类的模型。
358403
希望识别到 猫 或 老鼠 才点击，而识别到 狗 不点击，则相关字段为
359404
@@ -396,6 +441,15 @@ graph LR;
396441
模型置信度阈值。可选，默认 0.3 。
397442
若为数组，长度需和 `expected` 数组长度相同。
398443
444+
- `order_by`: *string*
445+
结果排序方式。可选，默认 `Horizontal`
446+
可选的值：`Horizontal` | `Vertical` | `Score` | `Area` | `Random`
447+
可结合 `index` 字段使用。
448+
449+
- `index`: *int*
450+
命中第几个结果。可选，默认 0。
451+
假设共有 N 个结果，则 `index` 的取值范围为 [-N, N - 1]，其中负数使用类 Python 的规则转换为 N - index。若超出范围，则视为当前识别无结果。
452+
399453
举例：例如画面中可能出现 猫、狗、老鼠，我们训练了支持该三分类的检测模型。
400454
希望检测到 猫 或 老鼠 才点击，而识别到 狗 不点击，则相关字段为
401455

‎source/MaaFramework/Resource/PipelineResMgr.cpp

+106
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,19 @@ bool PipelineResMgr::parse_template_matcher_param(const json::value& input, MAA_
445445
return false;
446446
}
447447

448+
if (!parse_order_of_result(input, output.order_by, output.result_index, default_value.order_by,
449+
default_value.result_index,
450+
{
451+
MAA_VISION_NS::ResultOrderBy::Horizontal,
452+
MAA_VISION_NS::ResultOrderBy::Vertical,
453+
MAA_VISION_NS::ResultOrderBy::Score,
454+
MAA_VISION_NS::ResultOrderBy::Area,
455+
MAA_VISION_NS::ResultOrderBy::Random,
456+
})) {
457+
LogError << "failed to parse_order_of_result" << VAR(input);
458+
return false;
459+
}
460+
448461
if (!get_and_check_value_or_array(input, "template", output.template_paths, default_value.template_paths)) {
449462
LogError << "failed to get_and_check_value_or_array templates" << VAR(input);
450463
return false;
@@ -492,6 +505,19 @@ bool PipelineResMgr::parse_feature_matcher_param(const json::value& input, MAA_V
492505
return false;
493506
}
494507

508+
if (!parse_order_of_result(input, output.order_by, output.result_index, default_value.order_by,
509+
default_value.result_index,
510+
{
511+
MAA_VISION_NS::ResultOrderBy::Horizontal,
512+
MAA_VISION_NS::ResultOrderBy::Vertical,
513+
MAA_VISION_NS::ResultOrderBy::Score,
514+
MAA_VISION_NS::ResultOrderBy::Area,
515+
MAA_VISION_NS::ResultOrderBy::Random,
516+
})) {
517+
LogError << "failed to parse_order_of_result" << VAR(input);
518+
return false;
519+
}
520+
495521
if (!get_and_check_value(input, "template", output.template_path, default_value.template_path)) {
496522
LogError << "failed to get_and_check_value template_path" << VAR(input);
497523
return false;
@@ -554,6 +580,19 @@ bool PipelineResMgr::parse_ocrer_param(const json::value& input, MAA_VISION_NS::
554580
return false;
555581
}
556582

583+
if (!parse_order_of_result(input, output.order_by, output.result_index, default_value.order_by,
584+
default_value.result_index,
585+
{
586+
MAA_VISION_NS::ResultOrderBy::Horizontal,
587+
MAA_VISION_NS::ResultOrderBy::Vertical,
588+
MAA_VISION_NS::ResultOrderBy::Area,
589+
MAA_VISION_NS::ResultOrderBy::Length,
590+
MAA_VISION_NS::ResultOrderBy::Random,
591+
})) {
592+
LogError << "failed to parse_order_of_result" << VAR(input);
593+
return false;
594+
}
595+
557596
if (!get_and_check_value(input, "model", output.model, default_value.model)) {
558597
LogError << "failed to get_and_check_value model" << VAR(input);
559598
return false;
@@ -635,6 +674,19 @@ bool PipelineResMgr::parse_nn_classifier_param(const json::value& input,
635674
return false;
636675
}
637676

677+
if (!parse_order_of_result(input, output.order_by, output.result_index, default_value.order_by,
678+
default_value.result_index,
679+
{
680+
MAA_VISION_NS::ResultOrderBy::Horizontal,
681+
MAA_VISION_NS::ResultOrderBy::Vertical,
682+
MAA_VISION_NS::ResultOrderBy::Score,
683+
MAA_VISION_NS::ResultOrderBy::Area,
684+
MAA_VISION_NS::ResultOrderBy::Random,
685+
})) {
686+
LogError << "failed to parse_order_of_result" << VAR(input);
687+
return false;
688+
}
689+
638690
if (!get_and_check_value(input, "cls_size", output.cls_size, default_value.cls_size)) {
639691
LogError << "failed to get_and_check_value cls_size" << VAR(input);
640692
return false;
@@ -671,6 +723,19 @@ bool PipelineResMgr::parse_nn_detector_param(const json::value& input,
671723
return false;
672724
}
673725

726+
if (!parse_order_of_result(input, output.order_by, output.result_index, default_value.order_by,
727+
default_value.result_index,
728+
{
729+
MAA_VISION_NS::ResultOrderBy::Horizontal,
730+
MAA_VISION_NS::ResultOrderBy::Vertical,
731+
MAA_VISION_NS::ResultOrderBy::Score,
732+
MAA_VISION_NS::ResultOrderBy::Area,
733+
MAA_VISION_NS::ResultOrderBy::Random,
734+
})) {
735+
LogError << "failed to parse_order_of_result" << VAR(input);
736+
return false;
737+
}
738+
674739
if (!get_and_check_value(input, "cls_size", output.cls_size, default_value.cls_size)) {
675740
LogError << "failed to get_and_check_value cls_size" << VAR(input);
676741
return false;
@@ -871,6 +936,47 @@ bool PipelineResMgr::parse_roi(const json::value& input, std::vector<cv::Rect>&
871936
return !output.empty();
872937
}
873938

939+
bool PipelineResMgr::parse_order_of_result(const json::value& input, MAA_VISION_NS::ResultOrderBy& output,
940+
int& output_index, const MAA_VISION_NS::ResultOrderBy& default_value,
941+
int default_index,
942+
const std::unordered_set<MAA_VISION_NS::ResultOrderBy>& valid_values)
943+
{
944+
static const std::string kDefaultOrderFlag = "Default";
945+
std::string order;
946+
if (!get_and_check_value(input, "order", order, kDefaultOrderFlag)) {
947+
LogError << "failed to get_and_check_value order" << VAR(input);
948+
return false;
949+
}
950+
951+
const std::unordered_map<std::string, MAA_VISION_NS::ResultOrderBy> kOrderMap = {
952+
{ kDefaultOrderFlag, default_value },
953+
{ "Horizontal", MAA_VISION_NS::ResultOrderBy::Horizontal },
954+
{ "Vertical", MAA_VISION_NS::ResultOrderBy::Vertical },
955+
{ "Score", MAA_VISION_NS::ResultOrderBy::Score },
956+
{ "Area", MAA_VISION_NS::ResultOrderBy::Area },
957+
{ "Length", MAA_VISION_NS::ResultOrderBy::Length },
958+
{ "Random", MAA_VISION_NS::ResultOrderBy::Random },
959+
{ "Expected", MAA_VISION_NS::ResultOrderBy::Expected },
960+
};
961+
auto order_iter = kOrderMap.find(order);
962+
if (order_iter == kOrderMap.end()) {
963+
LogError << "order not found" << VAR(order);
964+
return false;
965+
}
966+
if (!valid_values.contains(order_iter->second)) {
967+
LogError << "current recognition not support order" << VAR(order);
968+
return false;
969+
}
970+
output = order_iter->second;
971+
972+
if (!get_and_check_value(input, "index", output_index, default_index)) {
973+
LogError << "failed to get_and_check_value index" << VAR(input);
974+
return false;
975+
}
976+
977+
return true;
978+
}
979+
874980
bool PipelineResMgr::parse_action(const json::value& input, Action::Type& out_type, Action::Param& out_param,
875981
const Action::Type& default_type, const Action::Param& default_param)
876982
{

‎source/MaaFramework/Resource/PipelineResMgr.h

+5-1
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#include <filesystem>
44
#include <set>
55
#include <unordered_map>
6+
#include <unordered_set>
67

78
#include <meojson/json.hpp>
89

@@ -44,7 +45,7 @@ class PipelineResMgr : public NonCopyable
4445
static bool parse_ocrer_param(const json::value& input, MAA_VISION_NS::OCRerParam& output,
4546
const MAA_VISION_NS::OCRerParam& default_value);
4647
static bool parse_custom_recognition_param(const json::value& input, MAA_VISION_NS::CustomRecognizerParam& output,
47-
const MAA_VISION_NS::CustomRecognizerParam& default_value);
48+
const MAA_VISION_NS::CustomRecognizerParam& default_value);
4849
static bool parse_nn_classifier_param(const json::value& input, MAA_VISION_NS::NeuralNetworkClassifierParam& output,
4950
const MAA_VISION_NS::NeuralNetworkClassifierParam& default_value);
5051
static bool parse_nn_detector_param(const json::value& input, MAA_VISION_NS::NeuralNetworkDetectorParam& output,
@@ -54,6 +55,9 @@ class PipelineResMgr : public NonCopyable
5455

5556
static bool parse_roi(const json::value& input, std::vector<cv::Rect>& output,
5657
const std::vector<cv::Rect>& default_value);
58+
static bool parse_order_of_result(const json::value& input, MAA_VISION_NS::ResultOrderBy& output, int& output_index,
59+
const MAA_VISION_NS::ResultOrderBy& default_value, int default_index,
60+
const std::unordered_set<MAA_VISION_NS::ResultOrderBy>& valid_values);
5761

5862
static bool parse_action(const json::value& input, Action::Type& out_type, Action::Param& out_param,
5963
const Action::Type& default_type, const Action::Param& default_param);

‎source/MaaFramework/Task/Actuator.cpp

+11-1
Original file line numberDiff line numberDiff line change
@@ -131,10 +131,20 @@ void Actuator::wait_freezes(const MAA_RES_NS::WaitFreezesParam& param, const cv:
131131
});
132132

133133
cv::Mat pre_image = controller()->screencap();
134+
if (need_to_stop()) {
135+
LogInfo << "Task interrupted";
136+
return;
137+
}
138+
134139
auto pre_time = std::chrono::steady_clock::now();
135140

136-
while (!need_to_stop()) {
141+
while (true) {
137142
cv::Mat cur_image = controller()->screencap();
143+
if (need_to_stop()) {
144+
LogInfo << "Task interrupted";
145+
return;
146+
}
147+
138148
auto ret = comp.analyze(pre_image, cur_image);
139149
if (ret.empty()) {
140150
pre_image = cur_image;

0 commit comments

Comments
 (0)