Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-11751][Manager] Modify doDeleteCheck method verify #11758

Merged
merged 9 commits into from
Feb 19, 2025

Conversation

Jzow
Copy link
Contributor

@Jzow Jzow commented Feb 13, 2025

Fixes #11751

Motivation

Resolve the issue of not being able to delete the data stream group under the data synchronization menu properly

Modifications

Note

2. Added Inlong Group Mode validation in the doDeleteCheck method

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @Jzow, thank you for submitting a PR to InLong 💖 We will respond as soon as possible ⏳
This seems to be your first PR 🌠 Please be sure to follow our Contribution Guidelines.
If you have any questions in the meantime, you can also ask us on the InLong Discussions 🔍

Jzow and others added 2 commits February 14, 2025 10:22
Apply suggestions from code review

Co-authored-by: fuweng11 <76141879+fuweng11@users.noreply.github.com>
@Jzow Jzow requested a review from fuweng11 February 14, 2025 02:46
@dockerzhang
Copy link
Contributor

Run 'mvn spotless:apply' to fix these violations.

@Jzow
Copy link
Contributor Author

Jzow commented Feb 14, 2025

Run 'mvn spotless:apply' to fix these violations.

Thanks, CI run normal.

@Jzow Jzow requested a review from fuweng11 February 17, 2025 03:03
fuweng11
fuweng11 previously approved these changes Feb 18, 2025
vernedeng
vernedeng previously approved these changes Feb 18, 2025
…/manager/service/group/InlongGroupServiceImpl.java
@dockerzhang dockerzhang dismissed stale reviews from vernedeng and fuweng11 via 5822f21 February 18, 2025 11:12
Copy link
Contributor

@vernedeng vernedeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please submit the test result

@Jzow
Copy link
Contributor Author

Jzow commented Feb 18, 2025

local testing, it has been verified.

manager-all.log:

[ ] 2025-02-19 00:22:23.021 - INFO [http-nio-8083-exec-7] .g.UpdateGroupCompleteListener:68 - begin to execute UpdateGroupCompleteListener for groupId=test_off_group, operateType=DELETE 
[ ] 2025-02-19 00:22:23.033 -DEBUG [http-nio-8083-exec-7] .m.s.s.InlongStreamServiceImpl:481 - begin to update inlong stream without check, request=InlongStreamRequest(id=19, inlongGroupId=test_off_group, inlongStreamId=test_off_group, name=null, description=, mqResource=test_off_group, dataType=null, dataEncoding=UTF-8, dataSeparator=124, dataEscapeChar=null, syncSend=0, dailyRecords=10, dailyStorage=10, peakRecords=1000, maxLength=10240, storagePeriod=1, extParams={"ignoreParseError":null,"kvSeparator":null,"lineSeparator":null,"useExtendedFields":false,"predefinedFields":null,"sinkMultipleEnable":false,"extendedFieldSize":0}, fieldList=null, extList=null, version=1, ignoreParseError=null, useExtendedFields=false, extendedFieldSize=0, sinkMultipleEnable=false, syncField=false, wrapType=INLONG_MSG_V0) 
[ ] 2025-02-19 00:22:23.036 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: select id, inlong_group_id, inlong_stream_id, name, description, mq_resource, data_type, wrap_type, data_encoding, data_separator, data_escape_char, sync_send, daily_records, daily_storage, peak_records, max_length, storage_period, ext_params, status, previous_status, is_deleted, creator, modifier, create_time, modify_time, version from inlong_stream WHERE is_deleted = 0 and inlong_group_id = ? and inlong_stream_id = ? 
[ ] 2025-02-19 00:22:23.038 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String), test_off_group(String) 
[ ] 2025-02-19 00:22:23.038 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==      Total: 1 
[ ] 2025-02-19 00:22:23.042 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: update inlong_stream SET inlong_group_id = ?, inlong_stream_id = ?, description = ?, mq_resource = ?, wrap_type = ?, data_encoding = ?, data_separator = ?, sync_send = ?, daily_records= ?, daily_storage= ?, peak_records= ?, max_length= ?, storage_period = ?, ext_params = ?, status = ?, previous_status = ?, is_deleted = ?, creator = ?, modifier = ?, version = ? + 1 where inlong_group_id = ? and inlong_stream_id = ? and is_deleted = 0 and version = ? 
[ ] 2025-02-19 00:22:23.044 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String), test_off_group(String), (String), test_off_group(String), INLONG_MSG_V0(String), UTF-8(String), 124(String), 0(Integer), 10(Integer), 10(Integer), 1000(Integer), 10240(Integer), 1(Integer), {"ignoreParseError":null,"kvSeparator":null,"lineSeparator":null,"useExtendedFields":false,"predefinedFields":null,"sinkMultipleEnable":false,"extendedFieldSize":0}(String), 100(Integer), 100(Integer), 0(Integer), admin(String), admin(String), 1(Integer), test_off_group(String), test_off_group(String), 1(Integer) 
[ ] 2025-02-19 00:22:23.048 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==    Updates: 1 
[ ] 2025-02-19 00:22:23.048 -DEBUG [http-nio-8083-exec-7] .m.s.s.InlongStreamServiceImpl:907 - begin to update inlong stream field, groupId=test_off_group, streamId=test_off_group, field=null 
[ ] 2025-02-19 00:22:23.049 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: delete from inlong_stream_field where inlong_group_id = ? and inlong_stream_id = ? and is_deleted = 0 
[ ] 2025-02-19 00:22:23.052 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String), test_off_group(String) 
[ ] 2025-02-19 00:22:23.054 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==    Updates: 0 
[ ] 2025-02-19 00:22:23.055 - INFO [http-nio-8083-exec-7] .m.s.s.InlongStreamServiceImpl:912 - success to update inlong stream field for groupId=test_off_group 
[ ] 2025-02-19 00:22:23.056 - INFO [http-nio-8083-exec-7] .m.s.s.InlongStreamServiceImpl:937 - begin to save or update inlong stream ext info, groupId=test_off_group, streamId=test_off_group, ext=null 
[ ] 2025-02-19 00:22:23.056 - INFO [http-nio-8083-exec-7] .m.s.s.InlongStreamServiceImpl:519 - success to update inlong stream without check for groupId=test_off_group streamId=test_off_group 
[ ] 2025-02-19 00:22:23.058 - INFO [http-nio-8083-exec-7] i.m.s.g.InlongGroupServiceImpl:668 - begin to delete inlong group for groupId=test_off_group by user=admin 
[ ] 2025-02-19 00:22:23.059 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: select id, inlong_group_id, name, description, tenant, mq_type, mq_resource, daily_records, daily_storage, peak_records, max_length, enable_zookeeper, enable_create_resource, inlong_group_mode, data_report_type, inlong_cluster_tag, ext_params, in_charges, followers, status, previous_status, is_deleted, creator, modifier, create_time, modify_time, version from inlong_group WHERE inlong_group_id = ? and is_deleted = 0 and tenant = ? 
[ ] 2025-02-19 00:22:23.062 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String), public(String) 
[ ] 2025-02-19 00:22:23.063 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==      Total: 1 
[ ] 2025-02-19 00:22:23.065 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: update inlong_group SET name = ?, description = ?, mq_type = ?, mq_resource = ?, daily_records = ?, daily_storage = ?, peak_records = ?, max_length = ?, enable_zookeeper = ?, enable_create_resource = ?, inlong_group_mode = ?, data_report_type = ?, in_charges = ?, previous_status = status, status = ?, is_deleted = ?, modifier = ?, version = ? + 1 WHERE inlong_group_id = ? and tenant = ? and is_deleted = 0 and version = ? 
[ ] 2025-02-19 00:22:23.067 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: (String), 1231313(String), NONE(String), test_off_group(String), 10(Integer), 10(Integer), 1000(Integer), 10240(Integer), 0(Integer), 1(Integer), 2(Integer), 0(Integer), admin(String), 40(Integer), 19(Integer), admin(String), 1(Integer), test_off_group(String), public(String), 1(Integer) 
[ ] 2025-02-19 00:22:23.068 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==    Updates: 1 
[ ] 2025-02-19 00:22:23.071 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: update inlong_group_ext set is_deleted = id where inlong_group_id = ? and is_deleted = 0 
[ ] 2025-02-19 00:22:23.072 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String) 
[ ] 2025-02-19 00:22:23.074 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==    Updates: 0 
[ ] 2025-02-19 00:22:23.074 -DEBUG [http-nio-8083-exec-7] .a.i.m.s.s.ScheduleServiceImpl:154 - begin to delete schedule info for groupId=test_off_group 
[ ] 2025-02-19 00:22:23.075 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: select id, inlong_group_id, name, description, tenant, mq_type, mq_resource, daily_records, daily_storage, peak_records, max_length, enable_zookeeper, enable_create_resource, inlong_group_mode, data_report_type, inlong_cluster_tag, ext_params, in_charges, followers, status, previous_status, is_deleted, creator, modifier, create_time, modify_time, version from inlong_group WHERE inlong_group_id = ? and is_deleted = 0 and tenant = ? 
[ ] 2025-02-19 00:22:23.076 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String), public(String) 
[ ] 2025-02-19 00:22:23.077 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==      Total: 0 
[ ] 2025-02-19 00:22:23.077 -ERROR [http-nio-8083-exec-7] .a.i.m.s.s.ScheduleServiceImpl:179 - inlong group not found by groupId=test_off_group 
[ ] 2025-02-19 00:22:23.079 - WARN [http-nio-8083-exec-7] i.m.s.g.InlongGroupServiceImpl:694 - failed to delete schedule info for groupId=test_off_group, error msg: Inlong group does not exist/no operation authority 
[ ] 2025-02-19 00:22:23.079 - INFO [http-nio-8083-exec-7] i.m.s.g.InlongGroupServiceImpl:698 - success to delete group and group ext property for groupId=test_off_group by user=admin 
[ ] 2025-02-19 00:22:23.081 - INFO [http-nio-8083-exec-7] .m.s.s.StreamSourceServiceImpl:442 - begin to logic delete all source info by groupId=test_off_group, streamId=null 
[ ] 2025-02-19 00:22:23.083 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: select id, inlong_group_id, inlong_stream_id, source_type, source_name, task_map_id, agent_ip, uuid, data_node_name, inlong_cluster_name, inlong_cluster_node_group, serialization_type, snapshot, report_time, data_time_zone, ext_params, version, status, previous_status, is_deleted, creator, modifier, create_time, modify_time from stream_source WHERE is_deleted = 0 and inlong_group_id = ? 
[ ] 2025-02-19 00:22:23.085 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: test_off_group(String) 
[ ] 2025-02-19 00:22:23.086 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==      Total: 0 
[ ] 2025-02-19 00:22:23.086 - INFO [http-nio-8083-exec-7] .m.s.s.StreamSourceServiceImpl:465 - success to logic delete all source by groupId=test_off_group, streamId=null 
[ ] 2025-02-19 00:22:23.087 - INFO [http-nio-8083-exec-7] .g.UpdateGroupCompleteListener:104 - success to execute UpdateGroupCompleteListener for groupId=test_off_group, operateType=DELETE 
[ ] 2025-02-19 00:22:23.089 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==>  Preparing: insert into workflow_event_log (id, process_id, process_name, process_display_name, inlong_group_id, task_id, element_name, element_display_name, event_type, event, listener, status, async, ip, start_time, end_time, remark, exception) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) 
[ ] 2025-02-19 00:22:23.090 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - ==> Parameters: null, 11(Integer), DELETE_GROUP_PROCESS(String), Delete Group(String), test_off_group(String), null, EndEvent(String), EndEvent(String), ProcessEvent(String), COMPLETE(String), UpdateGroupCompleteListener(String), 1(Integer), 0(Integer), 192.168.157.1(String), 2025-02-19 00:22:23.021(Timestamp), 2025-02-19 00:22:23.087(Timestamp), null, null 
[ ] 2025-02-19 00:22:23.092 -DEBUG [http-nio-8083-exec-7] o.a.i.l.j.BaseJdbcLogger      :137 - <==    Updates: 1 
[ ] 2025-02-19 00:22:23.098 - INFO [http-nio-8083-exec-7] .s.g.InlongGroupProcessService:267 - success to delete group for groupId=test_off_group by user=admin 

Image

Image

@Jzow Jzow requested review from vernedeng and fuweng11 February 18, 2025 16:32
@vernedeng vernedeng merged commit 663a1fe into apache:master Feb 19, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug][Manager] Failed to delete inlong group for groupId
4 participants