Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement] improve async submit streamload #357

Closed
wants to merge 2 commits into from

Conversation

JNSimba
Copy link
Member

@JNSimba JNSimba commented Apr 2, 2024

Proposed changes

Issue Number: close #xxx

Problem Summary:

Now when writing to multiple tables, when the checkpoint arrives, an HTTP request will be submitted. If the number of tables is large, the subsequent tables will continue to wait. Here, the submission will be turned into an asynchronous request to reduce the waiting on the connector side.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@JNSimba
Copy link
Member Author

JNSimba commented Apr 6, 2024

Considering that stopload should theoretically be very fast because it only submits http requests, so if stopload is particularly slow, it means there may be a problem on the Doris side. At this time, becoming asynchronous will only cause more congestion and create a larger backlog.
So close it first and look at the situation later.

@JNSimba JNSimba marked this pull request as draft April 6, 2024 12:28
@JNSimba JNSimba closed this Apr 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant