Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up pr review comment recollection #2975

Merged
merged 1 commit into from
Jan 22, 2025
Merged

Speed up pr review comment recollection #2975

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion augur/tasks/github/pull_requests/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@
from augur.application.db.models import PullRequest, Message, PullRequestReview, PullRequestLabel, PullRequestReviewer, PullRequestMeta, PullRequestAssignee, PullRequestReviewMessageRef, Contributor, Repo
from augur.tasks.github.util.github_task_session import GithubTaskManifest
from augur.tasks.github.util.github_random_key_auth import GithubRandomKeyAuth
from augur.application.db.lib import get_session, get_repo_by_repo_git, bulk_insert_dicts, get_pull_request_reviews_by_repo_id

Check warning on line 14 in augur/tasks/github/pull_requests/tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0611: Unused get_session imported from augur.application.db.lib (unused-import)

Raw Output:
augur/tasks/github/pull_requests/tasks.py:14:0: W0611: Unused get_session imported from augur.application.db.lib (unused-import)
from augur.application.db.util import execute_session_query
from ..messages import process_github_comment_contributors
from augur.application.db.lib import get_secondary_data_last_collected, get_updated_prs, get_core_data_last_collected

from typing import Generator, List, Dict

Check warning on line 19 in augur/tasks/github/pull_requests/tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0611: Unused Generator imported from typing (unused-import)

Raw Output:
augur/tasks/github/pull_requests/tasks.py:19:0: W0611: Unused Generator imported from typing (unused-import)

Check warning on line 19 in augur/tasks/github/pull_requests/tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0611: Unused Dict imported from typing (unused-import)

Raw Output:
augur/tasks/github/pull_requests/tasks.py:19:0: W0611: Unused Dict imported from typing (unused-import)


platform_id = 1
Expand Down Expand Up @@ -52,11 +52,11 @@
total_count += len(all_data)
all_data.clear()

if len(all_data):

Check warning on line 55 in augur/tasks/github/pull_requests/tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C1802: Do not use `len(SEQUENCE)` without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

Raw Output:
augur/tasks/github/pull_requests/tasks.py:55:11: C1802: Do not use `len(SEQUENCE)` without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)
process_pull_requests(all_data, f"{owner}/{repo}: Github Pr task", repo_id, logger, augur_db)
total_count += len(all_data)

if total_count > 0:

Check warning on line 59 in augur/tasks/github/pull_requests/tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)

Raw Output:
augur/tasks/github/pull_requests/tasks.py:59:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
return total_count
else:
logger.debug(f"{owner}/{repo} has no pull requests")
Expand Down Expand Up @@ -219,7 +219,7 @@
return pr_review_cntrb

@celery.task(base=AugurSecondaryRepoCollectionTask)
def collect_pull_request_review_comments(repo_git: str) -> None:
def collect_pull_request_review_comments(repo_git: str, full_collection: bool) -> None:

owner, repo = get_owner_repo(repo_git)

Expand All @@ -230,6 +230,11 @@

repo_id = get_repo_by_repo_git(repo_git).repo_id

if not full_collection:
# subtract 2 days to ensure all data is collected
core_data_last_collected = (get_core_data_last_collected(repo_id) - timedelta(days=2)).replace(tzinfo=timezone.utc)
review_msg_url += f"?since={core_data_last_collected.isoformat()}"

pr_reviews = get_pull_request_reviews_by_repo_id(repo_id)

# maps the github pr_review id to the auto incrementing pk that augur stores as pr_review id
Expand Down
2 changes: 1 addition & 1 deletion augur/tasks/start_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@
repo_task_group = group(
process_pull_request_files.si(repo_git, full_collection),
process_pull_request_commits.si(repo_git, full_collection),
chain(collect_pull_request_reviews.si(repo_git, full_collection), collect_pull_request_review_comments.si(repo_git)),
chain(collect_pull_request_reviews.si(repo_git, full_collection), collect_pull_request_review_comments.si(repo_git, full_collection)),
process_ossf_dependency_metrics.si(repo_git)
)

Expand Down Expand Up @@ -295,7 +295,7 @@
status = repo.collection_status[0]
raw_count = status.issue_pr_sum

issue_pr_task_update_weight_util([int(raw_count)],repo_git=repo_git,session=session)

Check warning on line 298 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
E1120: No value for argument 'issue_and_pr_nums' in function call (no-value-for-parameter)

Raw Output:
augur/tasks/start_tasks.py:298:12: E1120: No value for argument 'issue_and_pr_nums' in function call (no-value-for-parameter)

facade_not_pending = CollectionStatus.facade_status != CollectionState.PENDING.value
facade_not_failed = CollectionStatus.facade_status != CollectionState.FAILED_CLONE.value
Expand Down
Loading