Add purge crawl feature #65
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
navarone-feekery
force-pushed
the
navarone/add-purge-docs
branch
2 times, most recently
from
f0f4b44 to
e067a77
Compare
navarone-feekery
force-pushed
the
navarone/add-purge-docs
branch
from
e067a77 to
c009ddf
Compare
This comment was marked as resolved.
This comment was marked as resolved.
seanstory
reviewed
Aug 9, 2024
seanstory
approved these changes
Aug 21, 2024
| @@ -38,6 +38,7 @@ | |||
| let(:sink) { Crawler::OutputSink::Mock.new(crawl_config) } | |||
| let(:crawl_queue) { Crawler::Data::UrlQueue::MemoryOnly.new(crawl_config) } | |||
| let(:seen_urls) { Crawler::Data::SeenUrls.new } | |||
| let(:crawl_result) { FactoryBot.build(:html_crawl_result, content: '<p>BOO!</p>') } | |||
There was a problem hiding this comment.
Sometimes I wonder if you leave things like this in here just to see if we're actually reading/reviewing closely. :)
There was a problem hiding this comment.
I'm definitely not that sly, they're just easter eggs so I can laugh when I have to fix a test in a year :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add the Purge Crawl feature, which allows Crawler to delete outdated docs from the index.
Purge crawls can only run if the output sink is
elasticsearch. It can be disabled via the config yaml file by settingpurge_crawl_enabled: false.Purge crawls are performed after the primary crawl is finished. Crawler will perform a
_searchquery against the ES index to find all documents that have alast_crawled_atearlier than the primary crawl's start time.Crawler then sends all of these docs to another "purge" crawl. During this crawl, no links are extracted. The only intention is to determine if the pages that these docs represent still exist and are accessible. Pages that return a non-200 response, or are blocked by existing crawl rules, can be deleted.
Crawler will also re-index the docs it finds during the purge crawl, to ensure any changes are reflected and
last_crawled_atis up-to-date.When the purge crawl is complete, any docs that are remaining are deleted from the index. This is done in one attempt using the
delete_by_query.I will add documentation in another PR.
Checklists
Pre-Review Checklist
crawler.yml.exampleandelasticsearch.yml.example)v0.1.0)Release Note
Crawler will now purge outdated documents from the index at the end of the crawl. This is enabled by default. You can disable this by adding
purge_docs_enabled: falseto the crawler's yaml config file.