Skip to content

Commit

Permalink
Remove dev code
Browse files Browse the repository at this point in the history
  • Loading branch information
navarone-feekery committed May 29, 2024
1 parent 096659c commit 71bfcaa
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 2 deletions.
2 changes: 1 addition & 1 deletion lib/crawler/api/config.rb
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ def document_mapper
# Receives a crawler event object and outputs it into relevant systems
def output_event(event)
# Log the event
# event_logger << "#{event.to_json}\n"
event_logger << "#{event.to_json}\n"

# Count stats for the crawl
stats.update_from_event(event)
Expand Down
4 changes: 4 additions & 0 deletions lib/crawler/url_validator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ class InvalidCrawlConfigError < Error; end
attr_reader :raw_url, :checks, :results, :url_crawl_result

def initialize(url:, crawl_config:, checks: nil)
if configuration && configuration.crawler_domains.empty?
raise InvalidCrawlConfigError, 'Please configure at least one domain in the crawl config file.'
end

@crawl_config = crawl_config
# Default to running all checks for the given context
checks ||= valid_checks
Expand Down
2 changes: 1 addition & 1 deletion lib/crawler/url_validator/url_request_check_concern.rb
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ def redirect_validation_result(details) # rubocop:disable Metrics/AbcSize
end

# If we're running in a domain context, this is an inter-domain redirect that we cannot follow
unless @crawl_config
unless configuration
return validation_fail(:url_request, <<~MESSAGE, details)
The web server at #{url} redirected us to a different domain URL (#{location}).
If you want to crawl this site, please use #{location.domain_name} as the domain name.
Expand Down

0 comments on commit 71bfcaa

Please sign in to comment.