Add retry to bulk indexer #39
Conversation
lib/utility/es_client.rb
Outdated
| retries += 1 | ||
| if retries <= MAX_RETRIES | ||
| @system_logger.info("Bulk index attempt #{retries} failed: #{e.message}. Retrying...") | ||
| sleep(1.second) |
There was a problem hiding this comment.
IMO fallback should be added here too, exponential is a good candidate for 3 retries
There was a problem hiding this comment.
@artem-shelkovnikov I've added a temporary fallback; if a bulk index fails, it saves the failed payload to a file and outputs that file name to the log. Users can cross reference the file content with whatever error response they received.
In the future I'd like to implement something else, but for now I think this should suffice.
I also added exponentially increasing wait times for the retries.
Now, in testing this, I realised the bulk queue can get overloaded if it encounters a lot of errors and exponentially backs off for many seconds. The crawler continues to try sending crawl results and the bulk indexer trips over itself and chaos.
I have an idea to fix this but I think it's out of scope for this PR, so I'll do it in a follow-up PR.
Related to #37
We currently have no retry logic for bulk indexing. This is a problem when running the Crawler on Mac because of an intermittent
bad_record_macerror. Retrying often resolves the error (so this is a bandaid, not a fix).Retries are healthy in general so this was a good opportunity to add them.