[0.2] Adding ES verification step + explicit best-effort index creation during ES Sink initialization (#192) #207
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…ing ES Sink initialization (#192) ### Closes #53 and #172 This is a continuation of issue #53 but also closes #172. This PR will add the following steps during the initialization of the ES Sink: - A verification step that checks if crawler can reach the Elasticsearch instance provided in configs - An explicit attempt to create the output_index should the index ping fail (index ping step was added in #186 ) Thus, the flow during init will be like: `verify ES connection`--> `if all good, verify the output_index `--> `if index does not exist, attempt to create the index` --> `if index creation fails, system exit` Additional background: While working on this, I discovered that _technically speaking_, the _bulk command that Crawler uses to upsert documents is capable of auto-creating the index if it doesn't exist. However, this is dependent on the user having `auto_configure`, `create_index`, or `manage` index privileges. Therefore, while we may not _need_ an explicit index creation attempt, it is good to have because we can then explicitly log that it happened, and also provide a safe point to fail out at should something go wrong vs. waiting for _bulk to be called, at which point a crawl would have already begun. ### Checklists #### Pre-Review Checklist - [x] This PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check `crawler.yml.example` and `elasticsearch.yml.example`) - [x] This PR has a meaningful title - [x] This PR links to all relevant GitHub issues that it fixes or partially addresses - If there is no GitHub issue, please create it. Each PR should have a link to an issue - [x] this PR has a thorough description - [x] Covered the changes with automated tests - [x] Tested the changes locally - [x] Added a label for each target release version (example: `v0.1.0`) - [x] Considered corresponding documentation changes ### Related Pull Requests #186
navarone-feekery
approved these changes
Feb 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 0.2: