Skip to content

Updated reload_connections set false by default in in_opensearch.rb #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -605,10 +605,10 @@ request_timeout 15s # defaults to 5s

### reload_connections

You can tune how the opensearch-transport host reloading feature works. By default it will reload the host list from the server every 10,000th request to spread the load. This can be an issue if your OpenSearch cluster is behind a Reverse Proxy, as Fluentd process may not have direct network access to the OpenSearch nodes.
You can tune how the opensearch-transport host reloading feature works. By default it off but if you set true it will reload the host list from the server every 10,000th request to spread the load. This can be an issue if your OpenSearch cluster is behind a Reverse Proxy, as Fluentd process may not have direct network access to the OpenSearch nodes.

```
reload_connections false # defaults to true
reload_connections false # defaults to false
```

### reload_on_failure
Expand Down
12 changes: 8 additions & 4 deletions lib/fluent/plugin/in_opensearch.rb
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ class UnrecoverableRequestFailure < Fluent::UnrecoverableError; end
config_param :repeat, :bool, :default => true
config_param :http_backend, :enum, list: [:excon, :typhoeus], :default => :excon
config_param :request_timeout, :time, :default => 5
config_param :reload_connections, :bool, :default => true
config_param :reload_connections, :bool, :default => false
config_param :reload_on_failure, :bool, :default => false
config_param :resurrect_after, :time, :default => 60
config_param :reload_after, :integer, :default => DEFAULT_RELOAD_AFTER
Expand Down Expand Up @@ -215,7 +215,7 @@ def reachable_host?(host)
host: ["#{host[:scheme]}://#{host[:host]}:#{host[:port]}"],
user: host[:user],
password: host[:password],
reload_connections: @reload_connections,
# reload_connections: @reload_connections,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, could you describe why this line is commented out?
Would you mind if you revive this line?

request_timeout: @request_timeout,
resurrect_after: @resurrect_after,
reload_on_failure: @reload_on_failure,
Expand Down Expand Up @@ -343,14 +343,18 @@ def update_retry_state(error=nil)
@retry = retry_state(@retry_randomize)
end
@retry.step
if error.message.include?('EOFError (EOFError)')
log.error("Restart plugin because hit error #{error.message}")
start
end
#Raise error if the retry limit has been reached
raise "Hit limit for retries. retry_times: #{@retry.steps}, error: #{error.message}" if @retry.limit?
#Retry if the limit hasn't been reached
log.warn("failed to connect or search.", retry_times: @retry.steps, next_retry_time: @retry.next_time.round, error: error.message)
sleep(@retry.next_time - Time.now)
else
unless @retry.nil?
log.debug("retry succeeded.")
log.info("retry succeeded.")
@retry = nil
end
end
Expand All @@ -366,7 +370,7 @@ def run
run_slice(slice_id)
end
end
rescue Faraday::ConnectionFailed, OpenSearch::Transport::Transport::Error => error
rescue => error
update_retry_state(error)
retry
end
Expand Down
Loading