Error Elastalert2 logs

[ngms17]

Hi,

I am seeing this recorrent errors on my Elastalert logs. Everything is working fine but i can´t figure it out why this error is ocurring. Any insight on how to resolve this issue?

Thanks

Feb 17 14:44:39 evesuri elastalert[1001]: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Feb 17 14:44:39 evesuri elastalert[1001]: ERROR:elastalert:Error writing alert info to Elasticsearch: ConnectionError(('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))) caused by: ConnectionError(('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))
Feb 17 14:44:39 evesuri elastalert[1001]: Traceback (most recent call last):
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
Feb 17 14:44:39 evesuri elastalert[1001]:     httplib_response = self._make_request(
Feb 17 14:44:39 evesuri elastalert[1001]:                        ^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request
Feb 17 14:44:39 evesuri elastalert[1001]:     six.raise_from(e, None)
Feb 17 14:44:39 evesuri elastalert[1001]:   File "<string>", line 3, in raise_from
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request
Feb 17 14:44:39 evesuri elastalert[1001]:     httplib_response = conn.getresponse()
Feb 17 14:44:39 evesuri elastalert[1001]:                        ^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/http/client.py", line 1395, in getresponse
Feb 17 14:44:39 evesuri elastalert[1001]:     response.begin()
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/http/client.py", line 325, in begin
Feb 17 14:44:39 evesuri elastalert[1001]:     version, status, reason = self._read_status()
Feb 17 14:44:39 evesuri elastalert[1001]:                               ^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/http/client.py", line 286, in _read_status
Feb 17 14:44:39 evesuri elastalert[1001]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Feb 17 14:44:39 evesuri elastalert[1001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/socket.py", line 718, in readinto
Feb 17 14:44:39 evesuri elastalert[1001]:     return self._sock.recv_into(b)
Feb 17 14:44:39 evesuri elastalert[1001]:            ^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]: ConnectionResetError: [Errno 104] Connection reset by peer
Feb 17 14:44:39 evesuri elastalert[1001]: During handling of the above exception, another exception occurred:
Feb 17 14:44:39 evesuri elastalert[1001]: Traceback (most recent call last):
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/requests-2.31.0-py3.11.egg/requests/adapters.py", line 486, in send
Feb 17 14:44:39 evesuri elastalert[1001]:     resp = conn.urlopen(
Feb 17 14:44:39 evesuri elastalert[1001]:            ^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
Feb 17 14:44:39 evesuri elastalert[1001]:     retries = retries.increment(
Feb 17 14:44:39 evesuri elastalert[1001]:               ^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
Feb 17 14:44:39 evesuri elastalert[1001]:     raise six.reraise(type(error), error, _stacktrace)
Feb 17 14:44:39 evesuri elastalert[1001]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
Feb 17 14:44:39 evesuri elastalert[1001]:     raise value.with_traceback(tb)
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
Feb 17 14:44:39 evesuri elastalert[1001]:     httplib_response = self._make_request(
Feb 17 14:44:39 evesuri elastalert[1001]:                        ^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request
Feb 17 14:44:39 evesuri elastalert[1001]:     six.raise_from(e, None)
Feb 17 14:44:39 evesuri elastalert[1001]:   File "<string>", line 3, in raise_from
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request
Feb 17 14:44:39 evesuri elastalert[1001]:     httplib_response = conn.getresponse()
Feb 17 14:44:39 evesuri elastalert[1001]:                        ^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/http/client.py", line 1395, in getresponse
Feb 17 14:44:39 evesuri elastalert[1001]:     response.begin()
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/http/client.py", line 325, in begin
Feb 17 14:44:39 evesuri elastalert[1001]:     version, status, reason = self._read_status()
Feb 17 14:44:39 evesuri elastalert[1001]:                               ^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/http/client.py", line 286, in _read_status
Feb 17 14:44:39 evesuri elastalert[1001]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Feb 17 14:44:39 evesuri elastalert[1001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/lib/python3.11/socket.py", line 718, in readinto
Feb 17 14:44:39 evesuri elastalert[1001]:     return self._sock.recv_into(b)
Feb 17 14:44:39 evesuri elastalert[1001]:            ^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]: urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Feb 17 14:44:39 evesuri elastalert[1001]: During handling of the above exception, another exception occurred:
Feb 17 14:44:39 evesuri elastalert[1001]: Traceback (most recent call last):
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elasticsearch-7.10.1-py3.11.egg/elasticsearch/connection/http_requests.py", line 161, in perform_request
Feb 17 14:44:39 evesuri elastalert[1001]:     response = self.session.send(prepared_request, **send_kwargs)
Feb 17 14:44:39 evesuri elastalert[1001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/requests-2.31.0-py3.11.egg/requests/sessions.py", line 703, in send
Feb 17 14:44:39 evesuri elastalert[1001]:     r = adapter.send(request, **kwargs)
Feb 17 14:44:39 evesuri elastalert[1001]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/requests-2.31.0-py3.11.egg/requests/adapters.py", line 501, in send
Feb 17 14:44:39 evesuri elastalert[1001]:     raise ConnectionError(err, request=request)
Feb 17 14:44:39 evesuri elastalert[1001]: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Feb 17 14:44:39 evesuri elastalert[1001]: During handling of the above exception, another exception occurred:
Feb 17 14:44:39 evesuri elastalert[1001]: Traceback (most recent call last):
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elastalert2-2.17.0-py3.11.egg/elastalert/elastalert.py", line 1499, in writeback
Feb 17 14:44:39 evesuri elastalert[1001]:     res = self.writeback_es.index(index=index, body=body)
Feb 17 14:44:39 evesuri elastalert[1001]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elasticsearch-7.10.1-py3.11.egg/elasticsearch/client/utils.py", line 152, in _wrapped
Feb 17 14:44:39 evesuri elastalert[1001]:     return func(*args, params=params, headers=headers, **kwargs)
Feb 17 14:44:39 evesuri elastalert[1001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elasticsearch-7.10.1-py3.11.egg/elasticsearch/client/__init__.py", line 398, in index
Feb 17 14:44:39 evesuri elastalert[1001]:     return self.transport.perform_request(
Feb 17 14:44:39 evesuri elastalert[1001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elasticsearch-7.10.1-py3.11.egg/elasticsearch/transport.py", line 390, in perform_request
Feb 17 14:44:39 evesuri elastalert[1001]:     raise e
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elasticsearch-7.10.1-py3.11.egg/elasticsearch/transport.py", line 358, in perform_request
Feb 17 14:44:39 evesuri elastalert[1001]:     status, headers_response, data = connection.perform_request(
Feb 17 14:44:39 evesuri elastalert[1001]:                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 17 14:44:39 evesuri elastalert[1001]:   File "/usr/local/lib/python3.11/dist-packages/elasticsearch-7.10.1-py3.11.egg/elasticsearch/connection/http_requests.py", line 177, in perform_request
Feb 17 14:44:39 evesuri elastalert[1001]:     raise ConnectionError("N/A", str(e), e)
Feb 17 14:44:39 evesuri elastalert[1001]: elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))) caused by: ConnectionError(('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))

[jertel]

Check your Elasticsearch server to see if it's logging out errors during those requests.

[ngms17]

This are the logs on the Elasticsearch server side

[2025-02-11T15:35:06,661][WARN ][o.e.h.AbstractHttpServerTransport] [timon] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ES-IP:9200, remoteAddress=/Elastalert-IP:36398}
io.netty.handler.codec.PrematureChannelClosureException: Channel closed while still aggregating message
	at io.netty.handler.codec.MessageAggregator.channelInactive(MessageAggregator.java:436) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[?:?]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[?:?]
	at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:235) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[?:?]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelInactive(Netty4HttpHeaderValidator.java:205) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:412) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:377) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[?:?]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4WriteThrottlingHandler.channelInactive(Netty4WriteThrottlingHandler.java:172) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1352) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:850) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:811) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1575) ~[?:?]

[jertel]

Seems like a network issue. ElastAlert 2 is saying Elasticsearch closed the connection, and Elasticsearch is saying ElastAlert 2 dropped the connection. So something in the middle is messing with the connection.