Increase max event size to 1MB

[amykelly]

AWS has increased the max event size to 1 MB: AWS News

This is already reflected in the code in some places: Line 376 and Line 70

However, the variable MAX_EVENT_SIZE has not been updated. As such, we are still seeing errors reporting that the log event is too large.

[daipom]

Thanks for your contribution!

It looks like there are two types of size limits: one event size limit and an overall size limit.
The one event size limit was added in the following PR:

• Discard events that exceed limit size, 256KB. #46

AWS has increased the max event size to 1 MB: AWS News

Sorry I'm not familiar with CloudWatch.
Does this mean that the one-event size limit has recently changed?

[Watson1978]

I think CI error should be fixed by #266

[amykelly]

Makes sense! According the the limit docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html

Batch size - 1,048,576 bytes
Log event size - 1 MB

It's the log event size that has been updated last month

[daipom]

I think CI error should be fixed by #266

Thanks @Watson1978 !

It's the log event size that has been updated last month

I see! Thanks!
Then this fix looks good to me.

It seems that we need to change this test.

fluent-plugin-cloudwatch-logs/test/plugin/test_out_cloudwatch_logs.rb

Lines 968 to 982 in f3bc56e

	def test_too_large_event
	time = Fluent::Engine.now
	d = create_driver(<<-EOC)
	#{default_config}
	log_group_name #{log_group_name}
	log_stream_name #{log_stream_name}
	@log_level debug
	EOC
	d.run(default_tag: fluentd_tag) do
	d.feed(time, {'message' => '*' * 256 * 1024})
	end
	logs = d.logs
	assert(logs.any?{|log| log =~ /Log event in .* discarded because it is too large: 262184 bytes exceeds limit of 262144/})
	end

However, some CIs have passed without this test failing...

[daipom]

However, some CIs have passed without this test failing...

I understand.
real world tests are omitted on CI.
So this test does not run on CI.

[Watson1978]

CI should be fixed at #267

@amykelly Can you rebase your commit with master brunch?

[daipom]

#265 (comment)

It seems that we need to change this test.

@amykelly
And could you please fix this test? For example, as follows

diff --git a/test/plugin/test_out_cloudwatch_logs.rb b/test/plugin/test_out_cloudwatch_logs.rb
index d849876..0977d9e 100644
--- a/test/plugin/test_out_cloudwatch_logs.rb
+++ b/test/plugin/test_out_cloudwatch_logs.rb
@@ -974,11 +974,11 @@ class CloudwatchLogsOutputTest < Test::Unit::TestCase
         @log_level debug
       EOC
       d.run(default_tag: fluentd_tag) do
-        d.feed(time, {'message' => '*' * 256 * 1024})
+        d.feed(time, {'message' => '*' * 1024 * 1024})
       end
 
       logs = d.logs
-      assert(logs.any?{|log| log =~ /Log event in .* discarded because it is too large: 262184 bytes exceeds limit of 262144/})
+      assert(logs.any?{|log| log =~ /Log event in .* discarded .*/})
     end
 
     def test_do_not_emit_empty_record

[daipom]

Thanks!! @amykelly @Watson1978

[daipom]

@amykelly We have released v0.15.0!