Skip to content

enable input metrics #578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: 1.0
Choose a base branch
from
Open

enable input metrics #578

wants to merge 3 commits into from

Conversation

Watson1978
Copy link
Contributor

No description provided.

@Watson1978
enable input metrics
9bbb040 
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 mentioned this pull request May 20, 2025
Merged
@Watson1978 Watson1978 requested a review from daipom May 20, 2025 05:42
kenhys
kenhys approved these changes May 20, 2025
View reviewed changes
Copy link
Contributor

@kenhys kenhys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kenhys kenhys added the pending To be done in the future label May 20, 2025
@kenhys
Copy link
Contributor

kenhys commented May 20, 2025

When v1.19 is ready, deploy it!

@Watson1978
add --disable-input-metrics option
038d36f 
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 force-pushed the enable-input-metrics branch from 94d9952 to 0129dde Compare May 21, 2025 04:13
@Watson1978
change default value in enable_input_metrics
ce48ad9 
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 force-pushed the enable-input-metrics branch from 0129dde to ce48ad9 Compare May 21, 2025 04:14
daipom added a commit to fluent/fluentd that referenced this pull request May 21, 2025
@Watson1978 @daipom
metrics: enable input metrics by default (#4966)
eeb384e 
**Which issue(s) this PR fixes**: 
Fixes #

**What this PR does / why we need it**: 
I ran fluend with `--enable-input-metrics` option and read 10GB file
with the `in_tail` plugin
There was no significant performance degradation.


So, I think it is safe to always enable `--enable-input-metrics`.

### environment
* ruby 3.4.3 (2025-04-14 revision d0b7e5b6a0) +PRISM [x86_64-linux]

### results
* fluentd master branch (efbb51d)
    * `ruby bin/fluentd -c in_tail.conf`
        * 65.939232375 seconds
    * `ruby bin/fluentd --enable-input-metrics -c in_tail.conf`
        * 67.776236261 seconds
* fluentd 1.16.9 (daccbc6)
    * `ruby bin/fluentd -c in_tail.conf`
        * 106.312500419 seconds

### config
```
<source>
  @type tail
  tag test
  path "#{File.expand_path '~/tmp/fluentd/access-*.log'}"
  read_from_head true
  <parse>
    @type json
  </parse>
</source>

<match **>
  @type file
  path "#{File.expand_path '~/tmp/fluentd/log'}"
</match>
```

### script to generate 10GB
```ruby
# frozen_string_literal: true
require "json"
require "fileutils"

def data_generater(str)
  {
    "message": str * 1000,
  }.to_json
end

FILE_MAX_SIZE = 10 * 1024 * 1024 * 1024
FILE_PATH = "/home/watson/tmp/fluentd/access-1.log"

dir = File.dirname(FILE_PATH)
FileUtils.mkdir_p(dir)

File.open(FILE_PATH, "w") do |f|
  data = data_generater('a')
  loop do
    f.puts data
    break if File.size(FILE_PATH) > FILE_MAX_SIZE
  end
end
```

**Docs Changes**:
fluent/fluentd-docs-gitbook#578

**Release Note**: 
The same as the title.

---------

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Co-authored-by: Daijiro Fukuda <fukuda@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kenhys kenhys kenhys approved these changes

@daipom daipom Awaiting requested review from daipom

Assignees
No one assigned
Labels
pending To be done in the future
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Watson1978 @kenhys