Logs duplication in opensearch using fluent agents

[HusnainTaja]

Hi,

We are trying to pull logs from elasticsearch using fluentd and then fluentd forwarding the logs to fluentbit and then fluentbit is storing them into opensearch.

Now the issue is that whenever the fluentd agent runs after every interval so it is duplicating data into opensearch.
I have only 2 docs in elasticsearch index but in opensearch it is 50 as duplicated one.

Below are the configurations for fluentd and fluentbit

1.) Fluentd input configurations:

elasticsearch_input.yaml
- source:
    $type: elasticsearch
    host: 10.67.39.14
    port: '9200'
    tag: 1eb4fefb-c076-45f9-af99-4ea14ddcb2f1
    docinfo: true
    index_name: nap_logs_v1-005940
    ssl_verify: false
    user: "elastic"
    password: "eu=dmM1_JtVJ4RDkVxYH"
    scheme: https
    interval: 10s
    query: '{"query": {"match_all": {}}}'

2.) Fluentd main file

fluentd.yaml
system:
  log_level: error
  log_file: /home/atomiqroot/fluentd.log
config:
- !include config.d/*.yaml
- match:
    $type: forward
    server:
      name: 10.67.39.14
      host: 10.67.39.14
      port: 3000

3.) fluentbit inout file
collection.yaml

pipeline:
  inputs:
    - name: forward
      listen: 0.0.0.0
      port: 3000  # Fluent Bit listens for logs from Fluentd
      buffer_chunk_size: 8MB
      buffer_max_size: 2048MB

4.) Fluentbit main file

[root@atomiq-monitoring-catalog fluent-bit]# cat fluent-bit.yaml

# fluent-bit main configuration
service:
  flush: 5
  log_level: error
  log_file: /var/log/fluent-bit.log

pipeline:
  outputs:
    - name: opensearch
      match: 1eb4fefb-c076-45f9-af99-4ea14ddcb2f1
      host: <host>
      port: 443
      tls: 'on'
      http_user: <user>
      http_passwd: <user>
      index: test-dup4  # Index for first source
      tls.verify: 'off'
      suppress_type_name: 'on'
      #time_key: Timestamp
      #time_key_format: "%Y-%m-%dT%H:%M:%S.%LZ"

includes:
  - config.d/*.yaml
  
  
   

Please provide the solution