Opensearch - using with a fallback output results in indexed messages in a DLQ

[jem-davies]

slack link

If you run an opensearch (and presumably an elasticsearch output) using fallback with a Dead Letter Queue pattern, it results in successfully indexed messages in the DLQ, if other messages in the same batch fail.

[jem-davies]

Unable to reproduce - appears that the opensearch does use a batchError and therefore the fallback & reject_errored DLQ pattern does work:

# After creating 5 documents in index /test with id's 1 - 5

input:
  file:
    paths: 
      - ./createOneDeleteOne.txt
      # {"first_name":"Stephen", "last_name": "Thiel"}
      # {"first_name":"Daisy", "last_name": "Schoen"}
    scanner:
      lines: {}

pipeline:
  processors:
    - mapping: |
        if this.first_name == "Stephen" {
          meta id = 6
          meta action = "delete"
        }
        if this.first_name == "Daisy" {
          meta id = 7
          meta action = "index"
        }

output:
  fallback:
    - reject_errored:
        opensearch:
          urls: 
            - http://0.0.0.0:9200
          index: test
          action: ${! metadata("action") }
          id: ${! metadata("id") }
          batching:
            count: 2
    - stdout: {} 
    # Only seeing the expected 1 failed message in stdout:
    # ERRO Failed to send message to opensearch: status 404: unknown error  @service=bento label="" path=root.output.fallback.0.reject_errored
    # {"first_name":"Stephen", "last_name": "Thiel"}

[jem-davies]

services:
  opensearch:
    image: opensearchproject/opensearch:latest
    container_name: opensearch
    environment:
      - discovery.type=single-node
      - DISABLE_SECURITY_PLUGIN=true
    ports:
      - "9200:9200"
      - "9600:9600"

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.18.0
    container_name: opensearch-dashboards
    environment:
      - OPENSEARCH_HOSTS=http://opensearch:9200
      - DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
    ports:
      - "5601:5601"
    depends_on:
      - opensearch