Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Curator 7.0.1 fails because of closed indices #1733

Open
mike-somethingnooneelsehasusedyet opened this issue Jan 20, 2025 · 0 comments
Open

Comments

@mike-somethingnooneelsehasusedyet

(See also https://discuss.elastic.co/t/curator-7-0-1-fails-because-of-closed-indices/373190/1 but I haven't had any responses and maybe this is a bug.)

Elasticsearch version: 7.17.1 (yes, I know.)

Curator version: 7.0.1

Expected results: Curator behaves the same as 5.8.4.

Action results: Every action fails because of closed indices.

Config:

---
client:
  hosts:
    - foo.bar.redact
  port: 9200
  url_prefix:
  use_ssl: True
  certificate: /etc/pki/tls/certs/ca-bundle.crt
  client_cert:
  client_key:
  ssl_no_validate: False
  username: "curator"
  password: "redact"
  timeout: 30
    #  master_only: True
  master_only: False

logging:
#  loglevel: INFO
  loglevel: DEBUG
  logformat: default
    #  blacklist: ['elasticsearch', 'urllib3']
  blacklist: []

Example action

---
actions:

  10000-linux-syslog-allocation:
    action: allocation
    description: "Apply shard allocation filtering rules to
linux-syslog indexes"
    options:
      ignore_empty_list: True
      timeout_override: 300
      allocation_type: include
      key: datatype
      value: cold
    filters:
    - filtertype: closed
      exclude: True
    - filtertype: pattern
      kind: prefix
      value: linux-syslog-
    - filtertype: age
      source: field_stats
      field: '@timestamp'
      direction: older
      stats_result: min_value
      unit: days
      unit_count: 1

Result:

2025-01-13 14:43:44,519 WARNING            elasticsearch
log_request_fail:288  GET
https://foo.bar.redact:9200/.kibana-web-development_7,.tasks,linux-syslog-2024-14-avoidclosed-c2024-29c,linux-syslog-2025-02,idm-radius-auth-detail-2024.11.01,radius-auth-detail-2024.12.13,radius-log-2024.10.27,radius-log-2024.12.07,radius-log-2024.12.09,radius-log-2025.01.04,radius-log-2025.01.11,httpd-access-2023-14,httpd-access-2024-15,httpd-access-2024-52,linux-syslog-2024.11.03,linux-syslog-2024.12.12,linux-syslog8-2024.10.27,security-dhcp-2024.08.07,security-dhcp-2024.12.03,dns-queries-external-2024.11.01,dns-queries-external-2024.11.06,dns-queries-external-2024.11.11,dns-queries-external-2024.12.29,dns-queries-internal-2024.10.18,dns-queries-internal-2024.10.26,dns-queries-internal-2024.11.09,dns-queries-internal-2024.11.19,dns-rpz-2024.12,firepower-2024.07.17-avoidclosed-c2024.10.22c,firepower-2024.07.19-avoidclosed-c2024.11.06c,firepower-2024.08.05,firepower-2024.09.15-avoidclosed-c2024.12.18c,firepower-2024.10.28,firepower-2024.10.29,firepower-2024.11.14,firepower-2024.11.29,firepower-2025.01.04,firepower-2025.01.09,security-2024.08.31,security-2024.10.18,security-2024.11.17,security-2024.11.19,security-2024.11.28,security-2024.11.30,security-netflow-2024.12.22,security-vpn-2025.01.04,security-vpn-2025.01.12,linux-desktop-syslog-2024-07-avoidclosed-c2024-41c,linux-desktop-syslog-2024-23-avoidclosed-c2024-41c,server-eventlog-security-2024.10.14,server-eventlog-security-2024.12.08,server-eventlog-security-2024.12.25,server-firewall-prod-2024.07.28,server-firewall-prod-2024.10.26,server-firewall-prod-2024.11.09,server-firewall-prod-2024.11.15,server-horizon-2024.12,server-iis-2024.11.24,server-iis-2024.12.12,server-iis-2024.12.14,server-iis-2024.12.24,server-sql-2024.10.17,server-sql-2024.10.24,server-sql-2024.12.31,eventlog-2024.08.25-avoidclosed-c2024.11.28c,eventlog-2024.08.26-avoidclosed-c2024.11.28c,eventlog-2024.09.19-avoidclosed-c2024.12.19c,eventlog-2024.11.22,eventlog-2024.11.28,eventlog-2024.12.04,eventlog-2025.01.06,eventlog-2025.01.07,freenas-2024.11.16,freenas-2024.12.21,freenas-2024.12.24,access-2024.09.12,web-development-access-2024.12.02,access-2025.01.02,application-2024.10.28,application-2024.12.31,audit-2022-27,audit-2024-05,audit-2024-42,audit-2024-46/_stats/store,docs
[status:400 request:0.129s]
2025-01-13 14:43:44,519 DEBUG              elasticsearch
log_request_fail:308  > None
2025-01-13 14:43:44,519 DEBUG              elasticsearch
log_request_fail:313  <
{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index_uuid":"cTItOPS3SBiZs0jm5-U8xA","index":"httpd-access-2023-14"}],"type":"index_closed_exception","reason":"closed","index_uuid":"cTItOPS3SBiZs0jm5-U8xA","index":"httpd-access-2023-14"},"status":400}
2025-01-13 14:43:44,519 ERROR                curator.cli
   run:211  Failed to complete action: allocation.  <class 'KeyError'>: 'indices'

I did a DEBUG run of the action mentioned above with Curator 5.8.4 and from the output I can see that it never attempts to make a GET request for _stats/store,docs for the index httpd-access-2023-14, as 7.0.1 does. It seems like Curator 7.0.1 is not filtering out closed indices as it should by default and has been explicitly told to do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant