[Security Solution] Incorrect privilege warning when creating Custom Rule

[pborgonovi]

Description:

When creating a custom rule with an index pattern that matches the indices already included in the user’s read and write permissions, a warning message incorrectly states that the rule may not have the required read privileges for certain indices.

Kibana/Elasticsearch Stack version:

8.18 BC3

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Detection Rules RBAC

Pre requisites:

1. Have a custom role with read and write access to the following indices

alerts-security.alerts-default, apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, 
packetbeat-*, traces-apm*, winlogbeat-*, -*elastic-cloud-logs-*,
metrics-*, security_solution-*.vulnerability_latest, logs-cloud_security_posture.vulnerabilities_latest-default,
logs-cloud_security_posture.findings_latest-default, security_solution-*.misconfiguration_latest,
.kibana-event-log-*, .items-default, .lists-default, .alerts-security.alerts-default

2. Have an user assigned with this role

Steps to reproduce:

1. Create a new custom rule with the following index pattern:

apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, packetbeat-*, traces-apm*, winlogbeat-*,
-*elastic-cloud-logs-*

2. Observe the warning message: “This rule may not have the required read privileges to the following index patterns: [”-elastic-cloud-logs-”,“apm--transaction”,“endgame-”,“filebeat-”,“packetbeat-”,“traces-apm”,“winlogbeat-*”]”

Current behavior:

A false warning message appears, incorrectly suggesting that the user may not have the required privileges for certain index patterns, even though they are explicitly listed in the access role.

Expected behavior:

Since the user already has read and write access to these indices, no warning should be displayed.

Screenshots:

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[yctercero]

@rylnd could you do a quick check to see if this is desired behavior after the changes you made in #177658

[rylnd]

@pborgonovi can you please share the official permissions of the user in question?

GET _security/user/_privileges

I ask because in trying to reproduce this I noticed that copy/pasting index patterns into the "Create Roles" UI lead to index patterns with extra whitespace. This did cause the rule execution to (correctly) say that my user did not have permission, but ultimately this was due to an issue with the role, and not our permissions logic. The fact that you have a seemingly empty index pattern in your screenshot leads me to believe this might be the case here.

It's interesting that auditbeat-* is not mentioned in the warning message, but filebeat-* and packetbeat-* are. I don't know what this means, but wanted to call it out.

I suspected that this might be a bug where we incorrectly reported lack of read access to indices which do not exist, but I was able to disprove this with the following example:

1. Fresh kibana instance

2. Create a user with read/write permissions to the default security index pattern:

    "-*elastic-cloud-logs-*",
    ".alerts-security.alerts-default",
    "apm-*-transaction*",
    "auditbeat-*",
    "endgame-*",
    "filebeat-*",
    "packetbeat-*",
    "winlogbeat-*"
   

3. Create a query rule with the default index pattern

4. Run it, observe a lack of warnings

5. Add some data, e.g. auditbeat data, and run the rule again

With the above script, I did not see any read errors, which proves that we are not checking privileges for non-existent indices. The presence of auditbeat data, and my observation that we do check for its read access during rule execution, verified to me that this is working as expected.

@pborgonovi one more note that might help diagnose what's going on here: toggling a rule on/off does not reset its API token (so the rule would continue operating with a "previous" set of privileges); you need to edit the rule and perform a substantive change for new privileges to be picked up.

[pborgonovi]

Hey @rylnd

@pborgonovi can you please share the official permissions of the user in question?

There we go:

{
  "cluster": [
    "all",
    "manage"
  ],
  "global": [],
  "indices": [
    {
      "names": [
        "-*elastic-cloud-logs-*",
        ".alerts-security.alerts-default",
        ".alerts-security.alerts-default,apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,traces-apm*,winlogbeat-*,-*elastic-cloud-logs-*",
        ".items-default",
        ".kibana-event-log-*",
        ".lists-default",
        "apm-*-transaction*",
        "auditbeat-*",
        "endgame-*",
        "filebeat-*",
        "logs-*",
        "logs-cloud_security_posture.findings_latest-default,security_solution-*.misconfiguration_latest",
        "metrics-*",
        "packetbeat-*",
        "security_solution-*.vulnerability_latest,logs-cloud_security_posture.vulnerabilities_latest-default",
        "traces-apm*",
        "winlogbeat-*"
      ],
      "privileges": [
        "read",
        "write"
      ],
      "allow_restricted_indices": false
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": [
        "all"
      ],
      "resources": [
        "*"
      ]
    }
  ],
  "run_as": []
}

---

I reproduced the same steps you mentioned above:

1. Fresh kibana instance

Created a fresh Cloud deployment

2. Create a user with read/write permissions to the default security index pattern:

 "-*elastic-cloud-logs-*",
 ".alerts-security.alerts-default",
 "apm-*-transaction*",
 "auditbeat-*",
 "endgame-*",
 "filebeat-*",
 "packetbeat-*",
 "winlogbeat-*"

{
  "cluster": [
    "all"
  ],
  "global": [],
  "indices": [
    {
      "names": [
        ".alerts-security.alerts-default,apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,traces-apm*,winlogbeat-*,-*elastic-cloud-logs-*"
      ],
      "privileges": [
        "read",
        "write"
      ],
      "allow_restricted_indices": false
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": [
        "all"
      ],
      "resources": [
        "*"
      ]
    }
  ],
  "run_as": []
}

3. Create a query rule with the default index pattern

4. Run it, observe a lack of warnings

5. Add some data, e.g. auditbeat data, and run the rule again

Ingested auditbeat data:

Ran the rule again:

Am I missing something here in the steps?

[rylnd]

Am I missing something here in the steps?

@pborgonovi did you see a failure in step 4, before adding auditbeat data? Was this performed on a fresh install?

@pborgonovi I've only tested this with a local dev build on main; can you confirm where you're reproducing this behavior? Is it only on serverless, or both serverless and cloud? Is it reproducible locally?

[pborgonovi]

Hey @rylnd

@pborgonovi did you see a failure in step 4, before adding auditbeat data? Was this performed on a fresh install?

No failures in step 4. It was performed on a very fresh install.

@pborgonovi I've only tested this with a local dev build on main; can you confirm where you're reproducing this behavior? Is it only on serverless, or both serverless and cloud? Is it reproducible locally?

I've been testing on Cloud with latest BC but can try to reproduce it locally on main. (get back to you soon)

In the meanwhile, I also tried the following:

1. Created a role with the default security index patterns but this time entered one by one instead of selecting the security data view:

   -*elastic-cloud-logs-*",
        ".alerts-security.alerts-default",
        "apm-*-transaction*",
        "auditbeat-*",
        "endgame-*",
        "filebeat-*",
        "packetbeat-*",
        "winlogbeat-*"

2. Created a rule query with these index patterns and ran ir:

3. Ingested auditbeat data and ran the rule again:

[pborgonovi]

@rylnd

Local test on main branch:

1. Created a custom role with access to the following indices:

2. Created a custom rule query to these index patterns and ran it:

3. Ingested auditbeat data:

4. Ran the rule again:

It's just fine on main, I'm wondering what is different

[rylnd]

but this time entered one by one instead of selecting the security data view:

@pborgonovi are you implying that this is how the original privileges were created? Presumably you selected the long index pattern from the Index Privileges picker in the Role UI, right? If so, this does appear to produce a different set of privileges than selecting those patterns individually. By selecting that long pattern I get the following privileges:

GET _security/user/_privileges
# Response => 
{
  "indices": [
    {
      "names": [
        ".alerts-security.alerts-default,apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,traces-apm*,winlogbeat-*,-*elastic-cloud-logs-*"
      ],
      "privileges": [
        "read",
        "write"
      ],
      "allow_restricted_indices": false
    }
  ]
}

Compared to entering patterns individually:

GET _security/user/_privileges
# Response =>
{
  "indices": [
    {
      "names": [
        "-*elastic-cloud-logs-*",
        ".alerts-security.alerts-default",
        "apm-*-transaction*",
        "auditbeat-*",
        "endgame-*",
        "filebeat-*",
        "packetbeat-*",
        "winlogbeat-*"
      ],
      "privileges": [
        "read",
        "write"
      ],
      "allow_restricted_indices": false
    }
  ]
}

This is important because of how we check the user's privileges. Our logic can be summarized as:

1. Check if any indices exist in the specified pattern
   2a. If none exist, display warning about "no indexes matching" (you can see this in your last test above, before adding auditbeat data)
   2b. If some exist, ask Elasticsearch whether the user can read those indices
   3b. Report back if they cannot

The reason the form of these privileges seems to be important is because of how we perform the check in 2b. It's effectively:

POST _security/user/_has_privileges
{
  "index": [
    {
      "names": [
        "auditbeat-*",
        "filebeat-*"
      ],
      "privileges": [
        "read"
      ]
    }
  ]
}

Where auditbeat-* and filebeat-* represent extant index patterns being checked.

In the multi-pattern role, with multiple index names/patterns, the above _has_privileges call returns true for those patterns. In the single-pattern role, the same call reports false for both.

I'm not yet sure if this is a bug in the privileges API, whether this is due to our misuse of that API, or whether one of those sets of privileges is incorrect. We'll need to do some investigation to figure out which is the case.

[pborgonovi]

@rylnd

@pborgonovi are you implying that this is how the original privileges were created? Presumably you selected the long index pattern from the Index Privileges picker in the Role UI, right? If so, this does appear to produce a different set of privileges than selecting those patterns individually.

Yeap, this is the difference I noted too. After going through these steps with you I can understand it's likely to be an issue on privilegies side.

Thanks for the investigation.

[rylnd]

@pborgonovi it looks like this might be the relevant issue here (found via https://github.com/elastic/sdh-kibana/issues/3022). As is presented there, it's arguably a bug that we allow users to create these types of ineffective privileges, but the issue is more central to Kibana than the Detection Engine.

If the "correct" privileges work as expected (which they appear to), I could see this being added to a "known pitfalls" section of documentation. We could potentially also perform a check during rule execution to identify this; it should be as simple as checking whether their index privileges' names contain a ,. However, this would be work performed during every rule execution, so not a negligible change either.

[pborgonovi]

@rylnd That makes sense.

@yctercero how should we handle this with documentation?

[rylnd]

cc @nastasha-solomon as she may have an idea of where to slot this in documentation

[yctercero]

@nastasha-solomon do we have a troubleshooting section to add this to?

[nastasha-solomon]

Hey, thanks for the ping. There is this troubleshooting page that's generally used for detection rule issues. I'd just need to add a new section for custom rules.

If someone can file a doc issue in the docs-content repo, I can go ahead and add the task to a future sprint. When setting up the doc issue, please provide a brief summary of problem, why and when it's happening, and what users can do to address it. Thank you!

[rylnd]

@nastasha-solomon I created elastic/docs-content#599 👍