[RORDEV-1414] ES node details audit reporting #1116

mgoworko · 2025-05-18T17:48:37Z

🚀New (ES) Added ES node name and cluster name in the ES cluster audit reporting

Summary by CodeRabbit

New Features
- Audit logs now include Elasticsearch node and cluster names for improved traceability.
- Audit log format updated with additional environment context fields.
Refactor
- Audit settings and related configuration types have been renamed and streamlined.
- Environment and node settings are now automatically extracted and passed internally, reducing manual configuration.
- Internal handling of Elasticsearch node name moved from external parameters to internal environment context.
Chores
- Updated internal version to 1.65.0-pre2.
Tests
- Test suites updated to support new audit log structure and environment context handling.
- Added reusable test utilities for node settings.
Documentation
- Audit log structure and settings documentation updated to reflect new fields and configuration changes.

# Conflicts: # core/src/test/scala/tech/beshu/ror/utils/TestsUtils.scala # gradle.properties

# Conflicts: # gradle.properties

...n/scala/tech/beshu/ror/accesscontrol/audit/AuditLogSerializerEnrichedWithEsNodeDetails.scala

core/src/main/scala/tech/beshu/ror/accesscontrol/audit/AuditingTool.scala

core/src/main/scala/tech/beshu/ror/boot/ReadonlyRest.scala

core/src/main/scala/tech/beshu/ror/configuration/ReadonlyRestEsConfig.scala

es67x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala

# Conflicts: # core/src/main/scala/tech/beshu/ror/accesscontrol/audit/AuditingTool.scala # core/src/main/scala/tech/beshu/ror/boot/ReadonlyRest.scala # core/src/test/scala/tech/beshu/ror/integration/AuditOutputFormatTests.scala # core/src/test/scala/tech/beshu/ror/unit/acl/logging/AuditingToolTests.scala # core/src/test/scala/tech/beshu/ror/unit/boot/ReadonlyRestStartingTests.scala # core/src/test/scala/tech/beshu/ror/unit/boot/RorIndexTest.scala # core/src/test/scala/tech/beshu/ror/utils/TestsUtils.scala # es67x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es67x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es70x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es70x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es710x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es710x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es711x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es711x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es714x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es714x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es716x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es716x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es717x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es717x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es72x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es72x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es73x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es73x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es74x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es74x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es77x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es77x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es78x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es78x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es79x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es79x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es80x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es80x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es810x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es810x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es811x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es811x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es812x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es812x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es813x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es813x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es814x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es814x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es815x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es815x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es816x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es816x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es81x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es81x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es82x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es82x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es83x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es83x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es84x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es84x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es85x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es85x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es87x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es87x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es88x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es88x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es89x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es89x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # es90x/src/main/scala/tech/beshu/ror/es/IndexLevelActionFilter.scala # es90x/src/main/scala/tech/beshu/ror/es/ReadonlyRestPlugin.scala # gradle.properties # integration-tests/src/test/scala/tech/beshu/ror/integration/suites/audit/LocalClusterAuditingToolsSuite.scala # integration-tests/src/test/scala/tech/beshu/ror/integration/suites/audit/RemoteClusterAuditingToolsSuite.scala # integration-tests/src/test/scala/tech/beshu/ror/integration/suites/base/BaseAuditingToolsSuite.scala

.../main/scala/tech/beshu/ror/audit/enrichers/AuditLogSerializerEnrichedWithEsNodeDetails.scala

coutoPL

Sorry, I forgot about this PR :(

coutoPL · 2025-06-13T17:42:42Z

audit/src/main/scala/tech/beshu/ror/audit/AuditLogSerializer.scala

@@ -19,5 +19,6 @@ package tech.beshu.ror.audit
 import org.json.JSONObject

 trait AuditLogSerializer {
-  def onResponse(responseContext: AuditResponseContext): Option[JSONObject]
+  def onResponse(responseContext: AuditResponseContext,


What about backward compatibility here? The existing custom serializers should work with the new approach too.

coutoPL · 2025-06-13T18:00:22Z

core/src/main/scala/tech/beshu/ror/accesscontrol/factory/decoders/AuditingSettingsDecoder.scala

      whenEnabled(c) {
        for {
          auditIndexTemplate <- decodeOptionalSetting[RorAuditIndexTemplate](c)("index_template", fallbackKey = "audit_index_template")
          customAuditSerializer <- decodeOptionalSetting[AuditLogSerializer](c)("serializer", fallbackKey = "audit_serializer")
          remoteAuditCluster <- decodeOptionalSetting[AuditCluster.RemoteAuditCluster](c)("cluster", fallbackKey = "audit_cluster")
-        } yield AuditingTool.Settings(
+          enableReportingEsNodeDetails <- c.downField("enable_reporting_es_node_details").as[Option[Boolean]]


Do we leave the toggle? Why?

coutoPL · 2025-06-13T18:01:00Z

core/src/main/scala/tech/beshu/ror/es/EsNodeSettings.scala

+ */
+package tech.beshu.ror.es
+
+case class EsNodeSettings(nodeName: String, clusterName: String)


coutoPL · 2025-06-13T18:03:35Z

es67x/src/main/scala/tech/beshu/ror/es/utils/EsEnvProvider.scala

-      esVersion = EsVersion(major = Version.CURRENT.major, minor = Version.CURRENT.minor, revision = Version.CURRENT.revision)
+      esVersion = EsVersion(major = Version.CURRENT.major, minor = Version.CURRENT.minor, revision = Version.CURRENT.revision),
+      esNodeSettings = EsNodeSettings(
+        nodeName = settings.get("node.name"),


Do you confirm these two will always be non-null? Even if the user doesn't set them? (I'm not sure if they are required settings)

coutoPL · 2025-06-13T18:05:37Z

gradle.properties

@@ -1,5 +1,5 @@
 publishedPluginVersion=1.64.2
-pluginVersion=1.65.0-pre1
+pluginVersion=1.65.0-pre2


coutoPL · 2025-06-13T18:09:26Z

...src/test/scala/tech/beshu/ror/integration/suites/audit/RemoteClusterAuditingToolsSuite.scala

@@ -61,4 +62,10 @@ class RemoteClusterAuditingToolsSuite
  override protected def baseRorConfig: String = resolvedRorConfigFile.contentAsString

  override protected def baseAuditDataStreamName: Option[String] = Option.when(isDataStreamSupported)("audit_data_stream")
+
+  // Adding the ES cluster fields is enabled in the /cluster_auditing_tools/readonlyrest.yml config file (`DefaultAuditLogSerializerV2` is used)
+  override def assertForEveryAuditEntry(entry: JSON): Unit = {


I'm not sure if this test covers this use case too, but I'd be nice to test the scenario as above:

1. The user starts ES with ROR with `DefaultAuditLogSerializerV1` configured 2. The user generates some traffic to create some audit documents 3. The user reconfigures ROR to use `DefaultAuditLogSerializerV2` and reloads config 4. The user generates some traffic to create some audit documents

The result - there should be no problem with the scenario above.

mgoworko added 14 commits April 12, 2025 12:35

qs

911872b

qs

23c9fd3

qs

f424f8e

Merge remote-tracking branch 'origin/develop' into 1414

0e3d102

qs

1ef3add

qs

90708dc

qs

d17387a

Merge remote-tracking branch 'origin/develop' into 1414

d7d1709

# Conflicts: # core/src/test/scala/tech/beshu/ror/utils/TestsUtils.scala # gradle.properties

qs

85c7115

Merge remote-tracking branch 'origin/develop' into 1414

eb74b87

# Conflicts: # gradle.properties

qs

d207a7f

qs

2fe43a2

qs

37efe2d

qs

65f1e9b

This comment was marked as outdated.

Sign in to view

mgoworko requested a review from coutoPL May 19, 2025 16:46

coutoPL requested changes May 19, 2025

View reviewed changes

mgoworko added 6 commits May 23, 2025 23:59

qs

b59a83b

merge fixes

eaf09fc

qs

2a74995

qs

7afc0b9

qs

50062d9

mgoworko force-pushed the RORDEV-1414-es-node-details-reporting branch from 1169b5a to 50062d9 Compare May 25, 2025 16:34

qs

72b9af7

This comment was marked as outdated.

Sign in to view

mgoworko requested a review from coutoPL May 25, 2025 16:59

coutoPL reviewed May 25, 2025

View reviewed changes

.../main/scala/tech/beshu/ror/audit/enrichers/AuditLogSerializerEnrichedWithEsNodeDetails.scala Outdated Show resolved Hide resolved

review changes and fix Scala 2.11 cross compile

01cceae

This comment was marked as outdated.

Sign in to view

versioned QueryAuditLogSerializer and fixed unit tests

809c1b7

This comment was marked as outdated.

Sign in to view

mgoworko requested a review from coutoPL May 31, 2025 10:37

coutoPL reviewed Jun 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[RORDEV-1414] ES node details audit reporting #1116

[RORDEV-1414] ES node details audit reporting #1116

Uh oh!

mgoworko commented May 18, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

coutoPL left a comment

Uh oh!

coutoPL Jun 13, 2025

Uh oh!

coutoPL Jun 13, 2025

Uh oh!

coutoPL Jun 13, 2025

Uh oh!

coutoPL Jun 13, 2025

Uh oh!

coutoPL Jun 13, 2025

Uh oh!

coutoPL Jun 13, 2025

Uh oh!

Uh oh!

[RORDEV-1414] ES node details audit reporting #1116

Are you sure you want to change the base?

[RORDEV-1414] ES node details audit reporting #1116

Uh oh!

Conversation

mgoworko commented May 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

coutoPL left a comment

Choose a reason for hiding this comment

Uh oh!

coutoPL Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

coutoPL Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

coutoPL Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

coutoPL Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

coutoPL Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

coutoPL Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mgoworko commented May 18, 2025 •

edited

Loading