Support ECS nested format

[oyviaase]

Description

Change the structure of ECS formated logs to use nested json instead of a flat json structure. To improve the compatability with most log aggregation tools, since not all of them have implemented the abstraction done by elasticsearch to interpret 'flat' files to nested. And this reduces the workload of the log collectors as they will not need to parse and rewrite the logs to such a large extent.

There's a discussion in the spring-boot repository discussing flat vs nested that could be of interest in regards to this spring-projects/spring-boot#45063 (comment)

Implementation ideas

No response

[quarkus-bot]

/cc @dmlloyd (logging)

[dmlloyd]

/cc @andreaTP

[andreaTP]

Hi @dmlloyd 👋 !

When I added basic Json logging to plain Quarkus I had in mind only simple transformations, I'm not sure we want to add options to such a performance critical component at this point in time, but happy to hear more opinions.

Is using the quarkus-logging-json extension a possible workaround here? ( e.g. )

cc. @shawkins

[shawkins]

@andreaTP @oyviaase can an example of what keycloak / quarkus is producing today be shown along with what would be ideal?

keycloak/keycloak#31860 - is somewhat related from the Keycloak side. Depending on the exactly what the message is, we'd need to make sure it gets json formatting at a higher level - and is tracked as such, otherwise we could waste a lot of time lower down in the logging framework figuring out what message is nestable json.

[oyviaase]

I've done an attempt at converting the example provided in the keycloak loggin documentation the current flat ESC formated logs example:

{"@timestamp":"2025-02-03T14:53:22.539484211+01:00","event.sequence":9608,"log.logger":"io.quarkus","log.level":"INFO","message":"Keycloak 999.0.0-SNAPSHOT on JVM (powered by Quarkus 3.17.8) started in 4.615s. Listening on: http://0.0.0.0:8080","process.thread.name":"main","process.thread.id":1,"mdc":{},"ndc":"","host.hostname":"host-name","process.name":"/usr/lib/jvm/jdk-21.0.3+9/bin/java","process.pid":77561,"data_stream.type":"logs","ecs.version":"1.12.2","service.environment":"prod","service.name":"Keycloak","service.version":"999.0.0-SNAPSHOT"}

How it should formated if nested json is used for ecs:

{"@timestamp":"2025-02-03T14:53:22.539484211+01:00","event": {"sequence":9608},"log":{"logger":"io.quarkus","level":"INFO"},"message":"Keycloak 999.0.0-SNAPSHOT on JVM (powered by Quarkus 3.17.8) started in 4.615s. Listening on: http://0.0.0.0:8080","process":{"thread":{"name":"main","id":1},{"name":"/usr/lib/jvm/jdk-21.0.3+9/bin/java","pid":77561}},"mdc":{},"ndc":"","host":{"hostname":"host-name"},"data_stream":{"type":"logs"},"ecs":{"version":"1.12.2"},"service":{"environment":"prod","name":"Keycloak","version":"999.0.0-SNAPSHOT"}}

Keep in mind that I might have screwed up the nested formatting some where.

[shawkins]

Thank you for the example @oyviaase

@andreaTP https://github.com/quarkiverse/quarkus-logging-json looks promising. For keycloak/keycloak#31860 that would allow us to log the json structure, rather than just nesting the .toString.

A couple of other things to consider.

Quarkus extensions cannot be dymanically added to Keycloak - @mabartos is working on a project to simplify creating your own custom builds of Keycloak for that.

Also it looks like the JsonProviders work co-operatively, so one would need to completely re-implement the standard logging logic to have that be heirarchical (neither https://github.com/quarkiverse/quarkus-logging-json/blob/456df79bbd5676dd113e1114e1635bb57294ae84/runtime/src/main/java/io/quarkiverse/loggingjson/LoggingJsonRecorder.java#L92 nor the default match what is desired here).

Alternatively the standard JsonFormatter is capable of creating nested structures, but the logic that sits on top of that (jboss logmanager StructuredFormatter https://github.com/jboss-logging/jboss-logmanager/blob/7ade35fda200f01cdade3b1990bd7299a7108ff3/src/main/java/org/jboss/logmanager/formatters/StructuredFormatter.java#L205) assumes a flat record. We're just creating simple mappings to keys like "event.sequence" in the quarkus ECS logic. If there were a standard hierarchical StructuredFormatter (in quarkus or jboss-logmanager), that should address this case. Handling excluded fields would introduce more complexity to prevent empty nested objects from being written.