Log Drain Support for Sentry Logging

[AbhiPrasad]

You can now send structured logs to Sentry! See the docs here.

Currently you need to use one of our SDKs to send logs, but we're exploring the use case of Log Drains. This issue tracks supporting log drain providers.

If you're interested in log drain support, please leave a react on this GitHub issue. That will help us prioritize this accordingly. We created some child issues to track popular providers, but if there is a provider missing let us know and we'll open a new issue for it.

[sgarner]

I would like to see Sentry supported as a Log Sink for Vector.

This could be as simple as Sentry having a compatible HTTP endpoint for receiving logs:
https://vector.dev/docs/reference/configuration/sinks/http/

[Dhrumil-Sentry]

@sgarner how do you debug application code problems with logs from other sources? Just want to understand a few examples and your workflows today

[lucas-devx]

✅ Suggested Solution

To support generic log drain providers, expose a universal ingestion endpoint that accepts structured log payloads (JSON over HTTP) with provider-specific adapters for normalization.

🔧 Each adapter could map common log fields (e.g., timestamp, level, message, request) into Sentry's event schema.

📍Extend the existing ingestion pipeline or create a lightweight log-ingest service behind a /logdrain route.

[strokirk]

It's commonly useful to want see related logs from Apache, Nginx, other microservices, etc, when debugging an issue.

The Elastic Common Schema is amazingly useful to coordinate the naming of common log fields between multiple sources:
https://www.elastic.co/docs/reference/ecs

Anything with the same trace.id would be useful to correlate.

[AbhiPrasad]

@strokirk thanks for the feedback!

It's commonly useful to want see related logs from Apache, Nginx, other microservices, etc, when debugging an issue.

What log sinks/log drains are you using today for this?

The Elastic Common Schema is amazingly useful to coordinate the naming of common log fields between multiple sources:
https://www.elastic.co/docs/reference/ecs

We're relying on the OpenTelemetry (OTEL) semantic conventions (https://opentelemetry.io/docs/concepts/semantic-conventions/) when deciding naming of fields. OpenTelemetry is an open standard, so we are trying to follow it instead of any vendor specific naming strategies.

In this case, OTEL uses trace_id, which is queryable in Sentry logs:

[strokirk]

@AbhiPrasad Our current log sink is logging to a file in /var/log 🙃

I looked for a few minutes but couldn't find any reference to trace_id anywhere in the documentation you linked?

I think ECS is also an open standard, just as an addendum.

[AbhiPrasad]

I looked for a few minutes but couldn't find any reference to trace_id anywhere in the documentation you linked?

Ah for trace_id specifically, OTEL (and us at Sentry) considers this a first-class attribute that should almost always be sent with events. We optimize for this because we use the trace to connect your logs with other types of data (spans, errors, session replays, profiles, etc.).

See the OTEL spec here: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-traceid
And the Sentry spec here: https://develop.sentry.dev/sdk/telemetry/logs/#log-envelope-item-payload

Part of our challenge with supporting log drains/log sinks will be trying to make them trace connected.

I think ECS is also an open standard, just as an addendum.

ECS announced they would merge/donate attributes to OpenTelemetry: https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/. We are operating under the assumption that OpenTelemetry is the mains standard going forward.

Our current log sink is logging to a file in /var/log

Perhaps we can update sentry-cli to capture logs, and you can use the CLI to pipe stuff to Sentry. Does that seem like a reasonable solution?