Open in ChatGPTOpen in Claude

Trace-log correlation

OBI automatically enriches application logs with trace context by injecting trace_id and span_id fields at the kernel level. This links your logs directly to distributed traces in Coralogix without requiring any code changes, enabling you to navigate from a trace span to the exact log entries produced during that operation.

JSON log objects receive structured fields. From OBI v0.11.0, plain-text logs also receive trace context, as space-separated key=value fields — see Plain-text log enrichment.

Prerequisites

• OBI deployed via the Coralogix Helm chart.
• Linux kernel 6.0 or later (the log enrichment mechanism requires a UBUF-type iov_iter for overwriting user memory).
• CAP_SYS_ADMIN capability and permission to use bpf_probe_write_user.
• Kernel security lockdown mode set to [none] (verify with cat /sys/kernel/security/lockdown).
• Target application writes logs as JSON objects or plain text. Before OBI v0.11.0, only JSON was enriched and plain-text logs passed through unmodified.
• BPF filesystem mounted at /sys/fs/bpf.

How it works

1. Trace context capture: OBI records trace IDs and span IDs during traced HTTP/gRPC operations.
2. Log interception: Kernel-level eBPF probes capture write system calls from the instrumented application.
3. Field injection: OBI adds trace_id and span_id to each log line — as JSON members in a JSON object, or as key=value tokens in plain text — and writes the enriched line itself.
4. Original-line suppression: OBI overwrites the application's original buffer with NULL bytes so the container runtime doesn't also capture the un-enriched copy. This leaves a placeholder line in the container log that your pipeline should drop — see Filter the suppressed placeholder lines.
5. Pipeline passthrough: The enriched logs continue through your existing log shipping pipeline (Fluent Bit, OpenTelemetry Collector, or any other forwarder) to Coralogix.

For example, an application log entry like:

{ "level": "info", "message": "Request processed", "duration_ms": 42 }

Becomes:

{ "level": "info", "message": "Request processed", "duration_ms": 42, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" }

OBI only fills in fields that aren't already there. If your logger or SDK already injects trace context — for example through the Python LoggingInstrumentor — those values are preserved. For services that OBI detects as exporting OpenTelemetry traces themselves, only the trace field is injected, because a span ID generated by OBI wouldn't match the span the SDK emits.

Filter the suppressed placeholder lines

Every enriched write leaves one placeholder line in the container log: the application's original bytes, overwritten with NULL characters and terminated with a newline. OBI writes the enriched line separately, so suppressing the original is what stops the un-enriched duplicate from reaching your pipeline.

Drop the placeholders downstream by filtering lines that match ^[\x00\s]*$.

CRI and Docker JSON log envelopes serialize NULL as the \u0000 escape. The configurations below decode the JSON envelope before filtering, so the pattern matches real NULL bytes.

OpenTelemetry Collector

The container operator handles both CRI and Docker JSON formats and exposes the line in body:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    start_at: end
    operators:
      - type: container
      - type: filter
        expr: 'body matches "^[\\x00\\s]*$"'

Fluent Bit

[INPUT]
    Name              tail
    Path              /var/log/pods/*/*/*.log
    multiline.parser  cri
    Tag               kube.*

[FILTER]
    Name    grep
    Match   *
    Exclude log ^[\x00\s]*$

Docker JSON driver

For the legacy Docker JSON log driver, parse the envelope first:

receivers:
  filelog:
    include: [/var/lib/docker/containers/*/*-json.log]
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: filter
        # Bracket access — `log` collides with expr-lang's math `log`.
        expr: 'attributes["log"] matches "^[\\x00\\s]*$"'

Writes larger than 8 KiB are only partially suppressed, so their leaked tail doesn't match this filter — see Limitations.

Plain-text log enrichment

New in v0.11.0

From OBI v0.11.0, plain-text log lines also receive trace context, as lowercase fixed-width key=value tokens:

request failed trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7

Enabled by default — check your log parsers before upgrading

Plain-text enrichment is on by default for every service selected by the log enricher. Non-JSON writes that earlier versions re-emitted unchanged now carry the extra fields, which can break downstream parsers for structured non-JSON formats. To keep the previous behavior, set plain_text.enabled: false before upgrading — JSON enrichment continues to work.

The key=value format targets unstructured and free-form text. It isn't a native encoding for every structured non-JSON format, so if your logs use a strict non-JSON layout, disable plain-text enrichment rather than letting the fields land mid-record.

Newline-delimited JSON is treated as structured JSON, not plain text: OBI enriches each JSON object record independently and leaves valid non-object records byte-identical.

Plain-text configuration

opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      field_names:
        trace_id: trace_id
        span_id: span_id
      plain_text:
        enabled: true
        placement: suffix
        multiline: first_line
      services:
        - service:
            - open_ports: '8080'

Parameter	Description	Values
field_names.trace_id	Field name used for the trace ID, in both JSON and plain-text output	Default trace_id. Must be non-empty, distinct from span_id, and free of whitespace, =, and control characters.
field_names.span_id	Field name used for the span ID, in both JSON and plain-text output	Default span_id. Same constraints as above.
plain_text.enabled	Whether non-JSON writes receive trace context	true (default) or false
plain_text.placement	Where the fields go on the line	suffix (default) or prefix
plain_text.multiline	Which non-empty physical lines within one intercepted write are annotated	first_line (default), last_line, or each_line

Field names are used both to recognize existing trace context and to inject what's missing, so renaming them also changes which existing fields OBI treats as already populated.

Empty lines and LF or CRLF line endings are preserved, and a write that isn't newline-terminated stays unterminated. OBI doesn't buffer writes or reconstruct logical multiline events across separate write calls, so multiline applies only within a single intercepted write.

If you run OBI with Configuration v2, the same settings live under extensions.obi.correlation.log_trace_annotation.

Runtime-specific stdout buffering

New in v0.8.0

OBI reads the trace context at the moment the write syscall fires. If your runtime buffers stdout and flushes asynchronously — on a different thread or after the request handler returns — the trace context is gone by the time the syscall reaches the enricher, and the log line is not enriched.

Runtime	Default stdout behavior	Works out of the box?
Go	fmt.Println calls write synchronously on the goroutine	Yes
Node.js	process.stdout.write() is synchronous	Yes
Java	System.out.println() flushes immediately by default	Yes
Ruby	puts and STDOUT.syswrite issue write synchronously on the request thread	Yes
Python	Block-buffered when stdout is not a TTY (for example, in Docker)	No — set PYTHONUNBUFFERED=1
.NET	Console.Out is block-buffered when stdout is a pipe	No — see .NET

Python

In Docker and Kubernetes, Python buffers stdout because it is not attached to a TTY. Set the PYTHONUNBUFFERED=1 environment variable on the container to force line-buffered output.

.NET

.NET's Console.Out wraps a StreamWriter with AutoFlush = false. When stdout is a pipe, writes accumulate until the buffer fills (4 KB) or the writer is flushed explicitly — at which point the write syscall fires from a finalizer thread or a later request that no longer carries the original trace context.

Configure auto-flush at application startup:

var stdout = new StreamWriter(Console.OpenStandardOutput()) { AutoFlush = true };
Console.SetOut(stdout);

Microsoft.Extensions.Logging.AddConsole() (the default ASP.NET Core console logger) does not work even with AutoFlush set, because it queues entries through an internal Channel and drains them on a dedicated writer thread that has no trace context.

Logging frameworks that work correctly:

• Console.WriteLine with AutoFlush = true — synchronous on the calling thread.
• Serilog WriteTo.Console() — synchronous by default.
• NLog targets/ColoredConsole with queueLimit=0 — synchronous mode.

There is no .NET equivalent to Python's PYTHONUNBUFFERED=1 environment variable.

Enable trace-log correlation

Step 1: Verify kernel requirements

Confirm your nodes meet the kernel version and capability requirements:

# Check kernel version (must be 6.0+)
uname -r

# Check lockdown mode (must be [none])
cat /sys/kernel/security/lockdown

# Verify BPF filesystem
mount | grep bpf

Step 2: Choose your log format

From OBI v0.11.0, both JSON objects and plain text are enriched, so no format change is required. JSON is still the better target: it produces structured trace_id and span_id fields that Coralogix parses directly, whereas plain text produces key=value tokens that your parsing pipeline has to pick up.

To output JSON, configure your logging framework accordingly:

• Python: Use a custom JSONFormatter with the logging module.
• Go: Use the zap library with its default JSON encoder.
• Java: Use Logback with LogstashEncoder.
• Node.js: Use the pino package.

Verify your application's log output is valid JSON:

cat /path/to/app.log | jq empty

If your application emits a structured non-JSON format (for example logfmt or a fixed-column layout), decide before upgrading whether the appended key=value fields are safe for your parsers — see Plain-text log enrichment.

Step 3: Configure the Helm chart

Enable trace-log correlation in your Coralogix Helm chart values:

opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      services:
        - service:
            - open_ports: '8080'

Replace 8080 with the port your application listens on. Add multiple entries to enrich logs from multiple services.

Trace export must also be enabled (it is by default in the Coralogix Helm chart). If you have customized your configuration, verify that otel_traces_export has a valid endpoint:

opentelemetry-ebpf-instrumentation:
  otel_traces_export:
    endpoint: http://otel-collector:4318/v1/traces

Step 4: Apply the configuration

Upgrade your Helm release to apply the changes:

Step 5: Verify enrichment

After the pods restart, check your application's log output for the injected fields:

kubectl logs <your-app-pod> | jq 'select(.trace_id != null)' | head -5

In Coralogix, navigate to Logs Explorer and filter for logs containing trace_id. Select a log entry and use the trace link to navigate directly to the associated trace in the Spans Explorer.

Enricher configuration options

Fine-tune the log enricher behavior using these optional parameters:

Parameter	Description	Default
cache_ttl	File descriptor cache lifetime	30 minutes
cache_size	Maximum number of cached file descriptors	Implementation-defined
async_writer_workers	Number of asynchronous writer worker shards	Implementation-defined
async_writer_channel_len	Queue capacity per worker shard	Implementation-defined

For the field-name and plain-text parameters, see Plain-text configuration.

Example with custom cache settings:

opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      cache_ttl: 15m
      cache_size: 1000
      services:
        - service:
            - open_ports: '8080'

Limitations

Changed in v0.11.0

• Per-write cap of 8 KiB: Only the first 8 KiB of a single write or writev is enriched and suppressed. Bytes past that reach the container log un-enriched, and they don't match the placeholder filter described in Filter the suppressed placeholder lines.
• Plain-text format is fixed: Plain-text enrichment emits space-separated key=value tokens only. Structured non-JSON formats that need a format-specific representation aren't supported — disable plain-text enrichment for those services. Before OBI v0.11.0, plain-text logs were never modified.
• No cross-write reconstruction: The multiline setting selects lines within a single intercepted write call. OBI doesn't buffer writes or reassemble logical multiline events that span separate writes.
• Span window: Logs are enriched only during active span windows. Logs written outside of a traced request do not receive trace context.
• Cache scope: File descriptors are cached with a configurable TTL (default 30 minutes). Extremely short-lived processes may not benefit from caching.
• Async not supported: Applications that use asynchronous write primitives are not yet supported.
• Synchronous writes required: Logs must be written on the request-handling thread before the handler returns. Buffered or queued logging (for example, Python without PYTHONUNBUFFERED, or .NET with Microsoft.Extensions.Logging.AddConsole()) is not enriched. See Runtime-specific stdout buffering.
• Kernel version: Requires Linux kernel 6.0+, which is newer than the 5.8+ required for basic OBI functionality.

Troubleshooting

Logs do not contain trace_id or span_id

1. Confirm kernel version is 6.0+: uname -r.
2. Check kernel lockdown mode: cat /sys/kernel/security/lockdown (must show [none]).
3. Verify the log_enricher.services section matches your application's port.
4. Ensure both trace export and log enricher are configured.
5. If your logs are JSON, verify they're valid: cat app.log | jq empty. A malformed object isn't enriched as JSON.
6. If your logs are plain text, confirm plain_text.enabled hasn't been set to false, and that the field names you're searching for match field_names.

Plain-text fields appear where a parser does not expect them

If a downstream parser started failing after upgrading to OBI v0.11.0, plain-text enrichment is appending trace_id and span_id to lines it previously left alone. Either move the fields with plain_text.placement: prefix, or set plain_text.enabled: false to restore the earlier behavior for non-JSON writes. JSON enrichment is unaffected either way.

Intermittent enrichment

If only some log entries are enriched, verify that the missing entries are written during an active traced request. Logs written outside of a traced span (for example, background tasks or startup logs) are not enriched.