OBI Distributed Tracing

Overview

OBI enables distributed tracing for applications, with certain limitations related to kernel versions and system configuration.

Tracing works by propagating the W3C traceparent header automatically—no manual configuration is required. OBI reads incoming trace headers, tracks execution, and injects outgoing traceparent headers into HTTP or gRPC requests. If the application already includes a traceparent, OBI uses that value instead of generating a new one. If no header is present, OBI creates a valid one following the W3C standard.

How trace context propagation works

OBI uses two primary methods for propagating trace context:

1. Network-level injection: Inserts trace headers at the packet level.
2. Library-level injection for Go: Writes directly into the application’s memory.

Depending on the language and system capabilities, OBI uses one or both methods. Some approaches rely on Linux kernel features and permissions.

Coralogix enables both trace context propagation methods by default using our helm installation. To disable context propagation, use one of the following methods:

1. Update your helm chart to use the following configuration:

   opentelemetry-ebpf-instrumentation:
     contextPropagation:
       enabled: false
   

2. Set the following environment variable:

   OTEL_EBPF_BPF_CONTEXT_PROPAGATION=disabled
   

These methods ensure compatibility with any OpenTelemetry-based tracing library. OBI modifies outgoing HTTP headers and, if encrypted traffic (HTTPS) is used, injects at the TCP/IP level. Note that encrypted traffic tracing only works between OBI-instrumented services and cannot pass through L7 proxies or load balancers.

• HTTP: Compatible with OpenTelemetry SDKs.
• HTTPS/TLS: Trace info added at TCP/IP level.
• gRPC/HTTP2: Not yet supported.

No privileged mode is required for network-level propagation.

Kubernetes deployment guidelines

For Kubernetes, OBI is best deployed as a DaemonSet with:

• hostNetwork: true
• Volume mount /sys/fs/cgroup
• CAP_NET_ADMIN capability

Example deployment:

spec:
  serviceAccount: obi
  hostPID: true
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
  - name: obi
    resources:
      limits:
        memory: 120Mi
    terminationMessagePolicy: FallbackToLogsOnError
    image: "obi:latest"
    imagePullPolicy: "Always"
    command: [ "/obi", "--config=/config/obi-config.yml" ]
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "http://otelcol:4318"
      - name: OBI_KUBE_METADATA_ENABLE
        value: "autodetect"
    securityContext:
      runAsUser: 0
      readOnlyRootFilesystem: true
      capabilities:
        add:
          - BPF
          - SYS_PTRACE
          - NET_RAW
          - CHECKPOINT_RESTORE
          - DAC_READ_SEARCH
          - PERFMON
          - NET_ADMIN
    volumeMounts:
      - name: cgroup
        mountPath: /sys/fs/cgroup # Required so OBI can monitor newly created sockets for outgoing requests.
      - mountPath: /config
        name: obi-config
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists
  volumes:
  - name: obi-config
    configMap:
      name: obi-config
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup

Without the /sys/fs/cgroup mount, some trace context may be lost because OBI relies on this to detect new sockets and propagate context reliably.

Kernel version requirements

For network-level context propagation, Linux kernel version 5.17+ is required. Some distributions, like RHEL 9.2, may include backported support. To bypass version checks:

OTEL_EBPF_OVERRIDE_BPF_LOOP_ENABLED=true

Go application support with memory injection

OBI also supports context propagation for Go applications using eBPF’s bpf_probe_write_user helper. This method enables propagation for HTTP, HTTPS, HTTP2, and gRPC, but it requires the container to:

• Run in privileged mode or
• Have CAP_SYS_ADMIN capability.

Kernel lockdown mode impact

Kernel integrity lockdown, often active with Secure Boot, blocks bpf_probe_write_user. OBI detects this automatically. To manually verify:

cat /sys/kernel/security/lockdown

If this shows anything other than [none], memory injection won’t work.

Container deployment for Go applications

To enable memory-level propagation in containers, mount /sys/kernel/security so OBI can read kernel lockdown mode:

services:
  obi:
    image: coralogix/obi:latest
    environment:
      OBI_CONFIG_PATH: "/configs/obi-config.yml"
    volumes:
      - /sys/kernel/security:/sys/kernel/security # Needed so OBI can verify if the system is running in kernel lockdown mode.
      - /sys/fs/cgroup:/sys/fs/cgroup # Required for tracking socket creation and ensuring proper trace propagation.

Without these mounts, OBI assumes the kernel is not in lockdown mode. It may not function as intended for memory injection.