Exposed metrics

Open in ChatGPTOpen in Claude

The following tables describe the metrics exported by OBI in both OpenTelemetry (OTEL) and Prometheus formats, for use within Coralogix observability workflows.

Application metrics

Family	Name (OTEL)	Name (Prometheus)	Type	Unit	Description
Application	http.client.request.duration	http_client_request_duration_seconds	Histogram	seconds	Time taken for outbound HTTP calls from client applications
Application	http.client.request.body.size	http_client_request_body_size_bytes	Histogram	bytes	Size of HTTP request payloads sent by clients
Application	http.client.response.body.size	http_client_response_body_size_bytes	Histogram	bytes	Size of HTTP responses received by clients
Application	http.server.request.duration	http_server_request_duration_seconds	Histogram	seconds	Time taken to process inbound HTTP requests on servers
Application	http.server.request.body.size	http_server_request_body_size_bytes	Histogram	bytes	Size of incoming HTTP request bodies received by servers
Application	http.server.response.body.size	http_server_response_body_size_bytes	Histogram	bytes	Size of outbound HTTP response bodies sent by servers
Application	rpc.client.duration	rpc_client_duration_seconds	Histogram	seconds	Duration of outbound gRPC service calls
Application	rpc.server.duration	rpc_server_duration_seconds	Histogram	seconds	Duration of inbound RPC service calls
Application	sql.client.duration	sql_client_duration_seconds	Histogram	seconds	Time taken for SQL client operations (Experimental)
Application	redis.client.duration	redis_client_duration_seconds	Histogram	seconds	Time taken for Redis client operations (Experimental)
Application	db.server.operation.duration	db_server_operation_duration_seconds	Histogram	seconds	Server-side database operation duration for Redis, Memcached, and SQL.
Application	messaging.publish.duration	messaging_publish_duration	Histogram	seconds	Time taken for message publication (e.g., Kafka) (Experimental)
Application	messaging.process.duration	messaging_process_duration	Histogram	seconds	Time taken for message processing (e.g., Kafka) (Experimental)
Application	mcp.client.operation.duration	mcp_client_operation_duration_seconds	Histogram	seconds	Duration of an MCP request, observed on the sender.
Application	mcp.server.operation.duration	mcp_server_operation_duration_seconds	Histogram	seconds	Duration of an MCP request, observed on the receiver.
Application	mcp.client.session.duration	mcp_client_session_duration_seconds	Histogram	seconds	Duration of an MCP session, observed on the sender.
Application	mcp.server.session.duration	mcp_server_session_duration_seconds	Histogram	seconds	Duration of an MCP session, observed on the receiver.
Application Process	process.cpu.time	process_cpu_time_seconds_total	Counter	seconds	Total CPU time consumed, broken down by state (system/user/wait)
Application Process	process.cpu.utilization	process_cpu_utilization_ratio	Gauge	ratio	CPU time usage change over time, normalized by CPU count
Application Process	process.memory.usage	process_memory_usage_bytes	UpDownCounter	bytes	Physical memory currently used
Application Process	process.memory.virtual	process_memory_virtual_bytes	UpDownCounter	bytes	Virtual memory committed
Application Process	process.disk.io	process_disk_io_bytes_total	Counter	bytes	Total disk I/O bytes transferred
Application Process	process.network.io	process_network_io_bytes_total	Counter	bytes	Total network I/O bytes transferred
Go runtime	go.memory.limit	go_memory_limit_bytes	UpDownCounter	bytes	Go runtime memory limit.
Go runtime	go.memory.gc.goal	go_memory_gc_goal_bytes	UpDownCounter	bytes	Committed GC heap goal. Omitted when the runtime offset and the fallback symbol are both unavailable.
Go runtime	go.memory.gc.cycles	go_memory_gc_cycles_total	Counter	count	Completed Go GC cycles.
Go runtime	go.memory.gc.pause.duration	go_memory_gc_pause_duration_seconds	Histogram	seconds	Cumulative stop-the-world GC pause duration. Requires Go 1.22+.
Go runtime	go.memory.used	go_memory_used_bytes	UpDownCounter	bytes	Memory in use, broken down by go.memory.type (stack or other). Requires Go 1.23+.
Go runtime	go.memory.allocated	go_memory_allocated_bytes_total	Counter	bytes	Cumulative allocated heap bytes. Requires Go 1.23+.
Go runtime	go.memory.allocations	go_memory_allocations_total	Counter	allocations	Cumulative heap allocation count. Requires Go 1.23+.
Go runtime	go.cpu.time	go_cpu_time_seconds_total	Counter	seconds	Cumulative CPU time, broken down by go.cpu.state (and go.cpu.detailed_state where applicable). Requires Go 1.23+.
Go runtime	go.goroutine.count	go_goroutine_count	UpDownCounter	goroutines	Current goroutine count. Omitted rather than reported partially when the runtime snapshot is incomplete.
Go runtime	go.processor.limit	go_processor_limit	UpDownCounter	processors	GOMAXPROCS value.
Go runtime	go.config.gogc	go_config_gogc_percent	UpDownCounter	percent	GOGC configuration value.
Go runtime	go.schedule.duration	go_schedule_duration_seconds	Histogram	seconds	Cumulative runnable-to-running goroutine latency. Requires Go 1.20+.
JVM	jvm.memory.used	jvm_memory_used_bytes	UpDownCounter	bytes	HotSpot JVM memory used, per memory pool. Sum by jvm.memory.type for heap and non-heap totals.
JVM	jvm.memory.committed	jvm_memory_committed_bytes	UpDownCounter	bytes	HotSpot JVM memory committed.
JVM	jvm.memory.limit	jvm_memory_limit_bytes	UpDownCounter	bytes	HotSpot JVM memory limit.
JVM	jvm.memory.used_after_last_gc	jvm_memory_used_after_last_gc_bytes	UpDownCounter	bytes	HotSpot JVM memory used after the last GC.
JVM	jvm.class.loaded	jvm_class_loaded_total	Counter	classes	Classes loaded since JVM start.
JVM	jvm.class.unloaded	jvm_class_unloaded_total	Counter	classes	Classes unloaded since JVM start.
JVM	jvm.class.count	jvm_class_count	UpDownCounter	classes	Classes currently loaded.
JVM	jvm.thread.count	jvm_thread_count	UpDownCounter	threads	Current executing thread count.
JVM	jvm.cpu.time	jvm_cpu_time_seconds_total	Counter	seconds	CPU time used by the JVM process.
JVM	jvm.cpu.count	jvm_cpu_count	UpDownCounter	processors	Processors available to the JVM.
JVM	jvm.cpu.recent_utilization	jvm_cpu_recent_utilization_ratio	Gauge	ratio	Recent CPU utilization of the JVM process.
JVM	jvm.gc.duration	jvm_gc_duration_seconds	Histogram	seconds	JVM garbage-collection pause duration, broken down by jvm.gc.name and jvm.gc.action.
Node.js runtime	nodejs.eventloop.utilization	nodejs_eventloop_utilization_ratio	Gauge	ratio	Event-loop utilization. Requires Node.js 14.10+.
Node.js runtime	nodejs.eventloop.time	nodejs_eventloop_time_seconds_total	Counter	seconds	Cumulative event-loop time, broken down by nodejs.eventloop.state (idle or active). Requires Node.js 14.10+.
Node.js runtime	nodejs.eventloop.delay.*	nodejs_eventloop_delay_*_seconds	Gauge	seconds	Event-loop delay statistics. min, max, mean, stddev, p50, p90, p99. Requires Node.js 16.14+.
Node.js runtime	v8js.gc.duration	v8js_gc_duration_seconds	Histogram	seconds	V8 garbage-collection pause duration.
Node.js runtime	v8js.memory.heap.used	v8js_memory_heap_used_bytes	UpDownCounter	bytes	V8 heap memory used.
Node.js runtime	v8js.memory.heap.limit	v8js_memory_heap_limit_bytes	UpDownCounter	bytes	V8 heap memory limit.
Node.js runtime	v8js.memory.heap.space.available_size	v8js_memory_heap_space_available_size_bytes	UpDownCounter	bytes	Available V8 heap-space memory.
Node.js runtime	v8js.memory.heap.space.physical_size	v8js_memory_heap_space_physical_size_bytes	UpDownCounter	bytes	Physical (committed) V8 heap-space memory.
Node.js runtime	v8js.resource.active	v8js_resource_active	Gauge	resources	Live resources keeping the Node.js event loop active, broken down by v8js.resource.type. Requires Node.js 16.14+.
Python runtime	cpython.gc.collections	cpython_gc_collections_total	Counter	collections	CPython garbage-collection runs.
Python runtime	cpython.gc.collected_objects	cpython_gc_collected_objects_total	Counter	objects	Objects reclaimed by the CPython garbage collector.
Python runtime	cpython.gc.uncollectable_objects	cpython_gc_uncollectable_objects_total	Counter	objects	Uncollectable objects found by the CPython garbage collector.
.NET runtime	dotnet.process.memory.working_set	dotnet_process_memory_working_set_bytes	UpDownCounter	bytes	Physical memory currently mapped to the .NET process. Requires .NET 8+.
.NET runtime	dotnet.gc.heap.total_allocated	dotnet_gc_heap_allocated_bytes_total	Counter	bytes	Bytes allocated on the managed heap since process start.
.NET runtime	dotnet.gc.last_collection.memory.committed_size	dotnet_gc_last_collection_memory_committed_size_bytes	UpDownCounter	bytes	Committed GC memory observed at the latest collection.
.NET runtime	dotnet.gc.collections	dotnet_gc_collections_total	Counter	collections	Garbage collections since process start, broken down by dotnet.gc.heap.generation (gen0, gen1, gen2).
.NET runtime	dotnet.gc.pause.time	dotnet_gc_pause_time_seconds_total	Counter	seconds	Total garbage-collection pause time.
.NET runtime	dotnet.jit.compiled_methods	dotnet_jit_compiled_methods_total	Counter	methods	Methods compiled by the JIT since process start.
.NET runtime	dotnet.jit.compiled_il.size	dotnet_jit_compiled_il_size_bytes_total	Counter	bytes	Bytes of IL compiled by the JIT since process start.
.NET runtime	dotnet.jit.compilation.time	dotnet_jit_compilation_time_seconds_total	Counter	seconds	Total JIT compilation time.
.NET runtime	dotnet.thread_pool.thread.count	dotnet_thread_pool_thread_count	UpDownCounter	threads	Current thread-pool thread count.
.NET runtime	dotnet.thread_pool.queue.length	dotnet_thread_pool_queue_length	UpDownCounter	work items	Work items currently queued to the thread pool.
.NET runtime	dotnet.thread_pool.work_item.count	dotnet_thread_pool_work_item_count_total	Counter	work items	Work items completed by the thread pool since process start.
.NET runtime	dotnet.monitor.lock_contentions	dotnet_monitor_lock_contentions_total	Counter	contentions	Monitor lock contentions since process start.
.NET runtime	dotnet.timer.count	dotnet_timer_count	UpDownCounter	timers	Active timers.
.NET runtime	dotnet.assembly.count	dotnet_assembly_count	UpDownCounter	assemblies	Assemblies currently loaded.
Network	obi.network.flow.bytes	obi_network_flow_bytes	Counter	bytes	Bytes transmitted between source and destination network endpoints
Network	obi.network.flow.packets	obi_network_flow_packets_total	Counter	count	Packets transmitted between source and destination network endpoints.
Network	obi.network.inter.zone.bytes	obi_network_inter_zone_bytes	Counter	bytes	Cross-zone network bytes within Kubernetes clusters (Experimental)
Network	obi.stat.tcp.io	obi_stat_tcp_io_bytes_total	Counter	bytes	TCP bytes transferred.
Network	obi.stat.tcp.retransmits	obi_stat_tcp_retransmits_total	Counter	count	TCP segment retransmissions.
Network	obi.stat.tcp.failed.connections	obi_stat_tcp_failed_connections_total	Counter	connections	TCP failed connection count, labeled by network.tcp.handshake.role (client or server) and failure reason. The reason label is disabled by default.
Network	obi.stat.tcp.successful.connections	obi_stat_tcp_successful_connections_total	Counter	connections	TCP connections that completed the three-way handshake, labeled by network.tcp.handshake.role (client or server).
GPU	gpu.cuda.kernel.launch.calls	gpu_cuda_kernel_launch_calls_total	Counter	count	CUDA kernel launches observed (cudaLaunchKernel).
GPU	gpu.cuda.graph.launch.calls	gpu_cuda_graph_launch_calls_total	Counter	count	CUDA graph launches observed (cudaGraphLaunch).
GPU	gpu.cuda.kernel.grid.size	gpu_cuda_kernel_grid_size	Histogram	count	Distribution of CUDA kernel launch grid sizes.
GPU	gpu.cuda.kernel.block.size	gpu_cuda_kernel_block_size	Histogram	count	Distribution of CUDA kernel launch block sizes.
GPU	gpu.cuda.memory.allocations	gpu_cuda_memory_allocations_bytes_total	Counter	bytes	Bytes requested through CUDA device memory allocations (cudaMalloc).
GPU	gpu.cuda.memory.copies	gpu_cuda_memory_copies	Histogram	bytes	Distribution of CUDA memory copy sizes, broken down by cuda.memcpy.kind.
GPU	gpu.cuda.memory.free.bytes	gpu_cuda_memory_free_bytes_total	Counter	bytes	Bytes released through CUDA device memory frees (cudaFree).
GPU	gpu.cuda.memset.bytes	gpu_cuda_memset_bytes_total	Counter	bytes	Bytes set through CUDA memory set operations (cudaMemset).
GPU	gpu.cuda.host.register.bytes	gpu_cuda_host_register_bytes_total	Counter	bytes	Bytes registered through CUDA host memory registration (cudaHostRegister).
GPU	gpu.cuda.stream.create.calls	gpu_cuda_stream_create_calls_total	Counter	count	CUDA stream creations observed (cudaStreamCreate).
GPU	gpu.cuda.stream.destroy.calls	gpu_cuda_stream_destroy_calls_total	Counter	count	CUDA stream destructions observed (cudaStreamDestroy).
GPU	gpu.cuda.event.record.calls	gpu_cuda_event_record_calls_total	Counter	count	CUDA event records observed (cudaEventRecord).
GPU	gpu.cuda.event.synchronize.calls	gpu_cuda_event_synchronize_calls_total	Counter	count	CUDA event synchronizations observed (cudaEventSynchronize).
GPU	gpu.cuda.stream.synchronize.calls	gpu_cuda_stream_synchronize_calls_total	Counter	count	CUDA stream synchronizations observed (cudaStreamSynchronize).
GPU	gpu.cuda.device.synchronize.calls	gpu_cuda_device_synchronize_calls_total	Counter	count	CUDA device synchronizations observed (cudaDeviceSynchronize).

Enabling runtime metrics

Go, JVM, Node.js, Python, and .NET runtime metrics are all enabled through the shared application_runtime metrics feature; jvm_runtime_metrics.sampling_interval controls the JVM sampling rate. The Node.js event-loop and V8 metrics require the Node.js injector (nodejs.enabled, on by default), and .NET runtime metrics require .NET 8 or later. Each metric's introducing release is noted in the table above; see the OBI changelog for details.

GPU (CUDA) metrics are controlled separately by ebpf.instrument_cuda (OTEL_EBPF_INSTRUMENT_CUDA): auto (the default) enables them when the NVIDIA toolkit is present, on forces them on, and off disables them. MCP operation and session metrics are emitted when GenAI instrumentation is active. The obi.stat.tcp.successful.connections counter belongs to the opt-in stats metrics family.

Span and service-graph metrics

OBI also supports exporting Span metrics and Service Graph metrics, which can be toggled via feature configuration options.

The legacy application_span and application_span_sizes span-metric features are deprecated in favor of application_span_otel. The deprecated features still emit the Grafana-convention traces_spanmetrics_* metrics and keep working with a startup warning, while application_span_otel emits the OpenTelemetry-convention traces_span_metrics_* metrics. application_span_sizes has no direct replacement. The closest are http.server.request.body.size and http.server.response.body.size. Enabling a legacy feature together with application_span_otel is rejected, and the * (all) selector keeps only the OTel version.

Metric naming

Prometheus metric names are derived from their OTLP definitions rather than declared separately. On the OTLP export path, target_info, traces_target_info, traces_host_info, traces_span_metrics_calls_total, and traces_span_metrics_duration are emitted in dot notation as target.info, traces.target.info, traces.host.info, traces.span.metrics.calls, and traces.span.metrics.duration; the Prometheus scrape path keeps the underscore names. For heap totals, sum jvm.memory.used by jvm.memory.type (the OBI-specific obi.jvm.heap.used metric is not emitted). The OBI changelog lists the metric renames this introduced.

Span attributes

service.peer.name, http.request.body.size, http.response.body.size, and obi.http.response.observed are opt-in and no longer emitted by default. Select them through attributes->select->traces->include; service-graph generation that relies on service.peer.name must select it explicitly. HTTP duration and body-size metrics now carry error.type by default on failed requests, so their Prometheus series gain an error_type label (empty for successful requests). Parsed attributes whose values can't be determined are omitted instead of emitted as empty strings, and the host.id data-point attribute is dropped from span metrics (it stays on the resource, through target_info). HTTP body-size histograms require the application_sizes metrics feature, which is part of application but not application_red. Route harvesting populates http.route for Java (on Linux), Django, FastAPI, Flask, Rails, .NET, Symfony, Laravel, and Slim.

Database duration metrics carry the db.namespace, db.response.status_code, and server.port attributes, and messaging spans carry messaging.operation.name.

HTTP spans emit url.query by default when present, and client HTTP spans include the query string in url.full. Sensitive parameters are automatically redacted; extend the redaction list with OTEL_EBPF_SENSITIVE_QUERY_PARAMS_ADD.

Capture defaults

Database error response bodies are not included on SQL spans by default. Enable that capture explicitly if you need sql.client.duration spans to carry database error responses.

DNS calls are excluded from span metrics and service-graph metrics by default. SQL operations (MySQL, PostgreSQL, MariaDB) are reported as SERVER spans, so aggregations or filters that key on span kind for SQL traffic should account for that.

Attribute selection for OBI metrics

Metrics and attributes follow OTEL dot.notation but are converted to Prometheus underscore_notation in Prometheus outputs.

You can customize which attributes are exposed or hidden by using the attributes->select section in the configuration.

Common attributes table

Metrics	Attribute Name	Default
Application (all)	http.request.method	Shown
Application (all)	http.response.status_code	Shown
Application (all)	http.route	Shown if routes config exists
Application (all)	k8s.daemonset.name	Shown if Kubernetes metadata is on
Application (all)	k8s.deployment.name	Shown if Kubernetes metadata is on
Application (all)	k8s.namespace.name	Shown if Kubernetes metadata is on
Application (all)	k8s.node.name	Shown if Kubernetes metadata is on
Application (all)	k8s.owner.name	Shown if Kubernetes metadata is on
Application (all)	k8s.pod.name	Shown if Kubernetes metadata is on
Application (all)	k8s.container.name	Shown if Kubernetes metadata is on
Application (all)	k8s.pod.start_time	Shown if Kubernetes metadata is on
Application (all)	k8s.pod.uid	Shown if Kubernetes metadata is on
Application (all)	k8s.replicaset.name	Shown if Kubernetes metadata is on
Application (all)	k8s.statefulset.name	Shown if Kubernetes metadata is on
Application (all)	k8s.cluster.name	Shown if Kubernetes metadata is on
Application (all)	service.name	Resource attribute only
Application (all)	service.namespace	Resource attribute only
Application (all)	target.instance	Shown
Application (all)	url.path	Hidden
Application (client)	server.address	Hidden
Application (client)	server.port	Hidden
Application (process)	various process-specific attributes (e.g., process.command, process.executable.path)	Shown if process metrics enabled
Application RPC	various RPC attributes (rpc.method, rpc.grpc.status_code)	Shown
Application (server)	client.address	Hidden
Network	network-specific attributes (e.g., src.address, dst.address, iface)	Mostly hidden or conditional
Network (TCP failed connections)	network.tcp.handshake.role: values client or server	Shown

Note: The obi.network.inter.zone.bytes metric shares the same attribute set as obi.network.flow.bytes but all are hidden by default except k8s.cluster.name, src.zone, and dst.zone.

Note: service.name and service.namespace are no longer default metric labels. They are still reported as resource attributes (through target_info and the OTLP resource). To keep them on metric series, add them to the metric's include list under attributes->select.

Internal OBI metrics

OBI provides several internal Prometheus-format metrics to help monitor its own health and activity:

Name	Type	Description
obi_ebpf_tracer_flushes	Histogram	Size of trace groups flushed from eBPF to pipeline
obi_otel_metric_exports_total	Counter	Total metric batches sent to remote OTEL collector
obi_otel_metric_export_errors_total	CounterVec	Count of OTEL metric export errors by type
obi_otel_trace_exports_total	Counter	Total trace batches sent to remote OTEL collector
obi_otel_trace_export_errors_total	CounterVec	Count of OTEL trace export errors by type
obi_prometheus_http_requests_total	CounterVec	Prometheus scrape endpoint request count by port and path
obi_instrumented_processes	GaugeVec	Number of processes instrumented by Obi, labeled by process name
obi_instrumentation_errors_total	CounterVec	Instrumentation errors, labeled by process executable name and error type
obi_avoided_services	GaugeVec	Services OBI skipped because they already carry OpenTelemetry instrumentation, labeled by service name, namespace, and telemetry type. Emitted only when the avoided-services internal metric is enabled.
obi_internal_build_info	GaugeVec	Obi version information including build timestamp and commit ID

obi_otel_trace_exports_total and obi_otel_trace_export_errors_total reflect the real outcome reported by the exporter rather than counting attempts. Internal metrics also carry host metadata as resource attributes.

OBI can additionally report eBPF probe latency and map utilization, but that family is opt-in. It requires the ebpf metrics feature with a Prometheus endpoint, or internal_metrics.bpf_metric_scrape_interval set above zero. See the upstream BPF metrics collection reference for the metric names and labels.