Observability Metrics Catalogue¶
This document is the canonical inventory of metric families used by the Observability package and its dashboards.
It defines the metrics that the observability system expects to expose, the signal type and unit associated with each metric, the labels used to dimension the metric, and the instrumentation responsible for producing it.
Dashboard specifications and PromQL references should reference metrics defined in this catalogue rather than introducing independent metric names.
This catalogue is intentionally concerned with metric identity and instrumentation. It does not define dashboard visualizations or canonical ontology concepts. Those concerns are handled by the dashboard specifications and the field matrix respectively.
1. Scope¶
The catalogue currently covers:
- HTTP server metrics
- HTTP client metrics
- Database metrics
- Redis metrics
- Process and runtime metrics
- Python runtime metrics
- System metrics
- Logging signals
- Tracing signals
The catalogue also records planned metric families that are referenced by the observability roadmap but are not yet part of the minimum implementation.
2. Metric Status¶
Each metric is assigned an implementation status.
| Status | Meaning |
|---|---|
IMPLEMENTED |
Instrumentation exists in the package and is intended to emit the signal |
DASHBOARD_CONTRACT |
Required by the dashboard specification but emission has not yet been verified against the implementation |
PLANNED |
Identified for a future observability capability |
DEPRECATED |
Previously referenced but no longer part of the canonical implementation |
The status is deliberately conservative. A metric appearing in a dashboard specification does not by itself prove that the current instrumentation emits that exact Prometheus family.
3. Metric Naming Principles¶
Metrics should:
- Follow OpenTelemetry semantic conventions where applicable.
- Preserve stable metric identities across supported framework implementations.
- Use consistent units.
- Avoid introducing framework-specific metric names when an existing semantic metric can represent the same concept.
- Keep dashboard-facing metric families stable between package releases.
- Keep dimensional labels stable unless a breaking change is intentional.
- Avoid high-cardinality attributes such as raw URLs, SQL statements, request bodies, or user identifiers as metric dimensions.
Prometheus metric names documented here represent the metric families consumed by the current dashboard implementation.
The underlying OpenTelemetry instrument name and the Prometheus-exported name may differ depending on exporter and naming conventions. This distinction will be resolved explicitly in the field matrix.
4. Metric Types¶
| Type | Description |
|---|---|
| Counter | Monotonically increasing value representing accumulated events or work |
| Gauge | Current value that may increase or decrease |
| Histogram | Distribution of observed measurements |
| UpDownCounter | Value that can increase or decrease and represents a current quantity |
For Prometheus exposition, counters and histograms may appear with suffixes such as _total, _bucket, _sum, and _count.
5. HTTP Server Metrics¶
HTTP server metrics describe inbound requests received by an instrumented application.
They provide the foundation for the Service Overview and HTTP/API dashboards.
5.1 Request Duration¶
| Property | Value |
|---|---|
| Metric | http_server_duration_milliseconds |
| Type | Histogram |
| Unit | milliseconds |
| Dimensions | http_method, http_status_code, http_target |
| Instrumentation | FastAPI |
| Dashboards | Service Overview, HTTP/API |
| Status | IMPLEMENTED |
The metric represents the duration of completed HTTP server requests.
The histogram provides the underlying count, sum, and bucket series required for:
- Request rate
- Average latency
- P50 latency
- P95 latency
- P99 latency
- Endpoint latency analysis
The dashboard layer currently consumes the exported histogram representation.
5.2 Active Requests¶
| Property | Value |
|---|---|
| Metric | http_server_active_requests |
| Type | Gauge |
| Unit | requests |
| Dimensions | http_method |
| Instrumentation | FastAPI |
| Dashboards | Service Overview, HTTP/API |
| Status | IMPLEMENTED |
Represents the number of HTTP requests currently being processed.
6. HTTP Client Metrics¶
HTTP client metrics describe outbound HTTP calls made by an instrumented service.
They are primarily used by the External Services dashboard.
6.1 Client Request Duration¶
| Property | Value |
|---|---|
| Metric | http_client_request_duration |
| Type | Histogram |
| Unit | milliseconds |
| Dimensions | http_request_method, http_response_status_code, server_address |
| Instrumentation | HTTPX |
| Dashboard | External Services |
| Status | IMPLEMENTED |
Represents the duration of outbound HTTP requests.
The server_address dimension identifies the downstream service or external dependency.
6.2 Client Active Requests¶
| Property | Value |
|---|---|
| Metric | http_client_active_requests |
| Type | Gauge |
| Unit | requests |
| Dimensions | http_request_method |
| Instrumentation | HTTPX |
| Dashboard | External Services |
| Status | IMPLEMENTED |
Represents outbound HTTP requests currently in progress.
7. Database Metrics¶
Database metrics describe operations performed against databases by the application.
The package currently supports database instrumentation through SQLAlchemy and psycopg2.
7.1 Database Operation Duration¶
| Property | Value |
|---|---|
| Metric | db_client_operation_duration |
| Type | Histogram |
| Unit | milliseconds |
| Dimensions | db_operation, db_system_name, db_name |
| Instrumentation | SQLAlchemy / psycopg2 |
| Dashboard | Database |
| Status | IMPLEMENTED |
Represents the duration of database operations.
The metric is intended to support:
- Database request rate
- Database latency
- Operation-level analysis
- Database system comparisons
7.2 Active Database Operations¶
| Property | Value |
|---|---|
| Metric | db_client_operation_active |
| Type | Gauge |
| Unit | operations |
| Dimensions | db_operation |
| Instrumentation | SQLAlchemy / psycopg2 |
| Dashboard | Database |
| Status | IMPLEMENTED |
Represents database operations currently executing.
7.3 Database Connection Usage¶
| Property | Value |
|---|---|
| Metric | db_client_connections_usage |
| Type | Gauge |
| Unit | connections |
| Dimensions | state |
| Instrumentation | SQLAlchemy |
| Dashboard | Database |
| Status | IMPLEMENTED |
Represents database connection usage by connection state.
7.4 Database Connection Wait Time¶
| Property | Value |
|---|---|
| Metric | db_client_connections_wait_time |
| Type | Histogram |
| Unit | milliseconds |
| Dimensions | db_operation |
| Instrumentation | SQLAlchemy |
| Dashboard | Database |
| Status | IMPLEMENTED |
Represents time spent waiting for database connections or connection-related resources.
8. Redis Metrics¶
Redis is treated as an external dependency for dashboard purposes, while its operations use the database client metric namespace.
8.1 Redis Operation Duration¶
| Property | Value |
|---|---|
| Metric | db_client_operation_duration |
| Type | Histogram |
| Unit | milliseconds |
| Dimensions | db_operation, db_system_name |
| Instrumentation | Redis |
| Dashboard | External Services |
| Status | IMPLEMENTED |
The same metric family is used for database and Redis operations.
The db_system_name dimension distinguishes Redis operations from other database systems.
8.2 Redis Active Operations¶
| Property | Value |
|---|---|
| Metric | db_client_operation_active |
| Type | Gauge |
| Unit | operations |
| Dimensions | db_operation |
| Instrumentation | Redis |
| Dashboard | External Services |
| Status | IMPLEMENTED |
Represents Redis operations currently executing.
9. Process and Runtime Metrics¶
Process metrics describe the health and resource consumption of the application process.
These metrics are primarily consumed by the Infrastructure dashboard.
9.1 Process CPU Utilization¶
| Property | Value |
|---|---|
| Metric | process_cpu_utilization_ratio |
| Type | Gauge |
| Unit | ratio |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents CPU utilization by the application process.
The dashboard converts the ratio into a percentage for visualization.
9.2 Process CPU Time¶
| Property | Value |
|---|---|
| Metric | process_cpu_time_seconds_total |
| Type | Counter |
| Unit | seconds |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents cumulative CPU time consumed by the process.
9.3 Process Memory Usage¶
| Property | Value |
|---|---|
| Metric | process_memory_usage_bytes |
| Type | Gauge |
| Unit | bytes |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents memory currently used by the process.
9.4 Process Virtual Memory¶
| Property | Value |
|---|---|
| Metric | process_memory_virtual_bytes |
| Type | Gauge |
| Unit | bytes |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents virtual memory allocated to the process.
9.5 Process Start Time¶
| Property | Value |
|---|---|
| Metric | process_runtime_start_time |
| Type | Gauge |
| Unit | Unix timestamp |
| Dimensions | exported_job |
| Instrumentation | Process Runtime |
| Dashboards | Service Overview, Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents the process start timestamp.
It can be used to derive process uptime.
9.6 Process Thread Count¶
| Property | Value |
|---|---|
| Metric | process_thread_count |
| Type | Gauge |
| Unit | threads |
| Dimensions | exported_job |
| Instrumentation | Process Runtime |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents the number of active process threads.
9.7 Open File Descriptors¶
| Property | Value |
|---|---|
| Metric | process_open_file_descriptor_count |
| Type | Gauge |
| Unit | descriptors |
| Dimensions | exported_job |
| Instrumentation | Process Runtime |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents the number of open file descriptors held by the process.
10. Python Runtime Metrics¶
Python runtime metrics provide visibility into garbage collection behavior.
They are particularly relevant to services running on CPython.
10.1 Garbage Collection Collections¶
| Property | Value |
|---|---|
| Metric | cpython_gc_collections_total |
| Type | Counter |
| Unit | collections |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | IMPLEMENTED |
Represents completed garbage collection cycles.
10.2 Garbage Collection Collected Objects¶
| Property | Value |
|---|---|
| Metric | cpython_gc_collected_objects_total |
| Type | Counter |
| Unit | objects |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | IMPLEMENTED |
Represents objects reclaimed by garbage collection.
10.3 Garbage Collection Uncollectable Objects¶
| Property | Value |
|---|---|
| Metric | cpython_gc_uncollectable_objects_total |
| Type | Counter |
| Unit | objects |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | IMPLEMENTED |
Represents objects that could not be reclaimed by the garbage collector.
11. System Metrics¶
System metrics describe host-level resource utilization and I/O.
The current implementation invokes:
SystemMetricsInstrumentor().instrument()
The exact exported metric names should therefore be verified against the installed OpenTelemetry System Metrics package before being treated as immutable implementation contracts.
11.1 System Memory Utilization¶
| Property | Value |
|---|---|
| Metric | system_memory_utilization_ratio |
| Type | Gauge |
| Unit | ratio |
| Dimensions | exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents system memory utilization.
11.2 Network I/O¶
| Property | Value |
|---|---|
| Metric | system_network_io_bytes_total |
| Type | Counter |
| Unit | bytes |
| Dimensions | direction, exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents network bytes transmitted or received.
The direction dimension distinguishes traffic direction.
11.3 Disk I/O¶
| Property | Value |
|---|---|
| Metric | system_disk_io_bytes_total |
| Type | Counter |
| Unit | bytes |
| Dimensions | direction, exported_job |
| Instrumentation | System Metrics |
| Dashboard | Infrastructure |
| Status | DASHBOARD_CONTRACT |
Represents disk I/O throughput.
12. Resource Attributes¶
Not every piece of observability information is a metric.
The Observability package creates a resource containing service, deployment, host, operating system, and process information.
The current resource builder creates the following attributes:
| Attribute | Type | Source | Purpose |
|---|---|---|---|
service.name |
string | ObservabilityConfig.service_name |
Canonical service identity |
service.version |
string | ObservabilityConfig.service_version |
Deployed service version |
deployment.environment |
string | OTEL_ENVIRONMENT |
Deployment environment |
service.instance.id |
string | Generated UUID | Unique service instance identity |
host.name |
string | Hostname | Host identity |
host.arch |
string | Platform | Host architecture |
os.type |
string | Platform | Operating system |
process.pid |
integer | Process | Process identity |
process.runtime.name |
string | Python implementation | Runtime identity |
process.runtime.version |
string | Python version | Runtime version |
These attributes are attached to emitted telemetry through the OpenTelemetry Resource.
They are therefore available across metrics, traces, and logs depending on the exporter and backend.
13. Logging Signals¶
Logs are not Prometheus metrics.
They are exported using OTLP and stored or queried through the logging backend.
The current dashboard contract requires the following fields or labels:
| Field | Signal | Purpose |
|---|---|---|
service.name |
Resource attribute | Identifies the service producing the log |
deployment.environment |
Resource attribute | Identifies deployment environment |
severity |
Log attribute | Identifies log severity |
trace_id |
Log correlation field | Associates log with distributed trace |
span_id |
Log correlation field | Associates log with specific span |
The current implementation also uses:
LoggingInstrumentor().instrument(set_logging_format=False)
This injects trace and span correlation information into Python logging records.
The precise backend field names should be confirmed during the field matrix stage.
14. Tracing Signals¶
Traces are exported using OTLP and consumed by the tracing backend.
The minimum tracing information required by the dashboard specification is:
| Attribute | Purpose |
|---|---|
service.name |
Identifies the service |
span.name |
Identifies the operation |
trace_id |
Identifies the distributed trace |
span_id |
Identifies the individual span |
duration |
Measures span execution time |
status |
Indicates span outcome |
Tracing is not represented as Prometheus metric families.
The Traces dashboard primarily operates against the tracing backend and therefore has a separate query model from the metric dashboards.
15. Instrumentation Coverage¶
The current package exposes instrumentation entry points for:
| Instrumentation | Package Integration | Signal Coverage |
|---|---|---|
| FastAPI | instrument_fastapi() |
HTTP server metrics and traces |
| Flask | instrument_flask() |
Framework telemetry |
| Django | instrument_django() |
Framework telemetry |
| HTTPX | Automatically enabled by start() |
HTTP client telemetry |
| Redis | instrument_redis() |
Redis telemetry |
| SQLAlchemy | instrument_sqlalchemy() |
Database telemetry |
| psycopg2 | instrument_psycopg2() |
PostgreSQL telemetry |
| System Metrics | Automatically enabled by metrics setup | Runtime and system telemetry |
| Python Logging | Automatically enabled by logging setup | Log and trace correlation |
16. Instrumentation Lifecycle¶
The package requires providers to be initialized before framework or database instrumentation is registered.
The intended lifecycle is:
Observability()
↓
.service(...)
↓
.start()
↓
.instrument_fastapi(...)
.instrument_sqlalchemy(...)
.instrument_redis(...)
.instrument_psycopg2(...)
The package enforces this ordering through _require_started().
The start() method establishes the global OpenTelemetry providers before integrations are instrumented.
17. Dashboard Mapping¶
| Dashboard | Metric Families | Additional Signals |
|---|---|---|
| Service Overview | HTTP Server, Process Runtime | Resource attributes |
| Infrastructure | Process, Python Runtime, System | Resource attributes |
| HTTP/API | HTTP Server | Traces |
| Database | Database | Database spans |
| External Services | HTTP Client, Redis | Client and Redis spans |
| Logs | None | Logs |
| Traces | None | Traces |
18. Metric to Operational Question Mapping¶
The catalogue should ultimately allow dashboards to answer operational questions without coupling those questions to a particular implementation.
| Operational Question | Primary Metric |
|---|---|
| Is the service receiving traffic? | HTTP server request duration count |
| How much traffic is the service receiving? | HTTP server request duration count |
| Are requests failing? | HTTP server request duration count + status code |
| Are requests becoming slow? | HTTP server duration histogram |
| Which endpoints are busiest? | HTTP server request duration count + route |
| Which endpoints are slowest? | HTTP server duration histogram + route |
| Is the process consuming excessive CPU? | Process CPU utilization |
| Is memory usage increasing? | Process memory usage |
| Is the host under memory pressure? | System memory utilization |
| Is garbage collection increasing? | CPython GC collections |
| Are database operations slowing down? | Database operation duration |
| Are database connections constrained? | Database connection usage |
| Are external HTTP services slowing down? | HTTP client request duration |
| Is Redis becoming a bottleneck? | Redis operation duration |
| What happened during a failed request? | Trace and log correlation |
| Where did a request spend its time? | Trace span duration |
| Which downstream dependency caused latency? | HTTP client, database, Redis spans |
19. Current Implementation Boundary¶
The following distinction is important.
The dashboard specifications describe the observability contract.
The Python implementation describes the instrumentation implementation.
The Prometheus backend describes the exported metric representation.
These three layers must not be assumed to be identical.
Dashboard Requirement
↓
Metric Catalogue
↓
OTel Instrument
↓
OTel Attributes
↓
Exporter
↓
Prometheus Metric Family
For example, a dashboard may require a concept such as process CPU utilization, while the actual OpenTelemetry instrumentation may expose a different instrument name or set of attributes before the Prometheus exporter transforms it.
The field matrix is therefore responsible for resolving these differences.
20. Validation Requirements¶
Before a metric is promoted from DASHBOARD_CONTRACT to IMPLEMENTED, it should be verified against a running instrumented service.
Validation should confirm:
- The metric is emitted.
- The Prometheus metric family name is correct.
- The metric type is correct.
- The unit is correct.
- Required labels are present.
- Label values are populated correctly.
- Cardinality is acceptable.
- The metric works with the canonical PromQL query.
- The metric remains available across supported framework implementations where applicable.
This prevents the dashboard layer from becoming the accidental source of truth for instrumentation behavior.
21. Future Metrics¶
The following metric families are candidates for future releases:
- Celery task metrics
- Kafka producer metrics
- Kafka consumer metrics
- RabbitMQ metrics
- PostgreSQL connection pool metrics
- AI/LLM token usage metrics
- Cache hit ratio
- Queue depth
- Business KPIs
Future metrics must be added to this catalogue before being referenced by dashboard specifications.
Each future metric should define:
- Metric identity
- Type
- Unit
- Attributes
- Instrumentation source
- Dashboard usage
- Cardinality expectations
- Implementation status
22. Versioning¶
| Property | Value |
|---|---|
| Version | 1.0.0 |
| Maintainer | Observability Package |
| Last Updated | YYYY-MM-DD |
| Prometheus Compatibility | 3.x |
| Grafana Compatibility | 12.x |
| OpenTelemetry Status | Implementation aligned, field validation pending |
23. Relationship to the Field Matrix¶
This document answers:
What telemetry does the observability package expose or require?
The field matrix answers the next question:
How is each telemetry concept represented in OpenTelemetry, and how does it map to our canonical observability model?
The field matrix should therefore take the metric families in this catalogue and expand them into their underlying OTel instruments and attributes.
For example:
Metric
http_server_duration_milliseconds
↓
OTel instrument
↓
OTel attributes
http.request.method
http.response.status_code
http.route
↓
Canonical field
request.method
response.status_code
request.route
↓
Observability ontology
HTTP Request
The field matrix should also record discrepancies where the current implementation uses custom names or exporter-specific names rather than the corresponding OTel semantic convention.
This makes the field matrix the authoritative reconciliation layer between the implementation and the eventual minimum observability ontology.