Observability Domain¶

The Observability domain describes the telemetry produced by applications and infrastructure, the systems used to collect and store that telemetry, and the dashboards and analytical interfaces used to understand service behaviour.

The domain covers three primary observability signals:

Metrics
Logs
Traces

These signals provide complementary views of the same operational environment.

Metrics describe measurable system behaviour over time. Logs provide event level information about what happened. Traces describe the execution path of individual requests across services and their dependencies.

Purpose¶

The purpose of the Observability domain is to provide a consistent model for understanding application health, performance, reliability, and dependencies across the organization's services.

The domain supports questions such as:

Is a service healthy?
How much traffic is a service receiving?
Are requests becoming slower?
Are errors increasing?
Which endpoints are responsible for performance problems?
Is the application or its infrastructure becoming the bottleneck?
Are database or external service dependencies causing latency?
What happened during a failed request?
Where is time being spent within a distributed request?

Observability Signals¶

Metrics¶

Metrics provide numerical measurements of system behaviour over time.

The Observability package uses metrics for areas including:

HTTP server activity
HTTP client activity
Database operations
Redis operations
Process runtime
Python runtime
System resources

Metrics are exported through OpenTelemetry and stored in Prometheus for querying and visualization.

The canonical set of metrics is defined in the Metrics Catalogue.

Logs¶

Logs provide event level information generated by applications and services.

The Observability package stores logs in Loki.

Required logging attributes include:

service.name
deployment.environment
severity
trace_id
span_id

Logs are queried using LogQL rather than PromQL.

Traces¶

Traces describe the execution path of individual requests through a service and its dependencies.

A trace consists of spans representing individual operations within the request.

The Observability package exports traces using OTLP and uses Jaeger for trace storage and investigation.

Tracing supports investigation of:

Slow requests
Failed requests
Database operations
Redis operations
Outbound HTTP requests
Parent and child operations
Distributed request execution

Observability Architecture¶

The observability stack follows a common collection and storage pattern.

Application
    │
    ▼
OpenTelemetry Instrumentation
    │
    ▼
OpenTelemetry SDK
    │
    ▼
OpenTelemetry Collector
    │
    ├───────────────┐
    │               │
    ▼               ▼
Prometheus        Loki
Metrics           Logs
    │
    │
    └───────────────┐
                    │
                    ▼
                  Grafana


Application
    │
    ▼
OpenTelemetry Instrumentation
    │
    ▼
OpenTelemetry SDK
    │
    ▼
OTLP
    │
    ▼
Jaeger
    │
    ▼
Grafana

The specific collector configuration and data extraction mechanisms are documented separately from this overview.

Dashboard Model¶

The Observability package provides dashboards at progressively increasing levels of detail.

Service Overview
        ↓
Infrastructure
        ↓
HTTP/API
        ↓
Database
        ↓
External Services
        ↓
Logs
        ↓
Traces

The Service Overview dashboard provides the initial assessment of service health.

The subsequent dashboards provide progressively deeper investigation into infrastructure, HTTP behaviour, databases, external dependencies, logs, and individual request traces.

Dashboard definitions are maintained separately under dashboard-specs/.

Canonical Contracts¶

The Observability domain uses several complementary documents as contracts.

Document	Purpose
Metrics Catalogue	Defines canonical observability metrics
PromQL Library	Defines reusable PromQL queries
Dashboard Standards	Defines dashboard design and behaviour
Dashboard Specifications	Defines individual dashboard requirements
Field Matrix	Defines the observable fields and attributes exposed by the stack
Source Mappings	Defines how telemetry maps from instrumentation to storage and visualization

These documents should reference one another rather than independently redefining the same concepts.

Relationship to the Organizational Data Model¶

The Observability domain is one of the organization's operational data domains.

Its immediate purpose is operational visibility through Prometheus, Loki, Jaeger, and Grafana.

Over time, observability data may also become a source for the broader Organizational Data Model.

This creates two related but distinct concerns:

Operational Observability
        │
        ├── Metrics
        ├── Logs
        └── Traces
                │
                ▼
        Organizational Data

The operational observability systems remain responsible for high volume telemetry collection, querying, and real time investigation.

The Organizational Data Model may consume selected observability information for broader analysis, institutional memory, reporting, and relationships with other organizational domains.

The extraction and persistence strategy for this secondary use is documented separately.

Scope¶

The Observability domain currently covers:

Application metrics
Runtime and system metrics
HTTP server metrics
HTTP client metrics
Database metrics
Redis metrics
Application logs
Distributed traces
Grafana dashboards
Prometheus queries
Loki queries
Jaeger based trace investigation

Future telemetry may include:

Celery task metrics
Kafka producer and consumer metrics
RabbitMQ metrics
PostgreSQL pool metrics
AI and LLM token usage
Cache hit ratios
Queue depth
Business KPIs

Future signals should be added to the relevant canonical contracts before being referenced by dashboards.

Domain Principles¶

The Observability domain follows these principles:

OpenTelemetry first

Instrumentation should follow OpenTelemetry semantic conventions whenever applicable.

Canonical definitions

Metrics and other observable fields should have a single canonical definition.

Separation of signals

Metrics, logs, and traces should remain distinct signals while retaining the ability to correlate them through shared identifiers such as service names and trace identifiers.

Reusable queries

Dashboard queries should be defined centrally where they are reused.

Consistent dashboards

Dashboards should follow common naming, layout, units, navigation, and visualization standards.

Progressive investigation

Dashboards should move from high level service health toward increasingly detailed investigation.

Operational and organizational concerns remain distinct

The observability stack is optimized for operational telemetry. Only the data required for broader organizational analysis should be extracted into the Organizational Data Model.