Observability in Production: Logs, Metrics, Traces and SLOs

Production observability means you can answer a new question about a misbehaving system without shipping new code. In practice that takes four things: structured logs that carry a trace ID, a small set of golden-signal metrics for every service, distributed traces from OpenTelemetry, and alerts on user-facing symptoms measured against service level objectives instead of on CPU graphs. Instrument once with OpenTelemetry, route everything through a collector, and choose backends that fit your size. The expensive mistakes are high-cardinality metrics, unsampled traces, log volume nobody reads, and alerts nobody acts on.

Key takeaways

• Instrument with OpenTelemetry and send everything through a collector, so backends can change without code changes.
• Every log line carries trace_id and span_id, and every signal uses the same service.name. Correlation is the point.
• Measure latency with histograms and percentiles, never averages, and keep metric labels bounded.
• Sample traces at the tail: keep every error and slow request, sample the rest.
• Page on SLO burn rate with multiwindow alerts, and send everything else to a queue. Every page links to a runbook.
• Monitor the monitoring. An observability stack that dies with the cluster it watches tells you nothing during the outage that matters.

What each signal is for

The workflow during an incident follows the table top to bottom. An alert on a metric tells you checkout errors rose at 14:02. Exemplars or a trace search show failing requests spend their time in the payment client. The logs for one of those trace IDs show the provider returning a new error code after a configuration change. Each step narrows the search, and each depends on shared identifiers.

Instrument once with OpenTelemetry

OpenTelemetry (OTel) gives you vendor-neutral SDKs, automatic instrumentation for common frameworks, and one wire protocol, OTLP, that every serious backend accepts. Most services can start with zero-code instrumentation and standard environment variables. For a Node.js service:

npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
 
export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.version=2.14.0
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318
export OTEL_TRACES_SAMPLER=parentbased_always_on
 
node --require @opentelemetry/auto-instrumentations-node/register dist/server.js

Java, Python, .NET and Go have equivalents: an agent, a wrapper or a small amount of setup code. Automatic instrumentation creates spans for inbound HTTP and gRPC, database calls, outbound HTTP and common queue clients, and propagates W3C trace context headers between services. Add manual spans for business operations that matter, such as "price cart" or "reserve inventory", and attach attributes you will want to filter on (tenant, plan, feature flag), never secrets or personal data.

Route through a collector

Applications send to an OpenTelemetry Collector, not directly to a vendor. The collector batches, enriches, filters and samples, and it is the one place to change when you switch backends. This configuration sends traces to Tempo, logs to Loki and metrics to Prometheus, all over OTLP:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch: {}
 
exporters:
  otlp_http/tempo:
    endpoint: http://tempo.observability:4318
  otlp_http/loki:
    endpoint: http://loki.observability:3100/otlp
  otlp_http/prometheus:
    endpoint: http://prometheus.observability:9090/api/v1/otlp
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, tail_sampling, batch]
      exporters: [otlp_http/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp_http/prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp_http/loki]

Notes that save debugging time. memory_limiter goes first so an overloaded collector sheds data instead of crashing. tail_sampling lives in the contrib distribution, and it only works if every span of a trace reaches the same collector instance; with more than one replica, put a first tier of collectors in front that uses the load_balancing exporter keyed on trace ID. Prometheus accepts OTLP only when started with its OTLP receiver enabled (--web.enable-otlp-receiver). On Kubernetes, the usual shape is an agent collector per node (DaemonSet) for local collection and a gateway Deployment for sampling and export.

Structured logs that join to traces

A log line nobody can filter is a log line nobody reads. Emit JSON with a consistent set of fields, and include the active trace context:

{
  "timestamp": "2026-10-04T14:02:17.412Z",
  "level": "error",
  "service.name": "checkout",
  "message": "payment authorization failed",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "order_id": "ord_8f2c",
  "provider": "card-gateway",
  "provider_code": "do_not_honor",
  "duration_ms": 1840
}

OTel log bridges for common loggers (pino and winston in Node.js, Logback and Log4j in Java, the standard logging module in Python, slog in Go) add trace_id and span_id automatically. Rules that keep logs useful and affordable:

• Log events, not narration. One line per meaningful event with fields, not five lines of "entering function".
• Use levels honestly. ERROR means someone should look. WARN means a handled anomaly worth trending. INFO means a business event. DEBUG is off in production and switchable per service at runtime.
• Sample the boring, keep the bad. Successful health checks and high-volume success paths can be sampled or dropped at the collector. Never sample errors.
• Keep personal data out. Log identifiers, not emails, tokens, card numbers or request bodies. Logs are copied to more places, and kept longer, than any database.

Metrics: golden signals with bounded labels

Every service exposes the four golden signals from Google's SRE book: latency, traffic, errors and saturation. For request-driven services that maps to rate, errors and duration; for resources such as queues, pools and disks, it maps to utilization, saturation and errors.

Record latency as a histogram, never as an average. An average of 200 ms can hide a p99 of several seconds that a meaningful share of users see on every page. With Prometheus-style metrics:

sum by (route) (rate(http_requests_total{job="checkout"}[5m]))
 
sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))
 
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))

Choose histogram buckets around the thresholds you care about (if your objective is 300 ms, have buckets near 250, 300 and 400 ms), or use native histograms where your stack supports them, which avoid fixed buckets altogether.

Cardinality is what breaks metrics systems. Every unique combination of label values is a separate time series held in memory.

Put unbounded identifiers in traces and logs, where they belong, and link from metrics to traces with exemplars.

Traces: propagation and sampling

Traces are worth the effort once requests cross more than one service or one queue. Two things decide whether they work.

Propagation everywhere. HTTP and gRPC propagation is automatic with OTel instrumentation. Asynchronous hops are not always: put the traceparent into message headers or attributes when publishing to Kafka, SQS or RabbitMQ, and extract it in the consumer, so the consumer's work joins the producer's trace instead of starting a new one. If you run a service mesh, the proxies create spans for each hop but still rely on the application to propagate headers; our Istio guide covers that interaction.

Sampling at the tail. Head sampling decides at the start of a request, before you know whether it will fail, so it throws away exactly the traces you will want. Tail sampling, as in the collector configuration above, waits until the trace completes and keeps every error and slow request plus a percentage of the rest. The cost is collector memory for the decision window and the trace-ID routing requirement.

SLOs and alerts that page for the right reasons

Alerts on causes (CPU at 90 percent, one pod restarting) page people for things users never notice. Alerts on symptoms measured against an objective page them when users are affected and the trend matters.

Start with one service level indicator per critical user journey, for example "proportion of checkout requests that succeed in under one second", and an objective such as 99.9 percent over 30 days. The error budget is the remaining 0.1 percent: across 30 days, the equivalent of 43.2 minutes of total failure.

The Google SRE Workbook recommends alerting on burn rate, how fast you are consuming that budget, over two windows at once so alerts fire quickly and stop quickly:

As Prometheus rules for a 99.9 percent objective:

groups:
  - name: checkout-slo
    rules:
      - record: slo:checkout_errors:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
          / sum(rate(http_requests_total{job="checkout"}[5m]))
      - record: slo:checkout_errors:ratio_rate30m
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[30m]))
          / sum(rate(http_requests_total{job="checkout"}[30m]))
      - record: slo:checkout_errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
          / sum(rate(http_requests_total{job="checkout"}[1h]))
      - record: slo:checkout_errors:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[6h]))
          / sum(rate(http_requests_total{job="checkout"}[6h]))
 
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          slo:checkout_errors:ratio_rate1h > (14.4 * 0.001)
          and slo:checkout_errors:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its 30-day error budget 14x faster than sustainable"
          runbook_url: "https://runbooks.example.com/checkout/error-budget"
 
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          slo:checkout_errors:ratio_rate6h > (6 * 0.001)
          and slo:checkout_errors:ratio_rate30m > (6 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its 30-day error budget 6x faster than sustainable"
          runbook_url: "https://runbooks.example.com/checkout/error-budget"

Tools such as Sloth and Pyrra generate these rules from a short SLO definition, which beats maintaining them by hand across dozens of services. Around the rules, keep the hygiene that makes paging bearable:

• Every page links to a runbook: what the alert means, how to confirm impact, the first three things to try, and when to escalate. Our incident response playbook covers what happens after the page.
• Two severities that mean something. Page means a human acts now. Ticket means someone looks during working hours. Informational events go to dashboards, not to people.
• Review every page. If it fired and needed no action, fix the threshold or delete the alert. A team that learns to ignore pages will ignore the real one.

Choosing a stack for your size

For the first row, our own Trinetra is one option: a single small Go daemon per Linux server with live dashboards, readable alert rules and Telegram alerts, plus a fleet master with incidents and escalation once you have several servers. It needs no Prometheus and no external database. It does not export metrics to other systems yet, so if you expect to grow into a Prometheus stack soon, factor that in. Netdata and Uptime Kuma are other good fits at this size, and our Trinetra write-up compares them honestly.

Whatever you choose, keep OpenTelemetry at the application boundary. It is the part of the stack that makes every later change cheap.

What goes wrong in production

A label explodes. Someone adds user_id or a raw URL path as a metric label, the number of series multiplies, and Prometheus runs out of memory, taking every dashboard and alert with it. Enforce label allowlists at the collector or with relabeling, and alert on series count.

The monitoring dies with the system. The Prometheus and alerting stack runs in the same cluster or region as the workloads, and the outage that takes down one takes down both. Silence is then indistinguishable from health. Send an always-firing heartbeat alert (the Watchdog alert in kube-prometheus-stack) to an external dead man's switch that pages when the heartbeat stops.

Traces break at the queue. Requests look fine until they hit a message broker, after which the consumer's work appears as unrelated root traces. Propagate context through message headers.

Tail sampling drops half of every trace. Several collector replicas each see part of a trace and make different decisions. Route by trace ID before sampling.

The log bill arrives. Debug logging left on after an incident, or request bodies logged "temporarily", multiplies ingest. Put volume alerts on log ingestion per service, and give every team a view of what their service costs to observe.

Alerts flap. A threshold sits right at normal peak, the alert fires and resolves every evening, and people stop reading it. Use burn-rate windows, add for: durations to threshold alerts, and fix or delete anything that flaps.

Dashboards without owners. Hundreds of dashboards, none trusted, none updated after the last re-architecture. Keep one overview per service, owned by the team that runs it, built from the golden signals, and delete the rest. This matters more as systems split into many services, as our microservices patterns guide discusses.

Frequently Asked Questions

What is the difference between monitoring and observability?

Monitoring checks known failure conditions, such as a disk filling up or an error rate crossing a threshold, and tells you that something is wrong. Observability is the ability to work out why, including for failures nobody predicted, by querying logs, metrics and traces that share context. You need both: monitoring to get paged, observability to fix the problem.

What are the three pillars of observability?

Logs, metrics and traces. Logs record individual events in detail, metrics are cheap numeric time series for trends and alerting, and traces follow one request across services. Their value comes from being connected, so a trace ID in every log line and consistent service names across all three matter more than any single tool.

Is OpenTelemetry ready for production?

Yes. OpenTelemetry's tracing, metrics and logs specifications are stable, the SDKs for major languages are widely deployed, and every major observability vendor accepts OTLP. Instrumenting with OpenTelemetry and exporting through a collector lets you change backends without touching application code.

What should I alert on in production?

Page on symptoms users feel, measured against service level objectives: error rate and latency burning through the error budget faster than you can afford. Send capacity warnings such as disks filling or certificates expiring to a ticket queue. Every page should be actionable, link to a runbook, and be reviewed if it fires without needing action.

How do I reduce observability costs?

Logs and high-cardinality metrics drive most of the bill. Drop debug logs in production, sample successful request logs while keeping all errors, remove unbounded labels such as user or request IDs from metrics, and use tail sampling to keep slow and failed traces while sampling the rest. Set retention by signal instead of keeping everything for the same period.

Do I need Prometheus to monitor a few servers?

No. For a handful of Linux servers, a host monitor with sensible built-in alerts and an uptime check covers most failures with far less to operate. Prometheus and Grafana become worth their upkeep when you run many services, Kubernetes, or need custom application metrics and long-term queries.

Need help with observability?

We instrument and run production systems we are responsible for, so observability is how we sleep at night, not a dashboard project: OpenTelemetry instrumentation, collector pipelines, SLOs and alerting that pages for the right reasons. If we built it, we are on the pager for it, and you start with a call with the engineer who would do the work.

• Cloud Architecture and Infrastructure
• POC-to-Production Engineering
• Talk to an engineer