Structured Logging Best Practices for Production Systems

Structured logs store events as machine-readable records with consistent fields. A shared schema, trace context, redaction, and retention policy make those records easier to query during an investigation.

Why Structured Logging Matters in Production

Unstructured logs look like this:

ERROR: Failed to process order for user 4821 - payment declined

Structured logs look like this:

{
  "timestamp": "2025-06-05T09:14:33.421Z",
  "level": "error",
  "service": "order-service",
  "trace_id": "7d3f1a2b9c4e8f0d",
  "span_id": "a3c7b1e2",
  "msg": "payment declined",
  "user_id": 4821,
  "order_id": "ord_9f2k1",
  "amount_cents": 4999,
  "currency": "usd",
  "payment_provider": "stripe",
  "error_code": "card_declined"
}

The unstructured version is readable to a human who already knows the context. The structured version is queryable by any engineer, filtered by user_id, grouped by error_code, and automatically correlated with the trace that shows what happened before and after. That is the difference.

Structured Logging Best Practices

1. Use a Consistent Schema Across All Services

Define a baseline set of fields that every log entry emits, regardless of which service writes it:

• timestamp — ISO 8601, always UTC
• level — one of debug, info, warn, error, fatal
• service — the service name, matching your deployment identifier
• trace_id and span_id — critical for correlating logs with distributed traces
• msg — a short, static string describing the event (not a formatted sentence)
• env — production, staging, or development

Beyond the baseline, each service adds domain-specific fields. The baseline is what makes logs queryable across your entire fleet.

2. Keep the msg Field Static

The msg field should be a short, human-readable constant string — not a formatted sentence.

Wrong:

"msg": "Failed to charge user 4821 for order ord_9f2k1: card declined"

Right:

"msg": "payment declined"

Dynamic values belong in their own fields. Static messages allow you to group all instances of an event type in your log query tool without writing complex regex patterns. Every log aggregation tool handles field-level filtering efficiently; string matching against interpolated messages is fragile and slow.

3. Log Events, Not States

Each log entry should represent something that happened, not a description of the current system state. "Order processing started" is an event. "Order is being processed" is a state description. The event form is easier to sequence, correlate, and count.

4. Choose the Right Log Level

Teams that log everything at info end up with the same problem as teams that log nothing — signal buried in noise.

• debug — fine-grained detail useful only during development or when actively diagnosing a known issue. Should be off in production by default, configurable per-service without a restart.
• info — significant business events: request received, job completed, user authenticated.
• warn — something unexpected happened but the system recovered or degraded gracefully.
• error — an operation failed and a human should investigate.
• fatal — the process cannot continue.

If your production logs are mostly info level output from routine operations, you are building a haystack.

5. Include Trace Context in Every Log Entry

Every log line emitted during the handling of a request should carry the trace_id from that request's distributed trace. This single field is what turns logs from isolated records into a correlated story.

Most tracing libraries — OpenTelemetry, Jaeger clients, Datadog's tracer — expose the current trace context and can inject it into your logging library's context automatically. If you are using a language with middleware patterns, inject the trace context at the entry point and propagate it through every layer.

6. Do Not Log Sensitive Data

Logs are often shipped to third-party log aggregation services, retained for extended periods, and accessible to a wide range of team members. Review who can access the logging system and what retention and redaction policies apply before adding a field.

Concretely: never log passwords, full credit card numbers, raw API keys, or complete authentication tokens. Use opaque event or account identifiers for traceability. Avoid logging token fragments; apply the approved handling policy to payment and personal data.

7. Sample High-Volume Debug Logs

In high-throughput systems, emitting a structured log entry for every low-level operation — every cache lookup, every function entry — becomes expensive in both CPU and log ingestion cost. Use sampling for debug-level logs. A common pattern is to emit debug logs for a random percentage of requests, or to emit them only when the trace for that request has been flagged (for example, when it exceeded a latency threshold).

8. Test Your Log Output

Logs are code. They should be tested like code. Write tests that verify your service emits the expected fields in the expected format for key events. This prevents silent regressions where a refactor breaks the field name that your on-call alert queries for.

9. Define a Retention Policy Before You Ship

Log storage costs accumulate quickly. Before you go to production, decide:

• How long to retain each log level (debug logs may need only 24-48 hours; error logs may need 90 days)
• Whether to archive to cold storage after a retention window
• Who can query production logs and under what conditions

Defining this early prevents the common situation where teams are paying for months of verbose debug logs they never query.