Microservices Architecture Patterns That Work in Production

The microservices patterns that matter in production are the ones that keep a distributed system correct when calls fail, duplicate or arrive late: each service owning its data, a transactional outbox for reliable events, sagas instead of distributed transactions, idempotent consumers, and timeouts with bounded retries on every remote call, behind an API gateway that routes and does little else. Before any pattern, get the boundaries right, because most troubled microservices systems are distributed monoliths whose services must deploy together. If you cannot name the concrete scaling, team or deployment problem that splitting solves, a modular monolith is the better architecture.

Key takeaways

• Split along business capabilities with clear data ownership. If two services must always deploy together, they are one service.
• Never write to your database and publish an event as two separate steps. Use an outbox, and make every consumer idempotent.
• Replace distributed transactions with sagas that have explicit compensations, and prefer orchestration once a flow has more than a few steps.
• Every remote call gets a timeout, a retry policy with backoff and jitter, and a place in an overall deadline. Retries without a budget turn a slow dependency into an outage.
• Keep the API gateway thin: authentication, rate limiting and routing. Business logic in the gateway is a monolith at the edge.
• Contracts between services (APIs and event schemas) need versioning and tests, or every change becomes a coordinated release.

Should you split at all?

Microservices trade in-process function calls for network calls. That buys independent deployment, independent scaling and team autonomy. It costs latency, partial failure, eventual consistency, operational overhead and a much harder debugging story.

The pragmatic path is a modular monolith first: one deployable, with modules that own their tables, talk through defined interfaces, and never reach into each other's data. When a module has a real reason to leave, its boundary is already proven and extraction is mechanical. To move an existing monolith, use the strangler fig pattern: route one capability at a time to a new service at the edge, and delete the old code path once traffic has moved.

The pattern map

Boundaries and data ownership

Good boundaries follow business capabilities (pricing, ordering, payments, fulfillment), not technical layers (a "database service", a "notification service" every other service calls synchronously). Domain-driven design's bounded contexts are the most reliable tool here: an "order" in ordering and an "order" in shipping are different models with different rules, and each belongs to its own context.

Each service owns its data. Nothing else reads or writes its tables. That can mean separate databases, or separate schemas with separate credentials on one server when cost matters; what it cannot mean is a shared schema that three services migrate. Our database selection guide covers picking the store per service.

Signs your boundaries are wrong:

• Two services are always changed and deployed together.
• A request to service A cannot complete without a synchronous call to B, which calls C.
• Services share tables, or one service reads another's database "just for reporting".
• A change to one service's API requires a coordinated release of its consumers.

For reporting across services, stream changes into a warehouse with change data capture instead of querying service databases directly.

The edge: API gateway and BFF

Clients should not know your internal topology. An API gateway gives them one entry point and handles cross-cutting concerns: TLS, authentication and token validation, rate limiting, request routing, and request size limits.

When clients need different shapes of data, add a backend for frontend per client type. A mobile BFF aggregates three internal calls into one small payload for a slow network; a web BFF can return richer responses. The rule for both: no business logic. A gateway that computes prices or decides workflow state is a new monolith with worse tooling. API style choices, REST or GraphQL at the edge and gRPC inside, are covered in our REST vs GraphQL guide.

Synchronous calls: timeouts, retries and deadlines

Every remote call can hang. Without a timeout, a slow dependency holds threads, connections and memory until the caller falls over too. The minimum for every call:

• A timeout shorter than the caller's own deadline.
• Retries only for safe cases: idempotent operations, or requests carrying an idempotency key, and only on errors that a retry can fix (connection failures, 503, 429 after the indicated wait).
• Exponential backoff with jitter, so a fleet of clients does not retry in synchronized waves.
• A circuit breaker that fails fast when a dependency is clearly down, giving it room to recover.

type Attempt<T> = (signal: AbortSignal) => Promise<T>;
 
export async function callWithRetry<T>(
  attempt: Attempt<T>,
  { attempts = 3, baseDelayMs = 100, timeoutMs = 800 } = {},
): Promise<T> {
  let lastError: unknown;
  for (let i = 0; i < attempts; i++) {
    try {
      return await attempt(AbortSignal.timeout(timeoutMs));
    } catch (err) {
      lastError = err;
      if (!isRetryable(err) || i === attempts - 1) break;
      const ceiling = baseDelayMs * 2 ** i;
      await new Promise((r) => setTimeout(r, Math.random() * ceiling));
    }
  }
  throw lastError;
}

The random delay up to an exponentially growing ceiling is "full jitter", which spreads retries out better than a fixed backoff. isRetryable is where you encode which failures are worth retrying; a 400 never is.

Deadlines must shrink as a request goes deeper. If the edge gives a request 10 seconds, the gateway might allow 8, the first service 3 for its whole call to a dependency, and that dependency 1 per attempt. Propagate the remaining budget (gRPC does this natively with deadlines) so deep services stop working on requests the client has already abandoned.

Retries multiply. If three layers each make three attempts, one failing call at the bottom can become 27 requests against the service that is already struggling. Retry at one layer, usually the one closest to the user that knows whether the operation is safe to repeat. A service mesh can apply timeouts, retries and outlier detection uniformly across languages; our Istio guide covers that option and its default retry behavior.

Reliable events: the transactional outbox

The classic bug: a service saves an order, then publishes order.placed to the broker. If the process crashes between the two, the order exists and nobody downstream knows. Publishing first and saving second fails the other way. This is the dual-write problem, and retries do not fix it.

The outbox pattern writes the event to the service's own database in the same transaction as the business change. A relay publishes it afterward.

CREATE TABLE outbox (
  id            uuid        PRIMARY KEY,
  aggregate_id  text        NOT NULL,
  event_type    text        NOT NULL,
  payload       jsonb       NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  published_at  timestamptz
);
 
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;
 
BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES ($1, $2, $3, 'pending');
INSERT INTO outbox (id, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), $1, 'order.placed.v1', $4);
COMMIT;

The relay claims a batch, publishes it, and marks it sent. FOR UPDATE SKIP LOCKED lets several relay instances run without publishing the same rows:

BEGIN;
SELECT id, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;
 
UPDATE outbox SET published_at = now() WHERE id = ANY($1);
COMMIT;

The relay publishes the selected rows to the broker between the SELECT and the UPDATE. If it crashes after publishing but before committing, those rows are published again on the next run. That is why delivery is at least once and why the next pattern is mandatory. Change data capture tools such as Debezium can replace the polling relay by reading the outbox table from the database's replication log. Delete or archive published rows on a schedule so the table stays small.

Idempotent consumers

Brokers redeliver, relays republish, and clients retry. Every consumer must produce the same result when it sees a message twice. Record each processed message ID in the same transaction as the side effect:

BEGIN;
INSERT INTO processed_messages (consumer, message_id)
VALUES ('billing', $1)
ON CONFLICT DO NOTHING;
 
UPDATE invoices SET status = 'paid', paid_at = now()
WHERE order_id = $2 AND status <> 'paid';
COMMIT;

If the insert affects no rows, the message was already handled: skip the side effect, commit and acknowledge. For HTTP APIs, the same idea is an idempotency key header that the server stores with the result of the first request and returns for any repeat. Payment providers have used this for years for a reason.

Sagas instead of distributed transactions

Placing an order touches inventory, payments and shipping, each with its own database. There is no transaction spanning them, and two-phase commit across services couples their availability. A saga runs each step as a local transaction and defines a compensation for each: release the reservation, refund the payment.

Once a flow has more than a few steps, orchestration is easier to reason about. Durable workflow engines make it reliable: Temporal persists every step, so a coordinator crash resumes where it stopped instead of leaving the saga half done.

import { proxyActivities } from "@temporalio/workflow";
import type * as activities from "./activities";
 
const { reserveInventory, releaseInventory, chargePayment, refundPayment, createShipment } =
  proxyActivities<typeof activities>({
    startToCloseTimeout: "30 seconds",
    retry: { maximumAttempts: 5, initialInterval: "1 second", backoffCoefficient: 2 },
  });
 
export async function placeOrder(order: activities.Order): Promise<void> {
  const compensations: Array<() => Promise<void>> = [];
  try {
    await reserveInventory(order);
    compensations.unshift(() => releaseInventory(order));
 
    await chargePayment(order);
    compensations.unshift(() => refundPayment(order));
 
    await createShipment(order);
  } catch (err) {
    for (const compensate of compensations) {
      await compensate();
    }
    throw err;
  }
}

Two rules for compensations. They are business actions, not deletes: a refund is a new transaction with its own record. And they must be idempotent, because they will be retried. Design for the intermediate states, too: between the charge and the shipment, the order is "payment captured, not shipped", and support staff and customers may see it.

CQRS and event sourcing, used sparingly

Command Query Responsibility Segregation separates the model that handles writes from the models that serve reads. Start with the cheap version: read replicas, or read models built from the events you already publish. Event sourcing, storing every state change as an immutable event and deriving current state by replaying them, earns its complexity when you genuinely need a full audit history or temporal queries, as in ledgers. It brings schema evolution for events you can never delete and eventual consistency between writes and reads. Most systems need CQRS-lite, not event sourcing.

Contracts and versioning

Independent deployment only works if services can change without breaking their consumers.

• Make changes additive. Add fields; do not rename or remove them. Consumers ignore fields they do not know.
• Expand, then contract. Ship the new field or endpoint, migrate consumers, then remove the old one in a later release.
• Version events in their type name (order.placed.v1) and keep producing the old version until consumers have moved.
• Test contracts in CI. Consumer-driven contract tests (Pact is the common tool) fail the provider's build when a change would break a consumer. Schema registries enforce compatibility rules for event schemas.

What goes wrong in production

The distributed monolith. Services share a database or call each other synchronously for every request, so they deploy together and fail together. You have the costs of microservices and the coupling of a monolith. Fix the boundaries, or merge the services back.

Availability multiplies down a chain. If a request needs five services in sequence and each is available 99.9 percent of the time, the chain can be no better than about 99.5 percent (0.999 to the fifth power), before any network failures. Shorten synchronous chains, cache, and move work to asynchronous flows.

Retry storms. A dependency slows, every caller retries at every layer, and load on the struggling service multiplies until it falls over completely. Budget retries, use jitter, and let circuit breakers shed load.

Lost events from dual writes. Data saved, event never published, downstream state silently wrong. Usually found weeks later in reconciliation. Use an outbox.

Duplicate side effects. A redelivered message charges a card twice or sends two emails. Idempotent consumers and idempotency keys, everywhere a side effect leaves the system.

Stuck sagas. A compensation fails, nobody notices, and orders sit half complete. Orchestrators make these visible; with choreography, build a reconciliation job and an alert for flows that have not finished within their expected time.

More services than people. When the service count outgrows the team, ownership blurs, upgrades lag and on-call becomes guesswork. A useful check: every service has a named owning team, a runbook and a dashboard. Our observability guide covers tracing requests across those services, which you will need on the first bad night.

Frequently Asked Questions

What are the most important microservices patterns?

Database per service, the transactional outbox for reliable events, sagas for multi-service workflows, idempotent consumers, and timeouts with bounded retries and circuit breakers on every remote call. An API gateway or backend for frontend handles the edge. These patterns exist because every network call can fail, duplicate or arrive late, and the system has to stay correct when that happens.

When should you not use microservices?

Avoid them when one team owns the whole product, when you cannot yet say where the domain boundaries are, or when no concrete problem such as independent scaling or deployment is forcing the split. A modular monolith with clear internal boundaries gives most of the design benefits without distributed failure modes. You can extract services later along boundaries you have proven.

What is the saga pattern?

A saga runs a business transaction that spans several services as a sequence of local transactions, each with a compensating action that undoes its effect if a later step fails. Sagas can be choreographed through events or orchestrated by a coordinator such as Temporal or AWS Step Functions. They give eventual consistency, not the isolation of a database transaction, so the design must tolerate intermediate states.

What is the transactional outbox pattern?

The outbox pattern writes a business change and the event describing it to the same database in one transaction, then a separate relay publishes the event to the message broker. This avoids the dual-write problem, where a crash between saving data and publishing an event loses one of them. Delivery is at least once, so consumers must be idempotent.

Should each microservice have its own database?

Each service should own its data, and no other service should read or write its tables directly. That can mean a separate database, or separate schemas with separate credentials on a shared server to save cost. A shared schema couples services at the data layer and turns every migration into a coordinated release.

What is the difference between an API gateway and a service mesh?

An API gateway handles north-south traffic, requests from outside clients entering the system, with concerns such as authentication, rate limiting and routing. A service mesh handles east-west traffic between services, with mutual TLS, service identity, retries and telemetry. Many production systems use both.

Need help with your service architecture?

We design and run distributed systems, and we also tell teams when they should not split yet. Boundaries, data ownership, messaging and the operational basics that keep services correct at 3 a.m. If we built it, we are on the pager for it, and the first call is with the engineer who would do the work.

• Software Development
• POC-to-Production Engineering
• Talk to an engineer