Istio Service Mesh Guide: mTLS, Traffic and Ambient Mode

Istio is a service mesh: it moves mutual TLS, service-to-service authorization, traffic routing, retries and request telemetry out of your services and into proxies it manages, so every service gets them the same way without code changes. Istio has two data planes, both generally available since 1.24. Sidecar mode puts an Envoy proxy in every pod; ambient mode uses a per-node ztunnel for mTLS and L4 policy plus optional waypoint proxies for L7 features, and it is where we start new meshes. Adopt Istio when you need encrypted, identity-based traffic or consistent traffic control across many services, and skip it for a handful of services with no compliance driver, because a mesh is another distributed system to upgrade, secure and debug.

Key takeaways

• The strongest reason to run a mesh is identity: every workload gets a certificate, all traffic is mTLS, and authorization is written against service identities instead of IP addresses.
• Ambient mode cuts the per-pod cost to zero for L4 and lets you add L7 proxies (waypoints) only to the services that need retries, routing or HTTP-level policy.
• Roll out in stages: install, enroll one namespace in permissive mode, verify, go strict, then add deny-by-default authorization.
• Use Gateway API (HTTPRoute) for routing. In ambient mode VirtualService is Alpha, and mixing the two is unsupported.
• Most mesh incidents come from protocol detection, retry amplification, strict mTLS breaking clients outside the mesh, and policies that forget the waypoint's identity.
• Istio ships a minor release roughly every three months. Budget for upgrades the same way you budget for Kubernetes upgrades.

What a mesh gives you, and what it costs

The costs are real too. Every request crosses at least one extra proxy. Proxies use CPU and memory. The control plane becomes critical infrastructure, certificates become a dependency, and your on-call engineers need to read Envoy behavior, not just application logs. If two or three of the rows above are already handled well by your platform, the mesh may not pay for itself yet. Our microservices patterns guide covers the library-based alternatives.

Ambient or sidecar

Sidecar mode injects an Envoy proxy into every pod. It intercepts all inbound and outbound traffic and enforces both L4 and L7 policy at the pod. It is mature and complete, but every pod pays for a proxy, and joining or upgrading the mesh means restarting workloads.

Ambient mode splits the job in two. A ztunnel DaemonSet on each node handles mTLS, L4 authorization and TCP telemetry for every enrolled pod on that node, tunneling traffic over HBONE (HTTP CONNECT over mTLS). When a service needs L7 features, you deploy a waypoint, an Envoy proxy per namespace or per service, and only that traffic passes through it.

Istio publishes resource figures from its own benchmark (Istio 1.24, 1 KB HTTP requests, mTLS on, 2 proxy worker threads). Per 1,000 requests per second:

The per-request numbers are similar. The difference is the multiplier: sidecars scale with pod count, ztunnels with node count, and waypoints with the number of services that actually need L7. Treat these as Istio's numbers on Istio's test rig, and measure your own.

For a new mesh we start with ambient. For an existing sidecar mesh, migrate namespace by namespace, and read Istio's migration guide first: if you use L7 authorization policies, there is currently a window during the switch in which they are not enforced, so plan a maintenance window for those namespaces.

Install and enroll a namespace

Ambient needs the Gateway API CRDs. Following Istio's getting-started guide:

kubectl get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || \
  kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/experimental-install.yaml
 
istioctl install --set profile=ambient --skip-confirmation
 
kubectl label namespace payments istio.io/dataplane-mode=ambient

Pods in payments now have mTLS and L4 identity with no restart. Add a waypoint only when you need L7 features:

istioctl waypoint apply -n payments --enroll-namespace --wait
kubectl get gateway waypoint -n payments

--enroll-namespace labels the namespace with istio.io/use-waypoint: waypoint, so all traffic to its services passes through the waypoint. You can instead label individual Services to keep L7 processing off the services that do not need it.

For production, install with the Helm charts (base, istiod, cni, ztunnel), pin the version, keep the values in Git, and deploy them through the same pipeline as the rest of the cluster. Our GitOps guide covers that pattern. Run istiod with at least two replicas and a PodDisruptionBudget.

Sidecar mode, for comparison, is istioctl install --set profile=default, the label istio-injection=enabled (or a revision label such as istio.io/rev for canary control plane upgrades), and a rolling restart of every Deployment in the namespace.

Security: strict mTLS and deny-by-default

Step 1: verify, then enforce mTLS

New workloads accept both plaintext and mTLS by default (permissive). That is the right state for migration and the wrong state to stay in. Once every client of a namespace is in the mesh, make mTLS mandatory. Applied in the root namespace, this enforces it mesh-wide:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

Before you apply it, check in Kiali or in the istio_requests_total metric (the connection_security_policy label) that no plaintext traffic remains. Anything still sending plaintext, such as a monitoring system scraping pods from outside the mesh or a job in an unenrolled namespace, breaks the moment you go strict.

Step 2: deny by default, allow by identity

An AuthorizationPolicy with an empty spec allows nothing in its namespace. Then allow specific callers by service account:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-nothing
  namespace: payments
spec: {}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-ztunnel
  namespace: payments
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/payments/sa/waypoint
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-charges
  namespace: payments
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/orders/sa/orders
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/charges"]

The second policy is the one teams miss. A waypoint does not impersonate the original caller, so the destination ztunnel sees the waypoint's identity. With deny-by-default in place, ztunnel must allow traffic from the waypoint, and the waypoint enforces the real rule: only the orders service account may POST /v1/charges. In sidecar mode the same L7 rule uses a selector and is enforced by the destination sidecar, so there is no waypoint hop to allow.

This is what zero trust means at the network layer: no implicit trust from being inside the cluster, every call authenticated and authorized by identity. Our zero-trust architecture guide covers the identity, device and data layers around it.

Step 3: own your certificate authority

By default istiod acts as a self-signed CA. That is fine for one cluster. For multiple clusters, compliance audits, or controlled rotation of the root, plug in your own intermediate CA, for example through cert-manager's istio-csr or a Vault-backed issuer. Track the root and intermediate expiry dates as monitored items. A certificate authority that expires quietly is an outage for every service at once.

Traffic management with Gateway API

Use HTTPRoute for routing. It works in both data plane modes, it is the stable path in ambient, and the same API configures your ingress. A canary that sends 10 percent of requests to a new version, with a request timeout:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments
  namespace: payments
spec:
  parentRefs:
    - group: ""
      kind: Service
      name: payments
      port: 8080
  rules:
    - backendRefs:
        - name: payments-v1
          port: 8080
          weight: 90
        - name: payments-v2
          port: 8080
          weight: 10
      timeouts:
        request: 2s

payments-v1 and payments-v2 are ordinary Kubernetes Services selecting each version's pods. The parent payments Service is what clients call. Moving the weights by hand is fine for a first canary. For anything regular, let Argo Rollouts or Flagger move them and roll back on failed metrics, as described in the GitOps guide.

Connection limits and outlier detection stay in a DestinationRule, which both sidecars and waypoints apply:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
  namespace: payments
spec:
  host: payments.payments.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 200
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 500
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Outlier detection ejects a misbehaving pod from the load-balancing pool for a while instead of letting it keep failing a share of requests. maxEjectionPercent stops a bad deploy from ejecting so many pods that the remainder overload. The connection pool limits turn an overloaded dependency into fast, visible failures instead of an ever-growing queue. Derive the numbers from load tests of the service, not from an example.

Retries need a budget. Istio already retries failed HTTP requests twice by default. If three services in a call chain each retry, a single failure at the bottom can turn into many attempts against the service that is already struggling. Set retries deliberately, on idempotent calls only, at one layer of the chain, with per-try timeouts shorter than the overall timeout.

Observability without touching code

Sidecars and waypoints emit standard metrics for every request: istio_requests_total and istio_request_duration_milliseconds, labeled by source, destination and response code. In ambient mode, ztunnel reports TCP-level metrics only; HTTP metrics come from waypoints, so services without a waypoint show bytes and connections but not status codes. Kiali reads these metrics to draw the live service graph and validate your Istio configuration.

For tracing, point Istio at an OpenTelemetry collector and set a sampling rate with the Telemetry API:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: otel
      randomSamplingPercentage: 5

The otel provider is declared in the mesh config:

meshConfig:
  extensionProviders:
    - name: otel
      opentelemetry:
        service: opentelemetry-collector.observability.svc.cluster.local
        port: 4317

One rule the mesh cannot do for you: applications must propagate trace context headers (traceparent for W3C, or the B3 headers) from inbound to outbound requests. The proxy starts a span for each hop, but only the application knows which outbound call belongs to which inbound request. Without propagation you get one-hop traces. Our observability guide covers instrumenting the application side.

A rollout plan that does not cause an outage

1. Install the control plane and ztunnel. Enroll nothing. Confirm istiod, CNI and ztunnel are healthy on every node.
2. Enroll one low-risk namespace. Leave mTLS permissive. Watch error rates and latency for a few days against the same period before.
3. Check the protocol of every port. Name Service ports with a protocol prefix (http-, grpc-, tcp-) or set appProtocol. Run istioctl analyze -A.
4. Go strict per namespace once metrics show no plaintext callers.
5. Add deny-by-default authorization, starting with the most sensitive namespace. Apply each policy in staging first and watch for denials (HTTP 403 with RBAC: access denied at L7, reset connections at L4) before it reaches production.
6. Add waypoints only where you need L7 routing, L7 policy or HTTP metrics.
7. Write the runbook before you need it: how to remove a namespace from the mesh in an emergency (kubectl label namespace payments istio.io/dataplane-mode- and, if used, the waypoint label), who can do it, and how to tell when it is safe to re-enroll.

What goes wrong in production

Protocol detection guesses wrong. If Istio cannot tell from the port name or appProtocol what a port speaks, it sniffs. Server-first protocols such as MySQL, where the server talks before the client, break under sniffing in sidecar mode. Declare the protocol of every port.

Strict mTLS locks out legitimate clients. Prometheus scraping pods directly, a health checker outside the cluster, a CronJob in a namespace you forgot to enroll. Find them during the permissive phase, not after.

Retry storms. Default retries plus application retries plus client retries multiply load on the one service that is already failing. Audit the whole chain, not one hop.

Policies forget the waypoint. Under deny-by-default, traffic routed through a waypoint arrives at ztunnel with the waypoint's identity. Without an allow rule for it, every request through the waypoint is denied, and the symptom looks like a broken service rather than a policy.

Sidecar config grows with the mesh. In sidecar mode, every proxy receives configuration for every service it might call, so memory and config push time grow with mesh size. Scope it with Sidecar resources or discoverySelectors, or move to ambient, where ztunnel carries far less configuration.

Startup and shutdown ordering. Older setups suffered from applications starting before their sidecar was ready, and Jobs never completing because the sidecar kept running. Istio 1.27 and later use Kubernetes native sidecars by default on supported clusters, which fixes both. If you are still on an older version, set holdApplicationUntilProxyStarts, and plan the upgrade.

The control plane is down. Existing proxies keep their last configuration and certificates until they expire, so traffic keeps flowing, but new pods cannot get configuration or certificates. Run istiod with replicas spread across zones, alert on it, and know how long your workload certificates live.

Debugging without the tools. Learn these before an incident, not during one:

istioctl analyze -A                                  # configuration problems mesh-wide
istioctl proxy-status                                # is every proxy in sync with istiod
istioctl ztunnel-config workloads                    # what ztunnel knows about each pod
istioctl proxy-config routes deploy/orders -n orders # sidecar mode: the routes one proxy uses
kubectl logs -n istio-system -l app=ztunnel --tail=200

Frequently Asked Questions

What is Istio used for?

Istio is used to encrypt and authenticate service-to-service traffic with mutual TLS, to control which services may call which, to route traffic for canaries and failover, and to emit uniform request metrics and traces. It does this in proxies it manages, so application code does not change. Most teams adopt it for mTLS and authorization first and traffic management second.

Should I use Istio ambient mode or sidecar mode?

For a new mesh, start with ambient mode. It removes the per-pod sidecar, lets workloads join without a restart, and adds L7 proxies only where you need them. Stay on sidecar mode if you depend on EnvoyFilter, on features your Istio version supports only with sidecars, or on L7 policies you cannot migrate during a maintenance window.

Does Istio add latency?

Yes, every proxy hop adds some latency and consumes CPU and memory. Istio publishes its own measurements: in its Istio 1.24 test, a sidecar used about 0.20 vCPU and 60 MB per 1,000 requests per second, and a ztunnel about 0.06 vCPU and 12 MB. Measure the latency on your own traffic with and without the mesh before rolling it out widely.

Do I need a service mesh for microservices?

Not necessarily. A handful of services can get TLS from a library or a certificate manager, retries from a client library, and traces from OpenTelemetry. A mesh pays off when you have many services and teams, a requirement for encrypted identity-based traffic, or a need for consistent traffic control that you do not want to rebuild in every codebase.

What is the difference between Istio and Linkerd?

Both provide mTLS, traffic management and telemetry for Kubernetes. Istio has the broader feature set, supports Gateway API and ambient mode, and uses Envoy for L7. Linkerd is smaller and simpler to operate, with its own Rust proxy, and its open-source project now publishes only edge releases while stable releases come from vendors such as Buoyant.

Does Istio replace an API gateway?

Partly. Istio's ingress gateway, configured with Gateway API, handles routing, TLS termination and authorization at the edge. Product features such as API keys, developer portals, monetization and complex request transformation still belong in a dedicated API gateway, which can sit in front of or inside the mesh.

Need help with a service mesh?

We help teams decide whether they need a mesh at all, then design, roll out and operate Istio without a big-bang cutover: identity and mTLS first, policy second, traffic management where it earns its place. If we built it, we are on the pager for it, and you start with a call with the engineer who would do the work.

• Cloud Architecture and Infrastructure
• Cybersecurity and Compliance
• Talk to an engineer