k8wiz
Back to Articles

Ready to test your Istio knowledge? Try our Istio quiz!

How to roll out strict mTLS in Istio without breaking traffic

Move an Istio mesh from PERMISSIVE to STRICT mutual TLS one namespace at a time: find plaintext clients first, handle exceptions, then lock down the mesh.

7 min read
By k8wiz Team
  • istio
  • mtls
  • security
  • service-mesh

A fresh Istio install already encrypts most traffic. Sidecars use mutual TLS whenever both ends have a proxy, without any configuration. But every workload still accepts plain text too, because Istio starts in PERMISSIVE mode. Any pod without a sidecar, or anything outside the mesh that can reach a pod IP, can talk to your services unencrypted and unauthenticated.

Switching to STRICT closes that gap. Do it in one step on a busy mesh, though, and every client you forgot about breaks at once. This guide moves to strict mTLS gradually. You find the plaintext clients first, lock down one namespace at a time, handle the exceptions explicitly and finish with a mesh-wide policy. It assumes sidecar mode and follows the Istio security documentation.

How the modes work

Mutual TLS in Istio has two halves. On the client side, auto mTLS makes a sidecar send mutual TLS to destinations that have a proxy and plain text to destinations that don't. On the server side, a PeerAuthentication policy decides what a workload accepts.

There are three modes. PERMISSIVE accepts both mutual TLS and plain text. STRICT accepts only mutual TLS. DISABLE turns mutual TLS off, and Istio's docs say not to use it unless you have your own security layer. A policy with no mode inherits from the scope above it, and the mesh-wide default is PERMISSIVE.

Policies apply at three scopes. A policy in the root namespace with no selector covers the whole mesh; by default the root namespace is istio-system. A policy in another namespace with no selector covers that namespace. A policy with a selector covers matching workloads only. Istio applies the narrowest one that matches, so a workload policy beats a namespace policy, which beats the mesh policy.

Step 1: take stock of existing policies

Before you add anything, see what is already there. Old policies and destination rules from earlier experiments are a common source of surprises.

kubectl get peerauthentication --all-namespaces
kubectl get destinationrule --all-namespaces -o yaml | grep -n -B8 "mode: DISABLE"

The second command looks for destination rules that turn TLS off on the client side. A destination rule with trafficPolicy.tls.mode: DISABLE makes clients send plain text to that host, even if the host has a sidecar. Once the host goes STRICT, those clients fail. Remove the tls block or set mode: ISTIO_MUTUAL before you lock anything down.

Keep in mind that there can be only one mesh-wide policy and one namespace-wide policy per namespace. If you create two, Istio ignores the newer one. Name them default by convention so duplicates stand out.

Step 2: find the clients still sending plain text

Istio's telemetry tells you which connections use mutual TLS. On metrics reported by the destination proxy, the connection_security_policy label is mutual_tls when the connection used Istio's mTLS. Everything else is traffic that STRICT would reject.

Run this query in Prometheus over a window long enough to include batch jobs and nightly tasks. A week is a good start.

sum by (destination_workload_namespace, destination_workload, source_workload_namespace, source_workload) (
  rate(istio_requests_total{reporter="destination", connection_security_policy!="mutual_tls"}[1h])
)

For TCP services, run the same query on istio_tcp_connections_opened_total.

sum by (destination_workload_namespace, destination_workload, source_workload_namespace, source_workload) (
  rate(istio_tcp_connections_opened_total{reporter="destination", connection_security_policy!="mutual_tls"}[1h])
)

Plain-text clients often show up with source_workload="unknown". A client without a sidecar doesn't report its identity, so the destination can't fill it in. To find out who it is, look at the destination's access logs for source IPs, or check which namespaces in the cluster don't have sidecar injection.

kubectl get namespaces -L istio-injection

Step 3: move the clients into the mesh

Most plain-text clients are pods in namespaces without sidecar injection. Turn injection on and restart the workloads so new pods get a sidecar.

kubectl label namespace reports istio-injection=enabled --overwrite
kubectl rollout restart deployment -n reports

Once their pods have sidecars, auto mTLS takes care of the rest. Their traffic disappears from the query in step 2 without any further change.

Some clients can't join the mesh: a database proxy on a VM, a monitoring agent that scrapes pods directly, or a legacy job you don't own. Write them down. Each one needs an explicit exception in step 5, or it has to reach the service another way, such as through an ingress gateway.

Step 4: lock down one namespace

Start with a namespace whose clients are all in the mesh. Apply a namespace-wide policy named default with no selector.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT
kubectl apply -f payments-strict.yaml

Test it from both sides. A pod with a sidecar should still get through, and a pod without one should now fail.

kubectl run mtls-probe -n default --image=curlimages/curl --restart=Never --rm -i -- \
  curl -sS -o /dev/null -w "%{http_code}\n" http://api.payments:8080/healthz

That test pod runs in the default namespace, which usually has no injection, so it sends plain text. With the policy in place, curl prints 000 and exits with code 56, because the server side resets the connection. Run the same check from a meshed namespace and you get a normal status code.

Then watch the namespace for a while. The clients that break are the ones outside the mesh, so their own logs and error rates tell you more than the server's metrics. If something you missed breaks, set the mode back to PERMISSIVE. Istio pushes policy changes to the proxies almost in real time, so the rollback is quick.

Istio's docs also warn that proxies don't all receive a policy at exactly the same moment. Moving straight from DISABLE to STRICT can drop requests during that window. Go through PERMISSIVE first if any workload currently has mTLS disabled.

Step 5: handle exceptions explicitly

For a workload that must keep accepting plain text, add a workload-specific policy with a selector. It overrides the namespace policy for those pods only.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: legacy-gateway
  namespace: payments
spec:
  selector:
    matchLabels:
      app: legacy-gateway
  mtls:
    mode: PERMISSIVE

Often only one port needs plain text, such as a metrics port scraped from outside the mesh. Keep the workload STRICT and relax that port alone with portLevelMtls.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: api
  namespace: payments
spec:
  selector:
    matchLabels:
      app: api
  mtls:
    mode: STRICT
  portLevelMtls:
    9090:
      mode: PERMISSIVE

Two rules apply to portLevelMtls. The number is the container port, not the Service port. And Istio only honors it when a Service exposes that port; otherwise it ignores the setting without an error. portLevelMtls also only applies in a policy with a selector, so you can't use it in a namespace-wide or mesh-wide policy. The reverse holds in the root namespace, where Istio ignores policies that have a selector.

Prefer PERMISSIVE over DISABLE for these exceptions. With PERMISSIVE, meshed clients still use mutual TLS to that workload, and only the clients that can't keep sending plain text.

Step 6: lock down the whole mesh

When every namespace runs STRICT without trouble, move the setting to the root namespace. A policy in istio-system with no selector becomes the mesh-wide default.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

New namespaces now start out strict, which is the real gain. Namespace and workload exceptions keep working because they are narrower. After a while you can delete the namespace-level STRICT policies, since they repeat the mesh default.

Things that keep working, and one that gets better

Kubernetes health probes keep working under STRICT. The kubelet has no Istio certificate, so Istio rewrites HTTP, TCP and gRPC probes to go through the sidecar's agent by default. Only pods annotated with sidecar.istio.io/rewriteAppHTTPProbers: "false" need attention.

Authorization gets stronger. AuthorizationPolicy rules that match on principals or namespaces read the client identity from the mTLS certificate. Under PERMISSIVE, plain-text requests carry no identity, so those rules can behave in ways you didn't intend. The Istio docs recommend always using those fields together with STRICT mode, and finishing this rollout makes your existing identity-based rules mean what they say.

Test yourself

Ready to test your Istio knowledge? Try our Istio quiz!