k8wiz
Back to Articles

Ready to test your Cilium knowledge? Try our Cilium quiz!

How to troubleshoot dropped traffic in Cilium with Hubble

Find out why Cilium drops a pod's traffic: catch the drop with Hubble, read the drop reason, find the policy behind it and fix it.

9 min read
By k8wiz Team
  • cilium
  • hubble
  • troubleshooting
  • network-policy

A pod can't reach a Service, the request hangs, and the application logs say nothing useful. On a cluster that runs Cilium, the packet was most likely dropped inside an eBPF program on one of the nodes. Your usual tools don't see that. There is no iptables rule to count, and tcpdump in the pod only shows packets leaving and never coming back.

Cilium records every drop it makes, together with a reason code, and Hubble lets you read those records. This guide walks through the order we use on real incidents: make sure Hubble can see the traffic, catch the drop, read the reason, then find the policy or Service behind it. The commands and field names come from the Cilium 1.20 source and documentation.

Step 1: check that Hubble is running and reachable

Hubble runs inside each Cilium agent and keeps recent flows in a ring buffer. Hubble Relay collects those buffers from every node so you can query the whole cluster from your laptop. Start by checking that both are healthy.

cilium status
hubble status -P

The -P flag (--port-forward) makes the Hubble CLI open a port-forward to Hubble Relay on localhost:4245 for you. Without it, run cilium hubble port-forward in another terminal first. A healthy hubble status looks like this:

Healthcheck (via 127.0.0.1:4245): Ok
Current/Max Flows: 11917/12288 (96.98%)
Flows/s: 11.74
Connected Nodes: 3/3

Read two lines here. "Connected Nodes" should match your node count, or flows from the missing nodes will never show up. "Current/Max Flows" tells you how much history you have. The buffer is small (the Helm default for hubble.eventBufferCapacity is 4095 flows per node) and the oldest flows are overwritten first. On a busy node that can mean only a few seconds of history, so reproduce the problem while you watch rather than searching for something that happened an hour ago.

One more check saves time. Hubble only sees pods that Cilium manages. A pod with hostNetwork: true, or one started before Cilium was installed and never restarted, will not appear in flows at all.

Step 2: catch the drop while you reproduce it

Open a stream of dropped flows for the namespace you care about, then trigger the failing request from a second terminal.

hubble observe -P --namespace shop --verdict DROPPED --follow
kubectl -n shop exec deploy/frontend -- curl -sS -m 5 http://payments:8080/health

Every line that appears is a flow Cilium dropped. The Cilium documentation shows output like this for a client blocked by an ingress policy:

May  4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
May  4 13:23:48.854: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)

Each line holds a timestamp, the source and destination (pod names when Hubble can resolve them), the drop reason, the verdict and a summary. The repeated SYN with a one-second gap is the client's TCP stack retrying. That retry pattern is what a timeout looks like from the inside. By default Cilium drops policy-denied packets silently, so the client waits instead of getting an error.

If nothing shows up, widen the filter before you conclude that Cilium is innocent. Drop --namespace and use --pod shop/frontend (which matches both directions), or --to-ip with the Service or backend IP. The arrow tells you about direction: -> is a request, <- a reply, and <> means Hubble couldn't tell.

Step 3: read the drop reason

The text before DROPPED is the drop reason, and it decides where you look next. In JSON output the same value appears as drop_reason_desc, an enum name such as POLICY_DENIED. You can filter on it directly:

hubble observe -P --drop-reason-desc POLICY_DENIED --last 50

These are the reasons we see most often in practice, with their enum names from the Hubble flow API:

  • Policy denied (POLICY_DENIED, code 133): the endpoint is in default-deny for that direction and no rule allows this traffic.
  • Policy denied by denylist (POLICY_DENY, code 181): an explicit ingressDeny or egressDeny rule matched.
  • Service backend not found (SERVICE_BACKEND_NOT_FOUND, code 158): the client hit a Service address that has no usable backend.
  • Stale or unroutable IP (STALE_OR_UNROUTABLE_IP, code 151): the destination IP doesn't belong to a known endpoint, often a pod that was just deleted.
  • CT: Map insertion failed (code 155): the connection tracking map is full.
  • Authentication required (AUTH_REQUIRED, code 189): a policy asks for mutual authentication that hasn't completed.

The two policy reasons are worth telling apart. POLICY_DENIED means something is missing, so the fix is an allow rule. POLICY_DENY means a rule actively blocks the traffic, and adding allow rules won't help, because Cilium gives deny rules precedence over allow rules from any policy type, Kubernetes NetworkPolicy included.

Step 4: find the policy behind a policy drop

A policy drop tells you what was blocked but not which policy is involved. Hubble's policy verdict events carry that information. They are a separate event type that records the policy decision for a connection.

hubble observe -P --namespace shop --type policy-verdict --print-policy-names --last 20

For policy verdict events the verdict column reads ALLOWED, DENIED or AUDITED. With --print-policy-names, Hubble appends the policies it could match to the decision, for example DENIED BY block-metadata (CiliumNetworkPolicy). This relies on network policy correlation, which the Helm value hubble.networkPolicyCorrelation.enabled turns on by default.

For the common Policy denied case there is often no name to print, because no rule matched at all. Then look at the policies that put the pod into default-deny. Any policy that selects a pod and has an ingress section switches ingress to default-deny for that pod; the same goes for egress. List the candidates and check which ones select your pod's labels:

kubectl -n shop get ciliumnetworkpolicies,networkpolicies
kubectl get ciliumclusterwidenetworkpolicies
kubectl -n shop get pod -l app=payments --show-labels

The agent on the pod's node can confirm what it actually enforces. cilium-dbg endpoint list shows, for each endpoint, whether ingress and egress enforcement is on, in its POLICY (ingress) ENFORCEMENT and POLICY (egress) ENFORCEMENT columns.

NODE=$(kubectl -n shop get pod -l app=payments -o jsonpath='{.items[0].spec.nodeName}')
AGENT=$(kubectl -n kube-system get pods -l k8s-app=cilium --field-selector spec.nodeName=$NODE -o name)
kubectl -n kube-system exec $AGENT -c cilium-agent -- cilium-dbg endpoint list

A frequent surprise is an egress policy written for one destination that silently cuts off DNS. Once a pod has any egress rule, its lookups to kube-dns are blocked unless a rule allows them. In Hubble that shows up as dropped UDP flows to port 53, which you can spot with --to-port 53. If you use toFQDNs, our guide on restricting pod egress with toFQDNs covers the DNS rule it needs.

Before you push a fix to production, you can test it without risk by turning on policy audit mode (the Helm value policyAuditMode). Traffic that would be denied is then forwarded and reported with the AUDIT verdict, so hubble observe --verdict AUDIT shows exactly what the policy would block.

Step 5: handle Service and routing drops

Service backend not found points away from policy and toward the Service itself. Check that it has ready endpoints:

kubectl -n shop get endpointslices -l kubernetes.io/service-name=payments
kubectl -n shop get pods -l app=payments -o wide

An empty slice usually means the selector doesn't match the pod labels or the pods are failing readiness. There is a less obvious cause too. For a Service with externalTrafficPolicy: Local (or internalTrafficPolicy: Local), Cilium's datapath drops traffic with this reason when the node that receives it has no local backend, even if other nodes have healthy pods. The Helm value serviceNoBackendResponse defaults to reject, so the client typically gets an ICMP error and fails fast instead of hanging.

Stale or unroutable IP during a rollout is usually harmless: clients still hold connections or cached addresses for pods that are gone. If it keeps happening long after the rollout, look for an application that caches pod IPs instead of using the Service name.

Step 6: get the full record as JSON

The compact output hides most of what Hubble knows. For a single flow you want to quote in a ticket, ask for JSON and pick the fields with jq:

hubble observe -P --verdict DROPPED --namespace shop --last 1 -o jsonpb \
  | jq '.flow | {time, node_name, drop_reason_desc, traffic_direction, src: .source.pod_name, dst: .destination.pod_name, IP}'

The record includes the node that dropped the packet, the security identities of both sides and the traffic direction. The node name matters when only some nodes misbehave, because the next step is usually that node's agent logs.

If Hubble Relay itself is broken, you can still see drops on one node from inside its agent. cilium-dbg monitor reads the same event stream before Hubble processes it:

kubectl -n kube-system exec $AGENT -c cilium-agent -- cilium-dbg monitor --type drop

Its lines look like xx drop (Policy denied) to endpoint 25729, identity 261->264: 10.11.13.37 -> 10.11.101.61 EchoRequest, with numeric endpoint IDs and identities that you can map back with cilium-dbg endpoint list.

Step 7: keep an eye on drops after the fix

Ad hoc queries only help while someone is looking. The agent already exports cilium_drop_count_total with reason and direction labels, enabled by default, so a Prometheus alert on a rising rate of reason="Policy denied" catches the next broken policy before users report it. Hubble has its own drop metric too: add drop to hubble.metrics.enabled and it appears as hubble_drop_total with reason and protocol labels, plus optional source and destination context.

Keep Cilium on the latest patch release while you're at it. Cilium 1.20.5, released on 17 November 2026, fixes the source identity reported in drop notifications for traffic to a Service with no backends, which makes the Step 5 case easier to trace. It also fixes egress rules that combine toServices with toPorts: when the referenced Service didn't exist or had no backends, those rules allowed traffic to any destination on those ports.

Test yourself

Ready to test your Cilium knowledge? Try our Cilium quiz!