F5 LTM Module 7: Troubleshooting with tcpdump and Evidence

Cover, mindset, and the three PDF infographics

This page is the full Module 7 deck: F5-BIG-IP-LTM-Module-7.pdf (22 slides) plus the rest of the production module (HTTP, TLS, persistence, iRules, policies, logs, HA, ConfigSync, ten incident scenarios). Nothing from the PDF is skipped.

Module 7 cover: Client, DNS to VIP, Virtual Server, Pool to backend, Response — CLIENT → DNS/VIP → VIRTUAL SERVER → POOL/BACKEND → RESPONSE. Put a check at every hop.

Do not start by changing configuration. Convert “F5 is dropping traffic” into a measurable fact: client SYN on the external VLAN, no server-side SYN in tcpdump.

Quick answer

Find the last known-good packet. On a Standard VS, client TCP and server TCP are two conversations. Green VIP is one probe, not the user path.

TCP failure patterns infographic: handshake, SYN no SYN-ACK, SYN RST, handshake then RST, two BIG-IP handshakes — SYN only = no reply from destination path. RST = someone rejected. Retransmits = expected ACK missing. 3-way complete = TCP worked; the app can still fail.

tcpdump client-side vs server-side infographic scenarios A B C D — A: no client SYN (upstream). B: client SYN, no server SYN (BIG-IP decision). C: server SYN, no SYN-ACK (backend/network). D: server answers, client never gets it (return/client-side).

End-to-end troubleshooting decision tree — Do not ask “what setting should I change?” Ask “where does the traffic stop?”

Problem statement and master flow

Problem statement fields source FQDN VIP port timestamp expected actual changes — Lab: 198.51.100.50 → www.example.com → VIP 192.0.2.100:443. Expected login page. Actual timeout. Note cert/iRule/firewall changes.

Master troubleshooting flow find the last known good point — Ask at every stage: do I have packet evidence this layer works? If no, that is the investigation zone.

Flow · full module order after the PDF

Blue = client path. Orange = BIG-IP match. Green = server path. Purple = TLS/HTTP. Gray = policy/HA.

DNS and client-to-VIP (slides 6–7)

DNS troubleshooting expected 192.0.2.100 — If DNS returns 192.0.2.200 and the VIP is 192.0.2.100, stop. Check cache, split DNS, A vs AAAA, TTL.

PDF slide 6

dig www.example.com
nslookup www.example.com
host www.example.com
# expected A: 192.0.2.100

Client-to-VIP connectivity curl telnet nc tcpdump — If tcpdump on 0.0 shows no SYN from 198.51.100.50, the packet never reached BIG-IP. Check router/FW ACL, client route, NAT, security policy.

PDF slide 7

curl -v http://www.example.com
curl -vk https://www.example.com
telnet 192.0.2.100 443
nc -zv 192.0.2.100 443
tcpdump -nni 0.0 host 198.51.100.50

Virtual Server match (slides 8–9)

Virtual Server troubleshooting dest IP port protocol VLAN pool profiles — Classic miss: client 192.0.2.100:443, VS 192.0.2.100:8443. No match. Also check enabled state, VLAN list, type, pool, SNAT, iRules, policies.

VS matching more specific destination port VLAN protocol — 192.0.2.100:443/TCP matches VS-B on 443, not VS-A on 80. More-specific dest prefix, explicit port, source, VLAN, protocol. K14800: dest/src/port decide precedence.

PDF slides 8–9

tmsh show ltm virtual
tmsh list ltm virtual vs_web_https

Local Traffic › Virtual Servers › vs_web_https

Training mock · not live

Local Traffic › Virtual Servers › vs_web_https

Properties

Destination

192.0.2.100 /32

Service Port

443

Type

Standard

Default Pool

WEB_POOL

Documented VS fields from the PDF. Green here does not prove a client SYN arrived.

Pool, node, monitor (slides 10–13)

Pool WEB_POOL Web01 UP Web02 DOWN Web03 UP — Inspect members, not only the pool icon. Check assigned pool, IP:port, monitor, priority group, connection limits, disabled vs forced offline.

Node is IP, pool member is IP plus port — ICMP to 10.20.20.101 does not prove :443 is listening. Node ≠ pool member.

Health monitor send receive Host header TLS trailing space — Trailing space on the receive string fails silently. Reproduce Host, URI, TLS/SNI, port, timeout, firewall for the monitor source IP.

PDF slides 10–13

tmsh show ltm pool
tmsh show ltm pool WEB_POOL
tmsh list ltm pool WEB_POOL
tmsh show ltm node
tmsh show ltm node 10.20.20.101
curl -v http://10.20.20.101/health
curl -v -H "Host: www.example.com" http://10.20.20.101/health
curl -vk https://10.20.20.101/health
openssl s_client -connect 10.20.20.101:443 -servername app.example.com

Local Traffic › Pools › WEB_POOL › Members

Training mock · not live

Local Traffic › Pools › WEB_POOL › Members

Members

Member	Port	Status
10.20.20.101	443	UP
10.20.20.102	443	DOWN
10.20.20.103	443	UP

Same member mix as PDF slide 10.

Routing, ARP, SNAT (slides 14–17)

Routing troubleshooting longest prefix match 10.30.30.0/24 — Backend 10.30.30.101 needs 10.30.30.0/24 → 10.20.20.1. Missing route = no SYN to backend. Longest-prefix-match still applies.

ARP Incomplete even when route exists — Route can exist while ARP is Incomplete for 10.20.20.101. Wrong VLAN, tag mismatch, VMware port group, duplicate IP, mask.

Asymmetric routing vs SNAT return path — If the server replies via a router that bypasses BIG-IP, TCP never completes. Do not enable Automap blindly.

PDF slides 14–17

tmsh show net route
tmsh list net route
ip route show
tmsh show net arp
ip neigh show
arp -an

Common mistake

Seeing the original client IP on the server means the server must route replies back through BIG-IP. If it does not, that is asymmetric routing — one of the PDF’s most common silent failures.

Connection table, full proxy, TCP, RST (slides 18–22)

Connection table client-side and server-side after SNAT — Client 198.51.100.50:55000→192.0.2.100:443. After SNAT, backend sees 10.20.20.10:32000→10.20.20.101:443. Filter syntax varies by version. K40033505: this is data-plane, not Linux ss/netstat.

Standard VS two independent TCP connections — Do not compare TCP sequence numbers across the proxy. Handshake 1 can succeed while handshake 2 SYN-retransmits.

TCP handshake patterns A no SYN-ACK B SYN RST C reset after handshake — A: SYN SYN SYN = path/firewall/server not answering. B: SYN→RST = closed port or reject. C: handshake then RST = TLS/app/timeout/policy.

TCP retransmission backoff is a symptom not a root cause — t=0,1,3,7s SYN backoff. Symptom, not root cause. Loss, congestion, silent drop, busy server, asymmetry, MTU.

RST sender client BIG-IP backend or intermediate — Never say “F5 reset” without RST source IP and capture location. RST from 10.20.20.101 right after SYN usually means no listener.

PDF captures + connection table

tcpdump -nni 0.0 host 198.51.100.50
tcpdump -nni 0.0 host 10.20.20.101
tcpdump -nni 0.0 'host 198.51.100.50 or host 10.20.20.101'
tcpdump -nni 0.0 tcp port 443
tcpdump -nni 0.0 -s0 -w /var/tmp/app_issue.pcap host 198.51.100.50 and tcp port 443
tmsh show sys connection

F5 tcpdump metadata

Some releases add F5 Ethernet trailers / both-sides capture with interface modifiers (K00463541: :nnn, p). Syntax is version-specific. Confirm on the box. K6546: 0.0 is not rate-limited like switch taps — filter it. Use -n/-nn. Protect PCAPs.

HTTP, Host header, TLS, SSL bridging (the rest of Module 7)

The 22-slide PDF stops at RST. Production work continues: HTTP status source, name-based vhosts, two TLS sessions.

Two independent TLS sessions through BIG-IP — Official SSL admin: Client SSL decrypts the user. Server SSL re-encrypts to the pool. Client padlock does not prove backend TLS.

SSL bridging · two sessions

Source: BIG-IP 17.0 SSL Traffic Management.

HTTP / TLS proof

curl -v http://www.example.com
curl -vk https://www.example.com/
curl http://10.20.20.101/                    # may fail
curl -H "Host: www.example.com" http://10.20.20.101/   # name-based vhost
openssl s_client -connect 192.0.2.100:443 -servername www.example.com
openssl s_client -connect 10.20.20.101:443 -servername app.internal.example

Symptom	First evidence	Usual miss
301/302 loop	`curl -vk` Location vs Host	HTTP profile / iRule redirect vs VS port
403 / 404 / 500 / 503	Who sourced the status (BIG-IP vs app)	Blaming F5 for application 5xx
IP curl fails, Host curl works	Name-based vhost	Monitor without Host
Browser cert warning	SAN, SNI, chain, expiry on Client SSL	Wrong profile or missing intermediate
Direct HTTPS to member works, VIP fails	Server SSL SNI, ciphers, pool port	Direct test skipped LTM functions

Client SSL vs Server SSL

Client SSL: certificate, key match, chain, TLS versions, ciphers, SNI, expiry. Server SSL: backend port, TLS/cipher, SNI sent to backend, cert validation, vhost. Client-side TLS success does not prove backend TLS.

Persistence, iRules, policies, logs, HA

Uneven traffic can be persistence working. Random logouts can be app sessions stored on Web01 after persist expiry. ConfigSync status ≠ failover status.

Confirm option names on your version

tmsh show ltm persistence persist-records
tmsh list ltm rule
tmsh list ltm virtual vs_web_https
tail -f /var/log/ltm
grep -iE 'error|fail|monitor' /var/log/ltm
# also /var/log/messages and /var/log/audit

tmsh show cm failover-status
tmsh show cm sync-status
tmsh show cm device
tmsh show cm device-group
tmsh show cm traffic-group
tmsh show sys performance
tmsh show sys memory
tmsh show sys tmm-info

Check	Question	Do not
Persistence	Cookie / source-IP / timeout / NAT / OneConnect / PGA	Treat pin as a broken LB method first
iRule	Attached? event, Host, URI, pool name, HTTP profile, TLS terminated?	Debug with noisy logs in production
LTM Policy	Attached, enabled, condition order, action, healthy pool	Ignore iRule vs policy ownership
Logs	ltm vs messages vs audit	Assume one file covers every subsystem
CPU/memory	Correlate timestamp, SSL, logging, connections	Assume high CPU = TMM bug
HA	Who is Active? Who owns traffic-group-1? Floating IPs?	Confuse with ConfigSync
ConfigSync	Changes Pending → identify authority, then sync. Disconnected → CS Self IP, route, VLAN, FW, trust, NTP	Sync blindly

HA vs sync

Connection mirroring does not guarantee every application session survives failover. After failover: new Active, traffic-group owner, floating Self IP, VIP, VLAN map, route, SNAT, ConfigSync.

Ten production scenarios + VIP-green infographic

Eight step VIP green but app down workflow ending in backend firewall — Scenario 1 poster: client SYN reached BIG-IP, BIG-IP sent SYN, no SYN-ACK. Cause: backend firewall. Fix: allow 10.20.20.0/24 to :443. Then validate TCP + TLS + HTTP 200.

#	Given	Evidence move	Typical root cause
1	VIP green, timeout	DNS, client SYN, server SYN, TLS, HTTP, iRule	Upstream FW, backend FW, TLS, SNAT return, policy
2	Pool red	Reproduce monitor with Host/TLS/port	Monitor send string ≠ app; service down
3	Client SYN, no backend SYN	VS match, pool, iRule, limits	BIG-IP never opened server-side (infographic B)
4	Backend SYN, no SYN-ACK	Member capture, ARP, listener	Backend/network (infographic C)
5	Server OK, client fails	Return path, client TLS, iRule	Infographic D
6	Direct backend works, VIP fails	Diff source IP, Host, SNI, Server SSL, port	Direct test skipped LTM functions
7	HTTPS fails after cert change	openssl s_client, chain, key, SNI	Wrong Client SSL object or incomplete chain
8	Random logouts	Persist timeout + app session store + HA	Local sessions on one member
9	One server gets all traffic	Persist, NAT, OneConnect, PGA, limits	1000 users behind 203.0.113.10 source-IP persist
10	Works before failover, fails after	Active unit, traffic-group, floating IP, sync, VLAN, SNAT	Standby never had the same path

Traps, lab, interview

Trap	Looks like	Proof
Change config first	SNAT/iRule flapping	Written intake + last known-good packet (slide 2)
Green = healthy	VS/pool Available, timeout	Client SYN and server SYN in tcpdump
Monitor whitespace	Member DOWN, curl works	Exact send/receive + Host (slide 13)
Route without ARP	Route exists, Incomplete	Slide 15
Asymmetric return	Server capture OK, BIG-IP never sees reply	Slide 17
RST means F5	“F5 reset”	RST source IP (slide 22)
One TLS session	Browser padlock, backend fails	Server SSL / openssl to member
Sync to fix failover	Wrong traffic-group owner	failover-status vs sync-status
Wide 0.0 capture	Huge PCAP	K6546 + PDF security warning

Lab task · same PDF topology

1) DNS to 192.0.2.200. 2) VS port 443 vs 8443. 3) Firewall drop on member :443 (scenario C). 4) Strip Host from HTTP monitor. 5) Persist source-IP behind a proxy. 6) Save a filtered PCAP, then delete it.

Interview

“VIP is green, site times out. First packet question?” Did the client SYN reach BIG-IP, and did BIG-IP send a server-side SYN? Weak: “I would bounce TMM.”

Sources

F5-BIG-IP-LTM-Module-7.pdf — all 22 slides (Downloads). Infographics: slides 3, 11, 16.
K6546 tcpdump 0.0
K00463541 tcpdump modifiers
K40033505 connection table
K14800 VS matching
BIG-IP 17.0 SSL Traffic Management

Command flags vary by BIG-IP version. Confirm on the running release.

Module 7: find the last packet. Then name the root cause.

After this page you can