F5 LTM troubleshooting: prove where the packet died

The ticket

Monday 09:12. “HR portal is down. Direct IP works for some people. VIP does not.” That sentence already contains two different outages. If you jump to tcpdump or reboot a pool member, you will waste the change window.

Why two outages? “Direct IP works” means the server and the user-to-server firewall rule are healthy. “VIP does not” means something on the user-to-F5 path, or on the F5-to-server path, is different. Those are not the same packet, so they cannot have the same first fix.

Techclick Fast Track rule from the student workbook: walk the traffic path first. The reason string on tmsh show ltm virtual <name> detail often names the failure. Do not skip to captures until the object path is walked.

Hero · who talks to whom

Words before commands

Memorise these six objects before the runbook. If you mix them, every later CLI line will be read wrong. A student who says “server is down” when they mean “monitor is down” will send the wrong team into the change window.

How F5 forwards traffic (the missing explanation)

A router forwards the same packet. BIG-IP LTM does not. It is a full proxy. That one idea explains almost every “green but broken” ticket in this lesson.

Connection 1 — client side. The user’s laptop sends TCP SYN to the VIP 203.0.113.25:443. F5 answers SYN-ACK from the VIP. The laptop believes it is talking to HR. It is talking to F5.

Connection 2 — server side. After F5 accepts the client (and after SSL / HTTP profiles / iRules run), F5 opens a new TCP session toward a pool member. The source IP on that new session is usually a SNAT / self-IP (10.10.10.10 in this lab), not the laptop IP. The server and FW-INT-01 only see 10.10.10.10.

So there are always two questions, never one:

1. Did the user’s SYN reach the VIP, and did F5 accept it?
2. Did F5’s own SYN reach the pool member, and did the member answer F5?

If you only test from the laptop to the server IP, you skip question 2’s source address. That is why “direct works, VIP fails” is usually a firewall or SNAT problem, not a dead app.

Decide where it died

Decision feel · three sides

How to read the flowchart. First ask only: did the VIP return SYN-ACK? If no, stay on the left (Scenario A). Do not open pool members yet — F5 never received a client to load-balance.

If SYN-ACK is there, the VIP accepted the user. Now the problem is either F5 cannot open the server (middle box: B, C, D) or F5 did open it and HTTP came back wrong (right box: E, F).

The extra diamond “direct server = 200 but F5 curl times out” is Scenario D. It means the app is alive for the laptop’s source IP and dead for F5’s source IP. That is almost always FW-INT-01 or the host firewall, not a missing virtual server.

Which test next

Three vantage points. Same URL. Different source IP. That is the whole method. You change who is speaking, not the application, so each result points at a different box on the diagram.

Think of it as three phone calls to the same shop: from the street (user → VIP), from the shop’s back office (F5 → server), and from a neighbour who already has the shop’s private number (user → server IP). If the neighbour gets through and the back office does not, the shop is open — the back-door lock is the problem.

Six lab scenarios

Side A is the user laptop. Side B is the F5 box. Side C is the server or the firewall team. Every scenario uses all three. Read the explanation first, then the commands. The CLI only confirms the story you already formed.

A · User cannot reach F5

What this is. The first TCP handshake never completes with the VIP. F5 has no client session, so it will never open a server session. Pool health is irrelevant here.

Why it happens. DNS may still point at an old VIP. The virtual server may be disabled. The VS may listen on :443 while the user hits :80. The edge firewall may allow the office LAN to the server subnet but not to the VIP. The self-IP on the client VLAN may be missing, so F5 cannot even ARP for the VIP. Or the VS lives in another route domain.

What you see. Browser spinner. curl says timed out (drop) or connection refused (RST). show sys connection for that VIP is empty. You are still on Connection 1.

Symptom: browser spins. curl from the laptop: connection timed out or connection refused. No row in the F5 connection table.

1. Side A — prove the VIP from the user

   Resolve the name. Then force the VIP so DNS cannot hide a stale record.

   User PC · fail

   curl -vk --connect-timeout 5 https://hr.techclick-lab.in/ \
     --resolve hr.techclick-lab.in:443:203.0.113.25
   * Trying 203.0.113.25:443...
   * Connection timed out after 5001 ms
   * Closing connection
   curl: (28) Connection timed out

   Explanation of this output. Trying 203.0.113.25:443 means DNS (or --resolve) did its job. Connection timed out means no SYN-ACK came back in 5 seconds. A drop on the path, or F5 never answered. It does not prove the pool is down — F5 never got a chance to pick a member.

2. Side B — is the VS listening?

   F5 tmsh · dummy

   tmsh show ltm virtual vs_hr_https
   ---------------------------------------------------------
   Ltm::Virtual Server: vs_hr_https
   ---------------------------------------------------------
   Status
     Availability     : offline
     State            : disabled
     Reason           : The virtual server is disabled
   Destination        : 203.0.113.25:443
   Availability       : offline

   Source: F5 training lab on virtual-server status when children or the VS itself are offline (clouddocs virtual server and pool status). Enable the VS only after you know why it was disabled.

3. Side C — edge path

   If the VS is enabled and still no SYN-ACK, the packet never arrived: wrong DNS, edge firewall, missing self-IP/ARP, or the VS is in another route domain. show sys connection cs-server-addr 203.0.113.25 stays empty.

B · F5 is not reaching the backend

What this is. Connection 1 worked. Connection 2 did not. The user reached F5. F5 cannot open (or keep) a session to the pool member.

Why it happens. F5 has no route to 10.20.30.0/24. The server VLAN is not tagged on this BIG-IP. The pool member IP is typed wrong. SNAT is off, so the server replies to the laptop and the reply never returns through F5. Or F5 has a route, but the next hop is dead.

What you see. VIP handshake succeeds (sometimes even HTTPS lock appears), then the page hangs. From F5 bash, curl to the member fails with “no route to host” or timeout. The connection table may show a client-side flow with no server-side pair.

Symptom: VIP accepts TCP. Browser hangs after ClientHello or after GET. Pool member shows unknown or down. Ping from F5 to 10.20.30.41 fails, or curl from F5 cannot connect.

1. Side B — route and curl from F5

   F5 bash · fail

   curl -v --connect-timeout 5 http://10.20.30.41:80/health
   * Trying 10.20.30.41:80...
   * connect to 10.20.30.41 port 80 from 10.10.10.10 failed: No route to host
   * Closing connection
   curl: (7) Failed to connect

   Explanation. The source 10.10.10.10 in that curl line is F5’s server-side self-IP. “No route to host” means TMM does not know how to leave toward 10.20.30.41 — usually missing VLAN or missing route. Then run tmsh show net route 10.20.30.41. Fix routing before you touch the iRule. An iRule cannot send traffic down a path that does not exist.

2. Side C — SNAT missing

   Server default gateway is the core switch, not F5. Without SNAT, F5 sends to the server using the user’s IP as the source. The server replies straight to the laptop. The laptop already finished TCP with the VIP, so those replies look like stray packets and are dropped. Workbook trap: “Everything green, still fails — return path bypasses BIG-IP (no SNAT).” Turn on SNAT Automap or a SNAT pool so the server must answer 10.10.10.10.

C · Port down, monitor fail, member not reachable

What this is. The server box is alive (ICMP works) but the service F5 is asking for is not. Or the service is up, but the monitor question is the wrong question.

Why a monitor is not ping. A monitor is a small scripted client that F5 runs on a timer. mon_hr sends GET /health HTTP/1.1 plus a Host header, and it expects the word OK in the body. If the app listens on 8080 while the pool member is :80, the monitor gets connection refused and marks the member down. If /health returns a 302 login page, the receive string does not match (it is case-sensitive). Ping never sees any of that — ping only asks “is the IP up?”

Why the virtual server turns red. A VS that load-balances a pool inherits the pool’s health. If every member is down, F5 has nowhere to send Connection 2, so the VS is marked offline even though the VIP IP still exists. Users then get RST or a connection error, which looks like Scenario A until you read the Reason line.

Symptom: both members red. VS goes offline because children are down. Server ICMP is fine.

tmsh show ltm pool pool_hr_80 members
------------------------------------------------------------------
Ltm::Pool Member: pool_hr_80  10.20.30.41:80
------------------------------------------------------------------
Status
  Availability : offline
  State        : enabled
  Reason       : Pool member has been marked down by a monitor

tmsh show ltm monitor http mon_hr
------------------------------------------------
Ltm::Monitor: mon_hr
------------------------------------------------
Destination  : *:*
Interval     : 5
Timeout      : 16
Send String  : GET /health HTTP/1.1\r\nHost: hr.techclick-lab.in\r\n\r\n
Recv String  : OK

How to read that output. Reason marked down by a monitor means F5 asked /health and did not get OK in time. It does not mean the node is unplugged. Next, curl the same URL F5 uses. Workbook rule: if curl to the member succeeds and the monitor fails, the fault is the monitor definition (send/recv/Host). If curl also fails, the fault is the server or the path.

On the Linux box, nothing is bound to :80. App listens on 8080. Or /health returns a redirect and the receive string never matches.

ss -lntp | grep ':80'
# empty
ss -lntp | grep 8080
LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=2214,fd=18))

D · Direct to server works — via F5 fails

What this is. The application is healthy. The user’s laptop is allowed to reach it. F5 is not. You are looking at two source IPs hitting the same destination.

Why the firewall does this. Security teams write rules like “office LAN → HR servers :80.” They forget the second speaker. When traffic goes through F5, the server no longer sees 192.168.x.x from the laptop. It sees 10.10.10.10 (SNAT). FW-INT-01, or firewalld on the host, has no line for that IP, so it drops or silently discards the SYN. The user test still works because that test never uses 10.10.10.10.

Why this fools people. The VIP, pool, and monitor can all be green if the monitor is sourced from a different self-IP that is allowed, or if the monitor is ICMP. Users still fail. The Fast Track book calls this the most common LTM pattern: symptom visible to the user, invisible to every green icon in the room.

This is the interview gold. Laptop curl to 10.20.30.41:80 is 200. Same page through the VIP hangs. Do not rebuild the virtual server first.

Direct vs via ADC

# From laptop (works)
curl -sI http://10.20.30.41:80/health
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 2

# From F5 (fails)
curl -sI --connect-timeout 5 http://10.20.30.41:80/health
curl: (28) Connection timed out

How to explain the two curls. Same destination, same port, same URL. Only the source changed. Laptop source is allowed. F5 source is not. Ask firewall: allow 10.10.10.10 → 10.20.30.41/32:80 and .42 on FW-INT-01. Also check firewalld on the host if it only trusts the office LAN. Source: Techclick Fast Track trap — if the server firewall or ACL only permits the client or the VIP, the F5 source fails even when users can hit the server directly.

E · Wrong iRule on the virtual server

What this is. Both TCP connections work. HTTP is being rewritten or dropped after F5 accepts the user. The objects look green because monitors never run the iRule path your user hits.

Why iRules can break a green pool. An iRule is Tcl that runs on events such as HTTP_REQUEST. It can pick another pool, change the URI, or drop the flow. F5 evaluates switch / if from the top. The first match wins. If /app1* is listed before *.gif, then /app1/smile.gif is sent to the app servers, which have no image files — 404. The image pool never sees the request. That is the DevCentral teaching example.

Why a TCL error resets the user. If the iRule reads a variable that does not exist, F5 logs 01220001:3: TCL error and aborts that iRule instance. F5 documents that this may reset the associated connection. The pool stays green. Only that request dies. Look in /var/log/ltm, not at the member status.

VS is green. Pool is green. Some URLs 200, images 404, or the connection resets.

tmsh list ltm virtual vs_hr_https rules
ltm virtual vs_hr_https {
    rules { irule_hr_path }
}

# WRONG — /app1* matches before images
when HTTP_REQUEST {
  switch -glob [string tolower [HTTP::uri]] {
    "/app1*" { pool pool_hr_80 }
    "*.gif" -
    "*.jpg" { pool pool_hr_img }
    default { pool pool_hr_80 }
  }
}

01220001:3: TCL error: /Common/irule_hr_path <HTTP_REQUEST> -
can't read "foo": no such variable     while executing "set x $foo"
# Connection reset for that request

How to fix it. Put the more specific match first (images before /app1*). In lab only, add log local0. "URI=[HTTP::uri]" so you can see which branch fired. Remove or wrap that log before production — DevCentral warns debug logs fill the disk. Re-test one GIF and one app page.

curl -sI https://hr.techclick-lab.in/app1/smile.gif \
  --resolve hr.techclick-lab.in:443:203.0.113.25
HTTP/1.1 200 OK
X-Pool: pool_hr_img

F · curl returns 200 — students think it is fixed

What this is. Your test is too small. HTTP 200 (RFC 9110 / 2616) means “this request was understood and accepted.” It does not mean “HR works for every employee on every URL.”

Why students get fooled. curl -I http://203.0.113.25/ is a HEAD to the VIP IP on port 80 with no Host name. F5 can answer that from the VS itself (empty 200, default page, or a different pool). Real users open https://hr.techclick-lab.in/app1/login — different scheme, port, SNI, Host header, and path. An iRule may only break /app1/auth POST. A firewall may only allow the F5 self-IP, so your curl from F5 is 200 while users time out.

How to test like a user. Same name, same HTTPS port, same Host, same path, and if login is the complaint, a POST or a real browser. Then look at the connection table: you must see both the client-VIP pair and the SNAT-member pair.

curl -sI http://203.0.113.25/
HTTP/1.1 200 OK
Server: BigIP
Content-Length: 0
# This is the VS answering. It is not HR.

curl -sk https://hr.techclick-lab.in/app1/login \
  --resolve hr.techclick-lab.in:443:203.0.113.25 \
  -o /tmp/body -w '%{http_code} %{url_effective}\n'
302 https://hr.techclick-lab.in/app1/login
# Now you tested the real host, path, and scheme.

Runtime path

After go-live the packet is always the same. Memorise the chain used in Techclick F5 interview prep:

VLAN → virtual server → iRule / LTM policy → persistence → load-balancing method → pool member (monitor-checked) → SNAT → server.

Read that as a story. The packet arrives on a VLAN. The VIP is the front door. The iRule may change the destination. Persistence may pin the user to last week’s member. The load-balancing method picks among green members only. SNAT writes the source the server will see. If you skip a step in your head, you will skip it in the CLI.

Ops · proof after the fix

tmsh show sys connection cs-server-addr 203.0.113.25
Sys::Connections
203.0.113.80:53122  203.0.113.25:443  10.10.10.10:53122  10.20.30.41:80  tcp  12

How to read one connection line. Left pair is the user talking to the VIP. Right pair is F5 talking to the member. 203.0.113.80:53122 → 203.0.113.25:443 is Connection 1. 10.10.10.10:53122 → 10.20.30.41:80 is Connection 2. If the last two columns never appear, you are still in Scenario B or D — F5 accepted the user and never completed the server side.

Traps and proof

Workbook extras worth keeping in your pocket: HA pair out of sync after failover; SSL client vs server profile mismatch; persist pinning users to a dying member. Those are sister lessons — see the related F5 503 / SNAT lab after this quiz.

F5 LTM series: F5 LTM troubleshooting path · F5 SNAT concept and issues · F5 cookie vs source-address persistence · F5 SSL offload vs passthrough.

Sources

• Techclick Infosec, F5 BIG-IP LTM Troubleshooting Fast Track (3-day student workbook PDF) — walk the path first; monitor vs curl; SNAT return-path trap.
• Techclick Infosec, F5 Troubleshooting 3-Day Plan — tmsh show ltm virtual detail reason string.
• F5 training: Virtual server and pool status when members are down.
• F5 301b notes: Identify and resolve application issues (iRule events, TCL errors, logging).
• F5 DevCentral, iRules 101 — debugging / log local0. (do not leave debug in production).
• F5 sol13905 class notes — TCL error may reset the associated connection.
• RFC 9110 / 2616 — HTTP 200 means that request succeeded, nothing more.
• F5 tmsh reference: ltm pool.

Related: F5 SNAT concept and issues · F5 LTM 503 / SNAT evidence lab · F5 iRules — zero to production · LTM virtual servers and pools · Command ladder cheatsheet.