Operational Failures — When the Firewall Breaks at 3 AM

Before you reach for the keyboard — name the failure first

Production outages on a PA are rarely "the firewall died". 90% of the time you're chasing one of four patterns. Naming the pattern correctly in the first 60 seconds cuts MTTR by an order of magnitude. The framework: Symptom → Counter → CLI → Fix. Every failure below follows it.

① HA Heartbeat Flap — pair flips every two hours

Rahul at TCS gets paged: PA-5250 active firewall flipped to passive at 02:14, back to active at 04:18, and again at 06:21. Sessions resync, customers complain about momentary disconnects. The HA1 link itself shows "up". What's happening?

HA1 carries heartbeats and config sync on TCP/28771 + UDP/28769. HA2 carries session-sync. HA1 saturation — a noisy management VLAN, a duplex mismatch, or a switch buffer overflow on the intermediate switch — causes heartbeats to be delayed beyond the hello-interval timeout, even though the link technically stays "up". The peer declares the active dead, takes over, and the original active recovers a second later. Result: an oscillation every time the buffer fills.

② Commit Failures — "unknown error" and other crimes

Priya at HCL hits Commit at 4 PM Friday. The job spins for 28 minutes, then fails with: "Commit failed: unknown error". The candidate config has 14 changes she can't undo individually. Now what?

Commit failures come in three flavours. Knowing which flavour you have decides whether to revert, edit, or call TAC:

show jobs id <job-id>
show jobs all
show config-lock
debug swm status
show system resources follow

Job ID    Type    Status     Result    Description
4231      Commit  FIN        FAIL      validation Error: address-group 'web-servers'
                                       references missing address 'web-04' in rule
                                       'allow-web-prod' line 47

③ Dataplane CPU Spike — find the elephant in 4 commands

Aditya at Wipro sees dashboard alerts: Dataplane CPU 96% sustained. Latency on production-bound flows jumps from 8ms to 180ms. Drops on the WAN side. The dataplane is choking — but on what? An IPS overload? A massive DDoS? A single misbehaving flow?

show running resource-monitor second
show running resource-monitor ingress-backlogs
show running resource-monitor | match -A 4 "Packet Buffer"
show counter global delta yes | match drop

④ Log Forwarding Drop — when the SIEM goes quiet

Sneha at Infosys gets a Sev-2 ticket: "No traffic logs from PA-5250 in Splunk for the last 4 hours." She checks Monitor → Traffic → logs scroll fine on the firewall. System logs say "syslog forwarding enabled". The PA looks healthy. So where are the logs going?

This is the most dangerous failure of the four — because nothing visible breaks. The firewall keeps blocking and allowing traffic. Users notice nothing. The SOC has a gap. Detection misses an attack window.

Two failure modes dominate:

(1) Queue full. Logs generate faster than the management plane can ship them out. The forwarding queue fills, new logs are silently discarded. Counter: log_traffic_loss_queue_full.

(2) Connection silently dead. The syslog TCP socket reconnects in the background, the firewall thinks it's connected, but the SIEM closed the socket. The session ages out hours later. Logs in this window are lost.

debug log-receiver statistics
show counter global filter aspect log | match drop
show log-collector all
show logging-status
ping host <syslog-server>

Log incoming rate ............ 2,400 logs/sec
Log forwarding rate .......... 1,820 logs/sec   <-- shipping slower than generation
log_traffic_loss_queue_full .. 178,400          <-- losses confirmed
Syslog enqueue count ......... 9,224 (queue ceiling near)