FGCP HA heartbeat then session pickup

Lessons · FGCP design (A-P vs A-A) · Failover operations

Hero · one primary, two heartbeat cables

Why “HA is up” still drops TCP

The day-one ticket is always the same: “HA is configured, so sessions should survive.” Wrong. Three different jobs share the word HA, and only one of them is session failover.

• Device failover — primary dies or stops sending heartbeats. The subordinate becomes primary. That is FGCP. It does not require session-pickup.
• Link failover — a monitored data interface goes down. That unit is no longer eligible to stay primary. Heartbeat can still be perfect.
• Session failover — session-pickup was enabled before the event, so the new primary already has the TCP table. Without it, every session restarts.

Heartbeat first, then pickup

FGCP members share one virtual identity on the data plane: same virtual MAC, same configured IPs. The subordinate is not a second hop. It is a hot spare that must be told, continuously, that the primary is still alive.

That telling is the heartbeat. Official default hold-down is 1.2 seconds: hb-interval 2 × 100 ms × hb-lost-threshold 6. Each received heartbeat resets the timer. If the timer runs out, the subordinate assumes the primary is gone and negotiates to become primary.

Device, link, session failover

FGCP supports those three protections. They stack. Device and link failover happen whether or not you enabled pickup. Session failover does not happen unless pickup was already on and the session type is one FGCP will sync.

Journey · heartbeat, detect, promote, then pickup

Pickup, monitor, override

These three knobs decide whether a failover is useful or a second outage. Set them on purpose. Leave the rest at documented defaults unless you can name the failure they fix.

Runbook Side A / B / C

Two FortiGates, same model and firmware. Mode Active-Passive. Dedicated heartbeat on ha1 and ha2 (or two unused physical ports if the chassis has no HA ports). Data: wan1 and lan. Session-pickup on. Override off. Group name and password identical. Do not pick a cute group-id — leave default 0 unless you already have a matching standard.

Side A — cables / heartbeat / split-brain

1. Two physical hbdevs, member to member

   Cross-cable ha1↔ha1 and ha2↔ha2. Official two-member recommendation: double back-to-back heartbeat, not through a switch. hbdev cannot be VLAN, LAG, IPsec, switch port, or redundant interface. Source: HA heartbeat interface.

2. Do not share the only hello path

   One hbdev through a switch is split-brain waiting. If heartbeat is lost both ways, both units become primary and apply the same virtual MAC. Console both boxes before you change override.

Side B — product (System → HA)

1. Mode, identity, heartbeat

   System > HA — Mode Active-Passive. Same group name and password on both. Heartbeat interfaces ha1 and ha2. Device priority may differ (higher prefers primary only if override is on, or as a later tie-break). Source: HA active-passive cluster setup.

2. Session pickup

   Cluster Settings → enable Session pickup. CLI: set session-pickup enable. Add set session-pickup-connectionless enable if UDP/QUIC matters. Source: Session pickup.

3. Monitored interfaces

   Monitor wan1 and lan (or the two data links whose death should move the cluster). CLI: set monitor "wan1" "lan". Do not monitor a link that already flaps.

4. Leave override disabled unless failback is the design

   Default: uptime beats priority (after monitored-link count). A failed monitored interface resets HA uptime to 0. Override enabled: priority beats uptime — the original primary takes the job back when its links recover. Source: HA primary unit selection criteria.

config system ha
    set group-name <same-on-both>
    set mode a-p
    set password <same-on-both>
    set hbdev "ha1" 50 "ha2" 50
    set session-pickup enable
    set session-pickup-connectionless enable
    set override disable
    set monitor "wan1" "lan"
end

Side C — proof

1. Identity of the cluster

   get system ha status on both consoles. Health OK. Mode HA A-P. Number of members = 2. One line Primary, one Secondary. Configuration Status in-sync (checksums match). ses_pickup: enable. HBDEV ha1/ha2 up. MONDEV wan1/lan up. CLI prompt should read (Primary) on one box and (Secondary) on the other.

2. Start a long TCP, then fail over

   Hold an SSH or HTTPS session through the cluster (not to the FortiGate GUI). Then force failover in a window: GUI System > HA → member → Diagnostics and Tools → Actions → Force HA failover, or CLI execute ha failover set 1. Confirm execute ha failover status shows set. The new primary’s get system ha status Primary selected using mentions the EXE_FAIL_OVER flag.

3. Did the session live?

   If pickup is on and the flow is pass-through TCP without proxy UTM, the session should continue. Then execute ha failover unset 1 so election is normal again. Source: Force HA failover for testing and demonstrations. Do not leave the flag set.

get system ha status
# HA Health Status: OK
# Mode: HA A-P
# ses_pickup: enable
# override: disable
# Configuration Status: … in-sync
# HBDEV stats: ha1 up, ha2 up
# MONDEV stats: wan1 up, lan up
# Primary selected using: <timestamped reason>

execute ha failover set 1
execute ha failover status
# failover status: set

get system ha status
# Primary selected using: … EXE_FAIL_OVER flag set

execute ha failover unset 1

One failover after go-live

After the pair is in-sync, a real event does this:

1. Trigger: chassis down, hbdev silent past 1.2 s, or a monitored interface down (uptime on that unit resets to 0).
2. Remaining unit wins primary election (override off: monitored-link count → uptime if the gap is > 5 minutes → priority → serial).
3. New primary GARPs the virtual MAC on data interfaces (arps / arps-interval). Neighbors rewrite CAM. If GARPs are ignored, link-failed-signal can bounce the old primary’s ports for 1 s.
4. If session-pickup was on, pass-through TCP is already in the table. Proxy UTM and on-box sessions reconnect. UDP needs connectionless pickup.

Traps + proof

Proof · Health OK is a line, then a session that lived

Sources

• FortiOS 8.0 — FGCP (cluster, same model/firmware, heartbeat)
• FortiOS 8.0 — HA active-passive cluster setup (System → HA fields)
• FortiOS 8.0 — HA heartbeat interface (hbdev, physical-only, split-brain if hellos stop)
• FortiOS 8.0 — HA heartbeat (Ethertype 0x8890 / 0x8893)
• FortiOS 8.0 — Session pickup (TCP default path, connectionless / QUIC, delay)
• FortiOS 8.0 — Pass-through sessions (proxy UTM not failed over; flow UTM uninspected after failover)
• FortiOS 8.0 — HA primary unit selection criteria (override, 5-minute uptime window, monitored-link reset)
• FortiOS 8.0 — Check HA synchronization status (get system ha status, HBDEV / MONDEV, in-sync)
• FortiOS — Force HA failover for testing (execute ha failover set <cluster_id>; id 1 when not vcluster)
• FortiOS 8.0 — GUI HA actions (Force HA failover / Undo)
• Fortinet Community — Heartbeat packet lost (default 1.2 s = interval × 100 ms × lost-threshold)

Related: FGCP A-P vs A-A / split-brain · Policy lookup + NAT · Security profiles · Troubleshooting command center · FortiGate interview