topic Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu in SD-WAN

SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failure

WiliRGasparetto — Wed, 17 Jun 2026 17:16:23 GMT

SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failure

One of the most challenging SD-WAN troubleshooting scenarios is not simply proving that an ISP link went down.

The real challenge is proving the entire failover chain:

Did SD-WAN detect the failure?
Did the correct SD-WAN rule match?
Was a new path selected?
Did the VPN transport change correctly?
Did routing and next-hop selection follow the SD-WAN decision?
Did SecureXL keep stale acceleration state?
Did the firewall drop packets because the connection changed path?
Did the VPN overlay recover on the backup link?

In branch-to-branch environments, the symptom usually looks simple:

“When the primary link fails, the VPN tunnel between offices drops.”

But the root cause can be in several different layers.

This post shows a process-oriented way to troubleshoot SD-WAN failover when VPN tunnels between offices are impacted.

1. Example Scenario

Consider a simplified environment:

Branch-A Gateway
LAN: 10.10.10.0/24
Test Host: 10.10.10.100

Branch-B Gateway
LAN: 10.20.20.0/24
Test Server: 10.20.20.200

Branch-A ISP1: Primary link
Branch-A ISP2: Backup link

Branch-B ISP1: Primary link
Branch-B ISP2: Backup link

SD-WAN policy:
Prefer ISP1
Fail over to ISP2 if SLA fails
VPN overlay between Branch-A and Branch-B

The reported issue:

When ISP1 fails on Branch-A, traffic to Branch-B stops.
The VPN tunnel appears to drop or does not recover correctly through ISP2.

A common mistake is to debug only VPN.

In SD-WAN, VPN is only one part of the chain.

The better question is:

Where did the failover chain break?

2. The SD-WAN Failover Chain

For a VPN tunnel over SD-WAN to survive a link failure, several things must happen correctly:

If any step fails, the symptom can still look like:

"The VPN tunnel dropped.”

But the real root cause may not be VPN.

3. Start with a Scoped Flo

Before enabling broad debug, define one controlled test flow.

Example:

Source: 10.10.10.100
Destination: 10.20.20.200
Service: TCP/443
Protocol: TCP

The goal is to follow one packet flow during:

Normal state
Primary link failure
SD-WAN failover
VPN recovery
Traffic restoration

4. Clean Previous Debugs

Always start by clearing previous kernel debug:

bash
fw ctl debug 0

This avoids mixing old messages with the current test.

In failover troubleshooting, timing is critical. You need a clean capture of the exact transition:

Primary path healthy
Primary path fails
SD-WAN detects SLA failure
Backup path is selected
VPN transport changes
Traffic resumes or fails

5. Apply a Simple Debug Filter

Use a filter to reduce noise and focus on the test flow.

Example:

bash

fw ctl set string simple_debug_filter_saddr_1 10.10.10.100 -a
fw ctl set int simple_debug_filter_sport_1 0 -a
fw ctl set string simple_debug_filter_daddr_1 10.20.20.200 -a
fw ctl set int simple_debug_filter_dport_1 443 -a
fw ctl set int simple_debug_filter_proto_1 6 -a

Meaning:

Source IP: 10.10.10.100
Source port: any
Destination IP: 10.20.20.200
Destination port: TCP/443
Protocol: TCP

The `-a` applies the setting across relevant CoreXL workers.

This is important because without filtering, SD-WAN, VPN, firewall and SecureXL debug can generate too much noise to be useful.

6. SD-WAN Kernel Debug

Enable SD-WAN dataplane debug:

bash

fw ctl debug -m SDWAN all

fw ctl debug -m SDWANRB all

The `SDWAN` module helps understand how the SD-WAN dataplane handles the flow.

The `SDWANRB` module helps understand SD-WAN Rule Base evaluation.

This helps answer:

Did the flow enter SD-WAN logic?
Which SD-WAN rule matched?
Was the expected application/service/destination matched?
Was the failover rule evaluated?
Was a new transport selected after the link failure?

This distinction is important because many failover issues are actually rule-matching problems.

Examples:

The traffic matched a more generic SD-WAN rule.
The destination object did not match as expected.
The application was not identified at the time of decision.
The fallback behavior was not configured as assumed.
The gateway was still using an older SD-WAN configuration.

7. SD-WAN Steering Process Debug

Kernel debug shows dataplane behavior.

The `sdwan_steering` debug helps explain the steering logic.

A controlled option:

bash
fw debug sdwan_steering on TDERROR_SDWAN_SDWAN=1

A more verbose option, usually for a short window or TAC-guided troubleshooting:

bash
fw debug sdwan_steering on TDERROR_SDWAN_ALL=5

This helps answer:

Did SD-WAN detect the link degradation?
Did SLA state change?
Was the backup ISP considered healthy?
Was the steering table updated?
Was the selected path changed?
Did the process receive the latest SD-WAN policy?

The relevant logs are usually in:

bash
$FWDIR/log/sdwan_steering.elg*

Preserve them after the test:

bash
mkdir -p /logs/sdwan_debug_$(date +%F_%H%M)
cp $FWDIR/log/sdwan_steering.elg* /logs/sdwan_debug_$(date +%F_%H%M)/

8. Force SD-WAN Configuration Refresh

If the behavior does not match the expected policy, force SD-WAN to fetch the latest configuration:

bash
cpsdwan fetch_new

This is useful when you need to verify whether the gateway is using the expected SD-WAN policy and steering configuration.

Before repeating the failover test, validate whether a configuration refresh changes the behavior.

9. Firewall Drop and Connection State Debug

A VPN failover can fail even when SD-WAN selected the correct backup path.

Enable firewall drop and connection-state debug:

bash
fw ctl debug -m fw + drop conn

This helps answer:

Is the firewall dropping packets after the path changes?
Is there a state mismatch?
Is the return traffic asymmetric?
Is NAT changing during failover?
Is an old connection state still tied to the previous path?

During SD-WAN failover, existing connections may behave differently from new connections.

This is especially important when users report:

New sessions work after failover, but existing sessions break.

That may indicate a state, NAT, acceleration or path-change issue rather than a pure VPN failure.

10. Route Decision Debug

Enable route debug:

bash
fw ctl debug + route

This helps answer:

After failover, which next-hop is selected?
Is the gateway still trying to use the failed ISP?
Is there a more specific route overriding the SD-WAN decision?
Is the return path asymmetric?
Does the selected ISP match the expected interface?

In SD-WAN environments, route, next-hop and interface mapping must be consistent with the steering decision.

11. VPN Debug

If the SD-WAN path changes but the tunnel does not recover, enable VPN debug:

bash
fw ctl debug -m VPN all

If VPN acceleration is suspected:

bash
fwaccel dbg -m vpn all

This helps answer:

Did the VPN tunnel try to re-establish over the backup link?
Was the correct VPN transport selected?
Did IKE/IPsec negotiation fail?
Was the peer reachable through the new ISP?
Was NAT-T involved?
Did the encryption domain match?
Did the tunnel go down because of path failure or because negotiation failed?

In SD-WAN troubleshooting, do not assume VPN is the root cause.

Use VPN debug after validating SD-WAN rule matching, steering, routing and transport selection.

12. SecureXL / Acceleration Debug

In some cases, the traffic path is affected by acceleration.

Enable SecureXL debug:

bash
fwaccel dbg -m default all
fwaccel dbg -m sdwan all

This helps answer:

Was the flow accelerated?
Did SecureXL keep stale state?
Did the flow remain associated with the previous path?
Did new flows behave differently from existing flows?
Is SD-WAN logic reflected correctly in the accelerated path?

This is especially relevant when:

Failover works for new connections but not for existing flows.

13. Increase Debug Buffer and Capture with Timestamp

Before reproducing the issue, increase the debug buffer:

bash
fw ctl debug -buf 32000

Then capture kernel debug with timestamps:

bash
fw ctl kdebug -T -f > /var/log/kern_dbg_sdwan_failover &

The timestamp is important because you need to correlate:

time of link failure
time of SLA state change
time of steering decision
time of VPN event
time of traffic drop
time of recovery

After the test, stop the background capture:

bash
ps -ef | grep "fw ctl kdebug"
kill <PID>

Then stop debugs:

bash
fw ctl debug 0
fwaccel dbg resetall
fw debug sdwan_steering off TDERROR_ALL_ALL=0

Do not leave debug enabled in production.

14. Collect SD-WAN Internal Tables

For advanced troubleshooting or TAC escalation, it is useful to collect the internal SD-WAN kernel tables.

These tables provide a snapshot of the effective SD-WAN state inside the gateway. They help validate whether the SD-WAN policy, ISP mappings, VPN transports, NAT/DAIP mappings and selected paths were correctly pushed and loaded into the dataplane.

bash
{
fw tab -t sdw_gws_addrs_set1
fw tab -t sdw_gws_addrs_set2

fw tab -t sdw_vpn_nat_mapping_set1
fw tab -t sdw_vpn_nat_mapping_set2

fw tab -t sdw_vpn_transp_ids_set1
fw tab -t sdw_vpn_transp_ids_set2
fw tab -t sdw_vpn_transp_set1
fw tab -t sdw_vpn_transp_set2
fw tab -t sdw_selected_vpn_transp_set1
fw tab -t sdw_selected_vpn_transp_set2

fw tab -t sdw_next_hops_set1
fw tab -t sdw_next_hops_set2

fw tab -t sdw_vpn_daip_mapping_set1
fw tab -t sdw_vpn_daip_mapping_set2
fw tab -t sdw_vpn_daip_nat_mapping_set1
fw tab -t sdw_vpn_daip_nat_mapping_set2
fw tab -t sdw_vpn_daip_local_ips_set1
fw tab -t sdw_vpn_daip_local_ips_set2

fw tab -t sdw_vpn_dynamic_nat_set1
fw tab -t sdw_vpn_dynamic_nat_set2
fw tab -t sdw_vpn_dynamic_nat_sync

fw tab -t sdw_isp_uuid_to_id1
fw tab -t sdw_isp_uuid_to_id2
fw tab -t sdw_isp_info1
fw tab -t sdw_isp_info2
fw tab -t sdw_isp_id_isp_string_id1
fw tab -t sdw_isp_id_isp_string_id2
fw tab -t sdw_ifnum_isp_string_id1
fw tab -t sdw_ifnum_isp_string_id2
} > /var/log/sdw_tables_output.txt 2>&1
```

The output file can then be collected together with `sdwan_steering.elg`, kernel debug and VPN logs.

What these table groups help validate

The `set1` and `set2` tables should both be collected because SD-WAN can maintain parallel internal table sets. Depending on the update state, failover state or configuration refresh, one set may contain the active information while the other may represent an alternate or previous state.

This collection helps answer key questions during failover troubleshooting:

Does the gateway know all expected SD-WAN peers?
Are the VPN transports present?
Which VPN transport is selected?
Is the backup ISP mapped correctly?
Is the selected next-hop valid?
Are NAT / DAIP mappings correct?
Does the interface-to-ISP mapping match the expected design?
Was the SD-WAN configuration effectively loaded into the dataplane?

This is especially useful when the SD-WAN policy looks correct in the management plane, but the gateway behavior during failover does not match the expected design.

16. Practical Troubleshooting Logic

When a VPN tunnel drops after SD-WAN failover, I usually follow this logic:

1. Did SD-WAN detect the link failure?
2. Did the expected SD-WAN rule match?
3. Was the backup ISP considered healthy?
4. Was a new transport or next-hop selected?
5. Did VPN try to use the new transport?
6. Did NAT / DAIP mapping change correctly?
7. Did the firewall drop packets due to state, NAT or asymmetry?
8. Did SecureXL preserve stale state?
9. Did new flows work while old flows failed?
10. Did the SD-WAN internal tables match the expected configuration?

This avoids jumping directly to “VPN is broken”.

17. Evidence Package for TAC or Community Analysis

When opening a TAC case or asking the community, provide:

Gateway version and Jumbo Take
Management version
SD-WAN deployment type
Cluster or standalone
ISP/link design
VPN topology
Source/destination test flow
Exact timestamp of the failover test
Which link was failed
Expected path
Actual path
Whether new sessions work after failover
Whether existing sessions fail
Kernel debug file
sdwan_steering.elg files
SD-WAN internal tables output
Relevant VPN logs
Relevant fwd.elg / messages output

This improves the quality of the analysis and avoids generic troubleshooting.

18. Final Thought

In SD-WAN, a VPN tunnel dropping after link failure is not always a VPN problem.

It can be:

SD-WAN rule matching
SLA detection
steering decision
transport selection
routing
NAT / DAIP mapping
CP gateway state
firewall connection state
SecureXL acceleration
VPN negotiation

The goal is not to prove only that the link failed.

The goal is to prove the full chain:

That is the difference between troubleshooting a symptom and troubleshooting the SD-WAN architecture.

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

PhoneBoy — Wed, 17 Jun 2026 21:39:22 GMT

Nicely done!

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

Amit_Navon — Thu, 18 Jun 2026 09:11:57 GMT

Great post @WiliRGasparetto

Section 16 is true gold!

We are trying to simplify debugging with the new observability tools.

By the way - what was the issue in your recent debug session?

BR,

Amit

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Thu, 18 Jun 2026 10:48:38 GMT

Thank you @PhoneBoy

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Thu, 18 Jun 2026 10:58:25 GMT

Here, we used this approach to identify why SD-WAN was not failing over to the secondary link.

We collected all the necessary evidence, but in the end, we discovered that the root cause was a bug in Quantum Spark.

Even so, I believe this troubleshooting workflow is worth sharing with everyone, as it can be very useful for identifying potential SD-WAN issues — or even confirming when SD-WAN is not the actual root cause

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

AlmogD1 — Thu, 18 Jun 2026 11:47:15 GMT

Great write-up. I fully agree with the main point here: when a VPN tunnel drops after an SD-WAN failover, the right approach is not to troubleshoot VPN only, but to validate the full chain: ISP/link state, SLA status, steering decision, selected path, VPN overlay behavior, routing/interface selection, firewall logs, and possible VPN/SecureXL/state-related issues.

Today, SD-WAN Observability already helps with the first part of this investigation by giving visibility into gateway status, link status, ISP status, SLA quality, and overlay tunnel health.

We are also working on a new, more comprehensive SD-WAN Observability dashboard that is designed to make this type of troubleshooting much easier. The goal is to bring the underlay, overlay VPN, SD-WAN steering, VPN health, traffic/session details, and raw logs into one place.

Some examples from the upcoming dashboard:

• Link and gateway health:
- Gateway status
- Link / ISP status
- Gateway SLA status
- Underlay health score
- Top link swappers by gateway

• Overlay and tunnel health:
- Tunnel status per gateway and peer
- Overlay VPN SLA
- Overlay tunnel MOS
- Overlay latency, jitter, and packet loss
- Overlay health score

• VPN-specific troubleshooting:
- VPN Overview & Drops, including IKE peers, total SAs, IKE SAs, IKE errors, kernel limit, overlay drops, and fragmentation drops
- VPN Events per Minute by Type, to identify spikes during failover or instability
- VPN Restarts, including IKED, VPND, VPNRAD, and policy restarts
- IKE SA Counts by Type
- IKE SA Usage vs Limits
- Concurrent IKE Negotiations
- Overlay Tunnel Connection Drops per Peer / ISP
- IPSec Fragmentation Count & Drops, which can help identify MTU/path issues

• Traffic and log correlation:
- VPN Gateway Connections table showing gateway, outgoing ISP, peer gateway, peer IP, user, source, destination, service, SD-WAN steering rule, and steering object
- Raw Logs data table showing the actual log actions such as Accept, Encrypt, Drop, or Reject, together with interface and direction details

This helps answer many of the questions raised in the post:

Did the ISP/link go down?
Did SLA quality degrade?
Did SD-WAN select a new path?
Which ISP was actually used?
Did the overlay tunnel move to the expected path?
Did VPN events or IKE errors spike during the failover?
Were there overlay drops or fragmentation drops?
Did the tunnel flap for a specific peer or ISP?
Did the traffic match the expected SD-WAN steering rule?
Did the firewall log show Accept, Encrypt, Drop, or Reject?

In addition, In R82.20 we are releasing an SD-WAN Events tab that will enable us to better understand what SD-WAN steering did and why. Is the link swap being due to a link / tunnel status change or due to better link quality. We can also pinpoint when the failure occurred and which steering objects and traffic affected.

This will not replace deep debug in every case, especially for low-level SecureXL, IKE/IPsec, NAT, or connection-state investigations, but it should significantly reduce the time needed to locate where the failover chain broke. It also gives the user and TAC/community a much better evidence package before moving into kernel debug.

In summary, with R82.20, Check Point is introducing a new on-premises monitoring capability that brings link metrics, gateway status, session details, logs, VPN/overlay visibility, and SD-WAN events into a single dashboard.

This unified view helps customers correlate underlay, overlay, and traffic data in one place, reducing the time and effort required for root-cause analysis and troubleshooting.

More details will be shared as the enhanced dashboard becomes available.

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Thu, 18 Jun 2026 19:47:21 GMT

Great point, @AlmogD1 .

I am really looking forward to testing these improvements planned for R82.20, especially in the context of SD-WAN Observability.

I believe this evolution can represent a major step forward in terms of visibility and troubleshooting for SD-WAN environments. Having this level of integrated observability can greatly help with day-to-day debugging, especially when we can correlate link status, ISP behavior, SLA quality, SD-WAN steering, overlay VPN health, SD-WAN events, sessions, and firewall logs in a single dashboard.

As soon as a test version becomes available, I would be very interested in validating these capabilities in a lab environment and exploring real troubleshooting scenarios, especially involving failover, path changes, SLA degradation, overlay VPN behavior, and correlation with firewall logs.

I also plan to share more content and posts here in the community about this topic, because I believe SD-WAN Observability is extremely relevant for our reality in South America. We have many environments with multiple ISPs, unstable links, quality variation, and a constant need for better operational visibility. For this reason, this evolution can bring significant value to customers, partners, and technical teams across the region.

Congratulations on the work, and thank you for sharing these details. I am following this closely and very excited to test this new capability in practice.

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

israelfds95 — Thu, 18 Jun 2026 19:48:46 GMT

Very informative post and a really interesting read.

It's great to see the number of debugging tools available, and the improvements Almog described will be a huge step forward. Right now, understanding exactly why an issue occurs can be quite challenging, especially with Quantum Spark SD-WAN environments.

Sometimes the root cause is a bug, which unfortunately is not always something we can immediately fix. For me, the biggest challenge today is not collecting the debug information itself, but rather knowing how to analyze and interpret the logs in order to identify the actual root cause and solution.

Any improvements that make troubleshooting more straightforward and provide clearer visibility into SD-WAN behavior will be greatly appreciated.

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

murilomuinhos — Thu, 18 Jun 2026 22:03:23 GMT

This post truly covered the entire troubleshooting process we had to go through. Excellent content!

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

murilomuinhos — Thu, 18 Jun 2026 22:10:46 GMT

Hi Amit, how are you? I was on the meet with...@WiliRGasparetto e @israelfds95

After updating to R82.00.10 Build 203, the service only stayed up when the link was switched in ISP redundancy, but at the end of the Security Association negotiation time, the tunnel would drop. As a workaround, I created PBRs forcing the exit through each provider's internet link. This made all tunnels up.

The Check Point team considered the behavior to be characteristic of a routing bug.

Before that, I collected the evidence and sent it to the ticket!

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

Mark89 — Fri, 19 Jun 2026 19:31:36 GMT

Nicely done!

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Fri, 19 Jun 2026 19:47:36 GMT

Thank's @murilomuinhos

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Mon, 22 Jun 2026 10:49:00 GMT

I agree with you, Israel, understanding the logs effectively and what's happening in the overall context is a difficulty for most analysts, and here in Brazil we've seen this topology a lot, with SD WAN being implemented alongside Quantum Spark.

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Mon, 22 Jun 2026 14:27:55 GMT

Thk's

Re: SD-WAN Failover Troubleshooting: When VPN Tunnels Drop Between Branch Offices After a Link Failu

WiliRGasparetto — Mon, 22 Jun 2026 16:24:33 GMT

Thk's