Re: Symmetric Packet Return SD-WAN, VPN and NAT is...

RafaelBohrer

Dear all,

First of all, let me explain the current topology (image below).

Headquarter

Security Gateway – Check Point 3920
GaiaOS 82.10 JHT 45
Blades: Firewall, SD-WAN, Site to Site VPN
WAN interfaces:

INTERFACE NAME	IP ADDR/MASK	NEXT HOP	ROUTER WAN	MTU	SD-WAN	COMMENTS
eth2	187.x.x.244/28	187.x.x.254	Bridged Mode	1500	Yes	ISP#1
eth2.1	187.x.x.245/28	187.x.x.254				eth2 alias
eth3				1500		Used by pppoe1
pppoe1	179.x.x.63/32	200.x.x.231	Bridged Mode	1492	Yes	ISP#2 Auto MTU by SG Use Peer as Default Gateway
eth4	192.168.15.244/24	192.168.15.1	Routed Mode 179.x.x.82	1500	Yes	ISP#3
eth5	192.168.1.244/24	192.168.1.1	Routed Mode Dynamic IP	1500	Yes	ISP#4 Satellite – Internet Only

LAN interfaces:

INTERFACE NAME	IP ADDR/MASK	MTU	COMMENTS
eth10.10	172.31.0.1/24	1500	Voice (VLAN10)
eth10.11	172.31.1.1/24	1500	Servers (VLAN11)
eth10.12	172.31.2.1/24	1500	Users (VLAN12)
eth10.15	172.31.5.1/24	1500	CFTV (VLAN15)
Mgmt	172.31.7.101/24	1500	Management (VLAN1)

SG Default Route: 0.0.0.0/0 -> Gateway 187.x.x.254 (priority None).
All WAN interfaces have “Inbound Symmetric Packet Return” enabled on SD-WAN Link Mapping.
NAT Policies

SERVICE	ORIGINAL SRC	ORIGINAL DST	ORIGINAL SRV	TRANSLATED SRC	TRANSLATED DST	TRANSLATED SRV
DVR	Any	179.x.x.63	tcp/3000-3002	Original	172.31.5.4	Original
DVR	Any	187.x.x.245	tcp/3000-3002	Original	172.31.5.4	Original
WEB APP	Any	179.x.x.63	http, https	Original	172.31.1.81	Original
WEB APP	Any	187.x.x.245	http, https	Original	172.31.1.81	Original

AWS Route53 configuration:
- vpn.company.com -> 187.x.x.244
- webapp.company.com -> 187.x.x.245 (Failover to 179.x.x.63)
- dvr.company.com -> 187.x.x.245 (Failover to 179.x.x.63)
Site to site VPN
- A single VPN tunnel between HQ and Remote Office #1 (FortiGate)
- HQ Central Gateway – 187.x.x.244 (eth2)
- Remotee Office #1 Satellite Gateway – 200.x.x.119 (eth0)
Remote Access VPN using Endpoint Security VPN Client (peer vpn.company.com).

Here’s the situation.

An external user access https://webapp.company.com or dvr.company.com:3001 to access my internal WebApp Server or DVR Server.
For both URL, Route53 DNS resolves to 187.x.x.245 or 179.x.x.53 depending of link disponibility.
The Security Gateway performs the NAT to internal server.
Every day, between 3AM and 6AM, the ISP#1 provider reset the PPPoE connection and the reconnection is performed normally by 3920. Then, the interface pppoe1 is UP and is CONNECTED.
After reconnection, there is no more possibility to access the services through IP 179.x.x.63, SD-WAN link status is DOWN.
The pppoe1 interface is no longer transmitting or receiving any traffic.
Manually, I disable the interface pppoe1 using “set interface pppoe1 state off”, waits for a minute and then “set interface pppoe1 state on”.
Here’s the very strange behavior. One of these things can happen:
- Interface pppoe1 still not transmitting or receiving traffic.
- Interface pppoe1 is transmitting traffic (outgoing traffic), but I can’t access the internal services remotely through NAT.
- Interface pppoe1 is transmitting and receiving traffic normally.

After research, I suspect that there is a problem with the return of traffic caused by the routes.

In my research, I noticed that if the "Use Peer as Default Gateway" option is enabled on the pppoe1 interface, it overrides the default route on the 3920 gateway.

Here’s the /var/log/messages when the ISP terminates the connection:

Oct 1 03:24:37 2026 POA_CPFW_01 pppd[3354344]: LCP terminated by peer

Oct 1 03:24:37 2026 POA_CPFW_01 pppd[3354344]: Connect time 2150.1 minutes.

Oct 1 03:24:37 2026 POA_CPFW_01 pppd[3354344]: Sent 1390575875 bytes, received 1382316478 bytes.

Oct 1 03:24:37 2026 POA_CPFW_01 pppd[3354344]: Modem hangup

Oct 1 03:24:37 2026 POA_CPFW_01 pppd[3354344]: Connection terminated.

Oct 1 03:24:38 2026 POA_CPFW_01 ntpd[15895]: Deleting interface #31 pppoe1, 179.104.43.63#123, interface stats: received=0, sent=0, dropped=0, active_time=129005 secs

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: PPP session is 765

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: Connected to d0:dd:49:19:58:23 via interface eth3

Oct 1 03:25:07 2026 POA_CPFW_01 kernel:[703327.587351] adpif_add: s18p64 added:0x0000000094423584(0:64)

Oct 1 03:25:07 2026 POA_CPFW_01 kernel:[703327.587358] adpif_add :0:18:64

Oct 1 03:25:07 2026 POA_CPFW_01 kernel:[703327.587542] pppoe1: renamed from ppp0

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: Using interface pppoe1

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: Connect: pppoe1 <--> eth3

Oct 1 03:25:07 2026 POA_CPFW_01 kernel:[703327.651543] adp_host_if_status: nothing to be done for now pppoe1

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: PAP authentication succeeded

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: peer from calling number D0:DD:49:19:58:23 authorized

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: not replacing default route to eth2 [187.x.x.254]

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: local IP address 179.x.x.63

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: remote IP address 200.x.x.231

Oct 1 03:25:09 2026 POA_CPFW_01 ntpd[15895]: Listen normally on 32 pppoe1 179.104.43.63 UDP 123

What I noticed is that the default route was not overridden.

Oct 1 03:25:07 2026 POA_CPFW_01 pppd[3354344]: not replacing default route to eth2 [187.x.x.254]

What is my need:

I need SD-WAN to steer all 4 WAN links (SD-WAN policies are configured).
I need inbound traffic to always return over the inbound link. If the traffic came in through pppoe1, the response should be on pppoe1. If the traffic came in through eth2, the response should be on erth2.
Remote users connect via VPN on the eth2 link (vpn.company.com -> 187.x.x.244).
NAT rules should continue to work.

My future desired scenario after solving the above issues:

Remote users can automatically connect over more than one WAN interface in the event of a failure of any of them.
Establish a second VPN connection between Remote Office #1 and Gateway 3920 on a second WAN link in HQ.

I tried many different configurations, and nothing seems to work. If I change the Default Route to use pppoe1 link, the remote users can’t connect through VPN anymore.

What am I missing?

I'll give some outputs.

POA_CPFW_01> show interface eth3
state on
mac-addr 00:1c:7f:cd:91:4a
type ethernet
link-state link up
mtu 1500
auto-negotiation on
speed 1000M
ipv6-autoconfig Not configured
monitor-mode off
duplex full
link-speed 1000M/full
comments Link Algar 1G PPPoE (ver interface pppoe1)
ipv4-address Not Configured
ipv6-address Not Configured
ipv6-local-link-address Not Configured
Statistics: 
TX bytes:91774683165 packets:210039716 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:447744346287 packets:391554077 errors:0 dropped:0 overruns:0 frame:0
SD-WAN: Not Configured

POA_CPFW_01> show interface pppoe1 
state on
mac-addr Not configured
type pppoe
link-state not available
mtu 1492
auto-negotiation off
speed N/A
ipv6-autoconfig Not configured
monitor-mode Not configured
duplex N/A
link-speed Not configured
comments Link Algar 1G
ipv4-address 179.x.x.63/32
ipv6-address Not Configured
ipv6-local-link-address Not Configured

Statistics: 
TX bytes:450 packets:8 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:782625 packets:14095 errors:0 dropped:0 overruns:0 frame:0

SD-WAN: 
interface-nexthop-ip-address     200.x.x.231
interface-tag                    ALGAR1G
interface-nat-ip-address         Not Set
interface-download-speed-value   1000
interface-upload-speed-value     1000
interface-circuit-id-value       0
interface-link-type-value        Undefined

POA_CPFW_01> show route all
Codes: C - Connected, S - Static, R - RIP, B - BGP (D - Default),
       O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA),
       IS - IS-IS (L1 - Level 1, L2 - Level 2, IA - InterArea, E - External),
       A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed,
       NP - NAT Pool, U - Unreachable, i - Inactive

S               0.0.0.0/0           via 187.x.x.254, eth2, cost 0, active age 15365, age 15365  
C               127.0.0.0/8         is directly connected, lo  
C               172.31.0.0/24       is directly connected, eth10.10  
                                        Rede Voz (VLAN10)  
C               172.31.1.0/24       is directly connected, eth10.11  
                                        Rede Servidores (VLAN11)  
C               172.31.2.0/24       is directly connected, eth10.12  
                                        Rede Desktop (VLAN12)    
C               172.31.5.0/24       is directly connected, eth10.15  
                                        Rede Cameras (VLAN15)  
C               172.31.7.0/24       is directly connected, Mgmt  
                                        Rede Gerëncia (VLAN1)      
C               179.x.x.63/32    is directly connected, pppoe1  
                                        Link Algar 1G  
C               187.x.x.240/28   is directly connected, eth2  
                                        Link Algar 100MB  
C            i  187.x.x.240/28   is directly connected, eth2  
                                        Link Algar 100MB  
C            i  187.x.x.240/28   is directly connected, eth2  
                                        Link Algar 100MB  
C               192.168.1.0/24      is directly connected, eth5  
                                        Link Starlink  
C               192.168.15.0/24     is directly connected, eth4  
                                        Link Vivo  
C               200.x.x.231/32  is directly connected, pppoe1  
                                        Link Algar 1G

POA_CPFW_01> ping -I eth2
/bin/ping: usage error: Destination address required
POA_CPFW_01> ping -I eth2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 187.x.x.244 eth2: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=25.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=23.6 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 23.560/24.335/25.111/0.775 ms

POA_CPFW_01> ping -I pppoe1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 179.x.x.63 pppoe1: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=20.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=20.1 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 20.032/20.052/20.072/0.020 ms

POA_CPFW_01> ping -I eth4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.15.244 eth4: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3052ms

POA_CPFW_01> ping -I eth5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.1.244 eth5: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3046ms

The result of ping on interface pppoe1 is good, but SD-WAN probing is marking DOWN.

It is problably I am missing someting in the configuration.
Hope someone can give me some hint to how to solve this.

If any addicional information is needed, please let me know.
My best regards.
Rafael Bohrer.

PhoneBoy

This might require TAC to assist here.

Note that Remote Access VPN usually gets tied to a gateway IP unless you're changed to DNS resolution: https://support.checkpoint.com/results/sk/sk103440

AmirArama

Hi,

Regarding VPN Clients, in R82.10 by default we apply symmetric return only for traffic passing through the gateway (your NAT services), but not to traffic destinated for the gateway itself (VPN Remote). in R82.20 it should work also for traffic to the gateway itself.

Regading the pppoe reconnect issue:
I agree that you need to open a service request to TAC and probably ask for raising a task to R&D.
this might be a known bug where the steering is not updated properly on the pppoe recreation.

Add to the SR the following info:

cpinfo file

take the following once the PPPOE works properly, and then again after it's reset and doesn't work anymore (down on cpview)

First raise steering trace level:
fw debug sdwan_steering on TDERROR_SDWAN_ALL=5

collect outputs and file

ip link show pppoe1
fw ctl iflist
ip rule show | grep fwmark
ip route show table all
$FWDIR/log/sdwan_steering.elg

after replication and files collection, please lower steering trace level:
fw debug sdwan_steering on TDERROR_SDWAN_SDWAN=1

you can mention my name in the SR

in the meantime as a workaround you can try to restart sdwan_steering process to see if it solves the issue and the pppoe isp becomes UP in cpview:
sdwan_steering_stop;sdwan_steering_start

Thanks

Are you a member of CheckMates?

Symmetric Packet Return SD-WAN, VPN and NAT issues.