Yuri Slobodyanyuk

Amazon AWS VPN network 172.14.1.0/24 strange case

Discussion created by Yuri Slobodyanyuk on May 10, 2018
Latest reply on May 10, 2018 by Dameon Welch-Abernathy

Good Thursday everyone, it came to my hands the strange case which I depict below for your awareness.
Topology:
Client_LAN(192.168.13.0/24) ---Checkpoint 700 (R77.20.70) -- Leased line ISP to Internet -- AWS VPN GW (5.5.5.5) -- AWS_LAN(172.14.1.0/24)
Problem: VPN tunnels come up just fine, but no traffic flows inside the tunnel in either direction. The traffic sent from the Lan to the Amazon network was being dropped on Antispoofing.
The tunnel is route based, was configured as per sk111733, still no joy. The Amazon side has no debug capabilities - you have to open ticket with them and hope for the best. Some of the efforts that had not helped:
- Upgrade to the latest firmware
- Change of the client Lan (was 192.168.1.0/24), just in case and good idea anyway.
- deleting/recreating configurations on the Amazon AWS side, which actually gave hint to a solution - configruing VPN to reach ANY network but 172.14.1.0/24 on the Amazon side worked just fine.

 

The error we were getting as I said was: "[cpu_0][fw4_0]fw_log_drop_ex: Packet proto=1 192.168.13.228:0 - 172.14.1.153:10919 dropped by fw_antispoof_log Reason: Address spoofing"
The Routing as well as Policy Based Routing were working just fine:
# ip ro show table 74
169.254.21.230 dev vpnt2
169.254.23.166 dev vpnt1
172.14.1.0/24 via 169.254.23.165 dev vpnt1
172.14.1.0/24 via 169.254.21.229 dev vpnt2 metric 77
192.168.13.0/24 dev LAN1

 

The next development was that if I disable the Antispoofing check and do sniffer of the ICMP pings being sent from the Lan to the Amazon Lan, the Checkpoint saw those echo pings coming from the WAN interface (so expectedly was dropping on the Antispoofing):
[vs_0][fw_0] vpnt1:o[84]: 192.168.13.3 - 172.14.1.178 (ICMP) len=84 id=50846
ICMP: type=8 code=0 echo request id=10942 seq=2068
[vs_0][fw_0] WAN:O[84]: 192.168.13.3 - 172.14.1.178 (ICMP) len=84 id=50846
ICMP: type=8 code=0 echo request id=10942 seq=2068
vs_0][fw_0] WAN:i[84]: 192.168.13.3 - 172.14.1.178 (ICMP) len=84 id=50846
ICMP: type=8 code=0 echo request id=10942 seq=2068
[vs_0][fw_0] vpnt1:I[84]: 192.168.13.3 - 172.14.1.178 (ICMP) len=84 id=50846
ICMP: type=8 code=0 echo request id=10942 seq=2068
I double checked and there was no physical loop on the WAN side or any cabling issue, no bridging whatsoever  - just a plain RJ45 coming from the WAN Checkpoint interface to the ISP Layer 2 equipment. It was defying any logic, and the Checkpoint Support folks thought the same.
The usual VPN debug did not help anything.
No other drop reasons were observed - all the traffic was being consistently dropped on Antispoofing.
Then I looked closely at this traffic - using sniffer with most precise timestamps (not shown here) and couldn't believe what I saw - the ICMP echo packet from the Lan behind the Checkpoint was being encrypted as should and sent out from VPN interface, then circa 70 milliseconds later the very same ICMP echo packet (same id) was returning from Amazon to the WAN interface of the Checkpoint! The timing was logical, as RTT from the client to that region of the Amazon was about 70 milliseconds. In other words the Amazon was mirroring back any traffic sent to 172.14.1.0/24. Unfortunately the client didn't have fat enough support contract with the Amazon to investigate the issue in depth, so changing the AWS network solved this (even to 172.14.2.0/24).
Just wanted to let you know of some crazy tricks this SDN thing can play :).

Best regards

Yuri

Outcomes