Any known issues with VPN on R82.10?

[Alex-]

SMS: R82.10 T50

Customer's HQ: 9100 ClusterXL running R81.20 with latest Take

Remote sites: 3600 ClusterXL running R81.20, being upgraded to R82.10 T50.

--

Each year; the customer runs a scenario which involves a VTI to a 3rd party, some routing to activate said scenario. The exercise happens with the HQ and a designated remote site. No NAT is involved.

This year, the chosen remote site was one already migrated to R82.10 T50.

On both locations, the VTI went up, IPSEC was established and routing was performed.

Everything went well at the HQ, but no traffic would be seen by the 3rd party from the branch site, just that the tunnel was active.

Findings

---------

fw monitor -F shows the correct chain: i - I o - O - Oe with both the LAN side VLAN on the inbound, and the VTI on the outbound

cppcap -i shows traffic in and out the VTI at HQ, nothing at branch

fwaccel off/on doesn't make any difference

vpn tu tlist shows the VPN connected, with outbound traffic but no inbound

vpn debug didn't report any issues

/var/log/messages and others neither

Logs show the traffic being encrypted

Directional VPN rules or "Any" provide the same result.

Failover, reboot, recreating the VTI, changing VTI number and similar tinkering didn't change the situation at all

Changing from empty group to defined groups to try a domain-based VPN (only for test, we need the routing functionalities) had the curious effect of losing SSH/HTTPS to the WebUI while production continued to work, because of the "clear the packets should be encrypted" situation, when neither the jump host nor the management interfaces of the gateways were part of the VPN definition.

Documentation on VTI was re-read for R82.10

As the ClusterXL was upgraded from R81.20 to R82.10 with Blink, we contemplated a rollback to R81.20 thanks to the snapshots still present on the gateways, but decided against it and went with only HQ with a VTI this year.

-----

It would appear that while the traffic is correctly sent to the VTI, something doesn't work as it would there. Since we ran out of time and starting a SR would take time and be difficult due to practical reason with the planned activity, we decided to call it a day and not use the site this year.

One thing that got me thinking is that it was one of my first use of R82.10 with Intel-based appliances. I've VPN with ARM-based appliances (3900 series) without any issues. For the rest, it's unclear what happened, if we're either experiencing a bug with that mix of setup and OS version, or if there was anything else we didn't notice.

[Timothy_Hall]

Did you have VPN acceleration disabled in fwkern.conf before the upgrade?  That will break VPNs when you upgrade to R82 or later,  IKE Phases 1 and 2 will complete, but no traffic will actually pass:

sk1000038: VPN traffic is dropped after upgrading a Security Gateway/ ClusterXL with "tunnel is acce...

Check Point removed the ability to disable SecureXL acceleration of VPN tunnels in R82 and then mandated the use of UPPAK mode for SecureXL in R82.10.  So when you upgrade from R81.20 to R82.10, you run into both of these requirements at once, which is almost certainly the source of your problem.

You might consider trying R82 with the latest JHFA applied instead, since you can still revert to KPPAK in that version to see whether UPPAK specifically is causing the problem.  In R82.10, you are pretty much stuck opening a case with TAC for VPN problems, as Check Point removed customers' ability to further isolate where the problem is without TAC assistance.

Also see these threads: R82 to R82.10 - VPN Issues & R82 - vpn accel is deprecated?

[Alex-]

Those branch firewalls never had to be tweaked with kernel values and the like. They're textbook branch offices: local cluster, centrally managed from HQ over an MPLS network (no management NAT over Internet and so on), local Internet line. I didn't mention it in the findings but fw ctl zdebug drop didn't return anything either.

Regarding fwkern.conf, as mentioned, they were never modified either centrally or locally.

Here's how they look like:

[Expert@GATEWAY-01:0]# cat /opt/CPsuite-R82.10/fw1/boot/modules/fwkern.conf
fwmultik_sync_processing_enabled=0
record_dns_enabled=1
iot_self_discovery=0

[Expert@GATEWAY-01:0]# cat /opt/CPsuite-R82.10/fw1/boot/modules/copy_fwkern.conf
fwmultik_sync_processing_enabled=0
record_dns_enabled=1
iot_self_discovery=3
num_of_sxl_devices=1
kiss_sxl_usermode_enabled=1
fwmultik_rss_num=1
fwmultik_fwk_core_num=3

So while I think the issue is with R82.10, I can't definitely prove it, it was impractical to start with SR and so on while being in operational setup for the customer's scenario.

R82.10 is a mixed bag so far, between the ElasticXL issues with hotfixes on 39x0 and the fact that upgrading the SMS from R82 to R82.10 blew the AIOps and IoT configuration (SR now at R&D), I'm indeed thinking of going longer with R82 until more hotfixes are deployed, or even wait for R82.20. On the other hand, MVC BLink upgrades were fast and completely transparent. 

[ccsjnw]

A new install of R82.10 doesn't contain any of the above fwkern parameters.

CheckPoint told us that between versions, some fwkern.conf settings that may have been appropriate previously, may not be appropriate anymore, and may cause more harm than good. In the past, some of the tweaks we used to apply, became unnecessary, and perhaps resulted in poorer, rather than enhanced performance, because CheckPoint reworked things internally withing the kernel.

In your shoes, I would temporarily remove all of the fwkern.conf setting, reboot and check your VPN connectivity again... I'd also check with TAC to see if they still recommend your fwkern.conf tweaks or not. 

Having said that, we have experienced problems with site to site VPNs in R82.00 and R82.10. The second tunnel would establish normally, but then would die after a period of time, and would not re-establish. The destination device is a third party device maintained by a different organisation - so troubleshooting becomes complicated. This problem only happened we ran more than one site-to-site VPN tunnel on the same Active / Standby HA Cluster (we are using ClusterXL, not ElasticXL). We are running SD-WAN on the Cluster where the second VPN tunnel is problematic and we think this might be the underlying cause. We have also had a long standing issue (that we cannot pin down exactly) where the SD-WAN in the HA Cluster, suddenly seems to get confused about which interface the traffic should be sent down. Heavy load / bandwidth utilisation and SD-WAN seems to be the cause, but more troubleshooting is required. When we previously used policy based routing in the CheckPoint platform, we didn't have the same issues as we see with SD-WAN.

[AmirArama]

You use SD-WAN, and the peer is Third party?
our SD-WAN not manage or intervene in VPN Tunnels to third parties. the VPNs are regular link selection / enhanced link selection tunnels. with no SD-WAN.

[ccsjnw]

SD-WAN is CheckPoint SD-WAN, used to route traffic over two separate external connections.

The other end of the site-to-site VPN tunnel is not CheckPoint equipment, and it's not managed by the site team - it is managed by an external organisation.

[AmirArama]

Yes, I understand.

in this case Check Point SD-WAN will not do anything to the VPN tunnels, or to traffic routed on this VPN.
The only thing can go wrong if traffic routed to VTI being matched with a Local Breakout rule, in this case the traffic will not get encrypted but instead sent to the ISPs in clear.

[Alex-]

SD-WAN is not used in this setup. All verifications point to outbound traffic sent to the VTI, just nothing happens afterwards.

[Alex-]

I don't think wiping kernel values set by the upgrade process "just to try" is recommended without knowing exactly what happens.

They relate to the following:

Segmentation fault while running commands

DNS requests stay in Medium path when DNS protections are removed (not the case here)

The IoT entry doesn't show in SK but as the customer uses IoT, it's likely there for a reason.

None of this seems to realte to VTI or VPN.

[AmirArama]

I suggest isolating the issue by checking each phase of the VPN traffic flow:

1. VPN Tunnel Establishment

First, confirm that the VPN tunnels are established.

• Run vpn tu tlist and verify that the tunnels are UP and have an outbound SPI (OUT SPI).

• Based on your previous response, this step appears to be OK.

2. Traffic Forwarding – Before Encapsulation

Verify that the traffic is correctly routed and encrypted:

• fw monitor – Verify that the connection is forwarded toward the expected VTI.

• vpn tu conn – Verify that the connection is associated with the expected VPN tunnel and check the tunnel state.

3. Traffic Forwarding – After Encapsulation

Verify that the encrypted traffic actually leaves the Security Gateway toward the peer's WAN IP.

Capture traffic on the outgoing interface:

tcpdump -nnei <interface_name> 'esp or udp port 4500'

Check whether ESP or NAT-T (UDP/4500) packets are transmitted toward the peer's WAN IP.

If the encrypted packets are leaving the outgoing interface, the next step is to investigate the receiving side:

• Does the peer receive these packets on its WAN interface?

• If yes, are they successfully decrypted?

• If decryption fails, are there any VPN-related drops or errors?

This should help narrow down whether the problem is related to tunnel establishment, traffic forwarding/encryption, or delivery/decryption on the receiving peer.