R80.20 Ipsec VPN issues

[Khalid_Aftas]

Hi,

After upgrade to r80.20 in multiple gateway, we started having issue with a lot of VPN that were running without problem in 80.10

case 1 : VPN with partner down, i had to make him disable NAT-T option for it to work again.

Case 2 (most critical) : Amazon Web Services, once phase 2 proposition from aws come, CP accept it, then decide to propose again another negotiation, during few minutes complete cut out of the traffic.

Other cases in other GW with simlar issues.

Opened a case in the TAC, they made me install some special hotfix, with no succes.

What changed in R80.20 regarding vpn ? i hope there is a solution for these issues.

[CPFC]
HOTFIX_R80_20_JUMBO_HF_MAIN Take: 87

[MGMT]
HOTFIX_R80_20_JUMBO_HF_MAIN Take: 87

[FW1]
HOTFIX_R80_20_JUMBO_HF_MAIN Take: 87
HOTFIX_R80_20_JHF_T87_190_MAIN
HOTFIX_R80_20_JHF_T87_174_MAIN
HOTFIX_R80_20_JHF_87_90_002_MAIN

FW1 build number:
This is Check Point's software version R80.20 - Build 100
kernel: R80.20 - Build 001

[SecurePlatform]
HOTFIX_R80_20_JUMBO_HF_MAIN Take: 87

[CPinfo]
No hotfixes..

[DIAG]
No hotfixes..

[PPACK]
HOTFIX_R80_20_JUMBO_HF_MAIN Take: 87

[CVPN]
HOTFIX_R80_20_JUMBO_HF_MAIN Take: 87

[CPUpdates]
BUNDLE_R80_20_JUMBO_HF_MAIN Take: 87

[Timothy_Hall]

There were major changes to SecureXL in R80.20.  Try disabling SecureXL VPN acceleration for the problematic peers using the vpn accel command (added in R80.20 Jumbo Take 47) as specified here:

sk151114: "fwaccel off" does not affect disabling acceleration of VPN tunnels in R80.20 and above

[Khalid_Aftas]

I've done it yesterday to do some fw monitor, i left it off, guess what, no issues today for AWS vpn... so indeed it's a workaround.

Can you please enlighten me on the core issue ? it's a bit disapointed to disable acceleration to fix this issue.

thx as always.

[Timothy_Hall]

Actually in R80.20 Jumbo HFA Take 73 and later, you don't need to disable SecureXL to get a complete fw monitor capture using -F.

I assume you disabled SecureXL with the fwaccel off command, which is not a permanent fix and may cause performance issues. Any time that SecureXL needs to be disabled to make things work it is always a good idea to open a TAC case so they can figure out why.  That said, I'd recommend the following (note doing the following will cause a brief VPN outage):

1) Disable SecureXL acceleration for the problematic VPN peers via the vpn accel command (you can also disable all SecureXL VPN acceleration with the vpn accel off command).

2) Turn SecureXL back on with fwaccel on (or just reboot).

SecureXL was completely overhauled in R80.20 in preparation for the upcoming Falcon accelerator cards.

[Khalid_Aftas]

i used vpn accel peerip not fwaccel off sorry.

I keep having incidents for s2s vpn poping 😄 i guess i need to disable it for almost all of them.

i'll check with the tac also.

[wizzolo]

Hi,

i've used for the same issue the command vpn accel off for a particular peer and seem to fix the problem but after installation policy the vpn stops to working and I need to use vpn tu to reset the tunnel.

Any idea ?

Thank you

[Vitaly_Timofeev]

Hi wizzolo,

Do you have keep IKE SAs checked?

In any case if it is checked or not I would advise to open a ticket to TAC as it is not an expected behavior.

[Vitaly_Timofeev]

Hi Khalid,

I would like to get more details regarding different SXL VPN issues you are having.

Currently we do not have known VPN issues with SXL which are present in the latest R80.20 JHF.

In case you still have difficulties with it - I would prioritize it in our group.

Are you running the latest JHF take?

Can you share the SRs which are still unresolved, or resolved, but fixes still not inside JHF?

[Khalid_Aftas]

Hi Vitaly,

Yes i have keep IKE SA enabeld, and we are running Take 87 HF.

I of course have a case with the TAC, now it's with an escalation engineer, he suggested to install latest non GA jumbo as first solution.

Kr,

Khalid

[wizzolo]

Just to share my point, I've the  Take: 103 with the issue

[Jonne_Hannon]

Hi all,

I had a VPN outage to AWS after installing policy on a R80.20 gateway running Take 87. I installed policy a second time and the tunnels re-established. My community contains 2 tunnels and is configured as per AWS & Check Point documentation, including DPD monitoring.  If you try installing policy a second time, does the tunnel come back up?

Thanks,

Jonne

[Khalid_Aftas]

Hi,

The issue is related to the new VPN acceleration brought in r80.20, TAC was not able to found anything, and we decided to give it a try later in r80.30, and ... same issue.

I hear the same stories with others colleagues working with AWS and CP vpn.

How can we tackle this ?

[wizzolo]

I've solved with command vpn accel off for a specific peer, try it.

Attention is not fw accel off and is necessary a new entry(rc or script) to make it persistent after reboot

[Khalid_Aftas]

Well that is what i explained, vpn accel off is only working for max 2 IPs.

We would like to have this fixed, instead of disabling acceleration and the limitation of 2

[Jonne_Hannon]

Hi all,

I believe my issue was solved by assigning an empty group to the AWS peers (Interoperable Devices) VPN Domain to stop the Encryption Fail Reason is "VPN failed to resolve MEP Gateway" error after policy installation.  I also had to make sure that ping was enabled on the routes for the AWS peers.  I have VPN acceleration enabled and it has been pretty stable for the past few months.

Thanks,

Jonne.

[Khalid_Aftas]

Well we dont use VTI, as we are running vsx, it's a regular VPN.

[Jonne_Hannon]

In my experience, AWS VPN isn't very stable as a regular VPN and should be configured with VTIs.  We initially had it configured as a regular VPN and it was dropping regularly.  Much more stable with VTIs.

[Khalid_Aftas]

We had it running for 3 years without issues like that on 77.30 till 80.10, VTIs are still not supported, and i don't think it will change anything, at the end it's the same pipepline.

Since the change of SecureXL from 80.20 it's dropping at every phase 2 timeout.. without acceleration 0 problems.

[Jung_Patrick]

Hello, Khalid.

Did you solve this problem?

I also upgraded from R77.30 to R80.40 T119, and most of the other peers were fine, but the AWS peer could not stabilize it until the end. (Last hotfix at the time when upgrading)

The tunnel was attached and then disconnected after a few minutes, or it was connected well and then disconnected within a few hours.

I am also a domain based vpn and do not use VTI.

Of the total of 21 peers, there are about 3 or 4 AWS.

IKE_SAs is off. (R77.30 and R80.40 also)

I tried vpn accel off but to no avail.

Only CP<->AWS is unstable and has many problems.

Therefore, it is not possible to upgrade, and it is difficult to receive support from TAC.

I made a lab in the office and tested it.
I'm not sure if it's the same as the client's symptoms. Something is unstable.

Thank you.

[eakselrod]

Hello All,

For issue 1 with NAT-T  this is a known issue that is described in sk165003 as the sk state, the fix is integrated into the following jumbo takes

• Jumbo Hotfix Accumulator for R80.40 starting from Take 83
• Jumbo Hotfix Accumulator for R80.30 starting from Take 219
• Jumbo Hotfix Accumulator for R80.10 starting from Take 283

For issue 2 it is very hard to provide any feedback without seeing debugs solely on this description. Is you opened SR for this please message me in private and I will look into it.

R81 now support VTI with VSX. Please see "what's new" section in the R81 sk166715

• Configure Dynamic Routing VPN through Virtual Tunnel Interface (VTI) in VSX mode.

[Khalid_Aftas]

Hi,

SR sent in DM, with the input of AWS and what they think about this particular issue

in summary : the GW send "UDP-Encapsulated-Tunnel" that make the tunnel stuck until rekeying from CP.

[Mobin_Iftikhar]

had same problem. Upgraded firewalls from R77.30 into R80.40 and traffic IKE traffic changed into NAT-T traffic. had to add NAT-T in the rulebase and exculade NAT-T services from the community. In addition to this also in Guidbedit allow offer_initiator_nat_t = true. I think CP made some changes how their overall NAT behavior. I have checked in splunk an couldn't find the remote gateway ever offer NAT-T port 4500 however; after upgrade CP gateways are acting very differently and changed IKE 500 traffic with NAT-T 4500 traffic.

In addition to this I have also disable NAT-T support from global properties of gateway as some vendor remote gateways could not support NAT-T traffic to make VPN works. 

[Jung_Patrick]

Mobin Have you ever solved this problem?

I have symptoms very similar to mine.

In R77.30, about 20 3rd party vendors and Site to site VPNs are connected, and about 3 or 4 AWS are also connected.

I installed R80.40 T119 (the latest at that time), and the problem was not resolved, so I restored it to its original state.

I also noticed NAT-T traffic, I honestly can't remember if I turned off NAT-T in July 2021.

But it seems you didn't change iniator_nat_t = with GuiDbedit.

Have you ever tweaked NAT-T to successfully upgrade and run aws stably?

Thank you.

[Eitan_Gilad-Lug]

Hi Khalid,

I will be happy to get the SR number offline, so i can take it with my Support team

thank you

Eitan