R81.10 VPN site-2-site to Cisco C8500-12X IOS XE (not Palo Alto as previously stated)

[BjornErichsen]

EDIT:
Sorry guys. I was misinformed - it now proves that the remote peer is in fact cisco C8500-12X, not Palo Alto firewalls... They are not making it easy on me 🙂


History:
I've seen several posts about difficulty maintaining VPNs between CheckPoint secure GW and Palo Alto FWs.

I am managing a CP R81.10 secure GW (VSX) with several VPNs to different vendors.
In late April we created yet another site-2-site VPN tunnel - towards Palo Alto (for the first time), and it worked flawlessly.
In early July we deployed most recent (at that time) Jumbo Hotfix.

Issue:
Since the JHF deployed in July it appears we have had problems when IPsec SA keys are renegotiated (at default time interval of 3660 seconds).
Note that the tunnel works for the vast majority of the time, and the tunneled subnets does re-establish communication eventually without manual intervention, but we do see traffic impact.

VPN Blade logs Rejects of various types - but generally in sequence:

From remote Palo to CP:
Child SA exchange: Ended with error
Initial exchange: Sending notification to peer: Invalid Key Exchange payload

Then from CP to remote Palo:
Child SA exchange: Received notification from peer: No proposal chosen MyMethods Phase2: AES-GCM-256 + HMAC-SHA2-384, No IPComp, No ESN, Group 20 (384-bit random ECP group)

And from Palo to CP
Informational exchange: Ended with error
Initial exchange: Sending notification to peer: Invalid Key Exchange payload

Actions:
We will be upgrading to latest Jumbo Hotfix (which claims to fix some VPN issues though none appear directly related) next week, but in case that does not solve the issue any help would be greatly appreciated.

We already have our eye on DPD @Palo since the CP side has not been configured with the tunnel as "Permanent", but I doubt that would cause IPsec renegotiation to fail sporadically.

Also we have requested Palo side to first try with a IKEv2 proposal that exactly matches CP configuration. This has not yet been implemented - as these proposals are "global" - but they are looking into it.

I'd really like to hear if anybody have fixed identical issues?


CP VPN community config:
We have a VPN community ("policy based") tunnel with verified encryption domains (subnets) at both ends.
Only allow encrypted traffic
IKEv2 only
Phase 1: AES-256,SHA384.Group 20
Phase 2: AES-GCM-256, PFS group 20
Not permanent and One VPN tunnel per subnet pair
Shared secret (which of course works)
Renegotiate IKE 1440 (minutes)
Renegotiate IPsec 3600 (seconds)
Anything not mentioned should be at default values for R81.10 (initial deployment for this VSX cluster was on R80.40)

Cisco IOS XE config (which I do not control):
#show crypto ikev2 proposal
IKEv2 proposal: VPN_XXXX_PROPOSAL_AES_CBC
     Encryption : AES-CBC-256 AES-CBC-192 AES-CBC-128
     Integrity  : SHA512 SHA384 SHA256
     PRF        : SHA512 SHA384 SHA256
     DH Group   : DH_GROUP_2048_256_MODP/Group 24 DH_GROUP_521_ECP/Group 21 DH_GROUP_384_ECP/Group 20
IKEv2 proposal: VPN_XXXX_PROPOSAL_AES_GCM
     Encryption : AES-GCM-256 AES-GCM-128
     Integrity  : none
     PRF        : SHA512 SHA384 SHA256
     DH Group   : DH_GROUP_2048_256_MODP/Group 24 DH_GROUP_521_ECP/Group 21 DH_GROUP_384_ECP/Group 20
IKEv2 proposal: default Disabled

#show crypto ipsec profile VPN_XXXX_PROFILE_10029
IPSEC profile VPN_XXXX_PROFILE_10029
        IKEv2 Profile: VPN_XXXX_PROFILE_10029
        Kilobyte Volume Rekey has been disabled.
        Security association lifetime:3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group:  group20
        Mixed-mode : Disabled
        Transform sets={
                TS_XXXX_AES_GCM256:  { esp-gcm 256  } ,
        }

#show crypto ikev2 profile VPN_XXXX_PROFILE_10029

IKEv2 profile: VPN_XXXX_PROFILE_10029
 Ref Count: 5
 Match criteria:
  Fvrf: INFRA
  Local address/interface:
   yyy.zzz.xxx.ooo
  Identities:
   address vvv.uuu.ddd.qqq 255.255.255.255
  Certificate maps: none
 Local identity: none
 Remote identity: none
 Local authentication method: pre-share
 Remote authentication method(s): pre-share
 EAP options: none
 Keyring: VPN_XXXX_KEYRING_10029
 Trustpoint(s): none
 Lifetime: 86400 seconds
 no lifetime certificate
 DPD: interval 10, retry-interval 5, periodic
 NAT-keepalive: disabled
 Ivrf: none
 Virtual-template: none
 mode auto: none
 AAA AnyConnect EAP authentication mlist: none
 AAA EAP authentication mlist: none
 AAA Accounting: none
 AAA group authorization: none
 AAA user authorization: none
 PPK Dynamic: 0 PPK Required : 0 PPK Instance ID:

[the_rock]

See if my post below helps and how we got it working.

Andy

Might not be exact same issue, but you may find it useful.

VPN route based query - Check Point CheckMates

[BjornErichsen]

Thanks - but IKEv1 (as I read your solution) is not really an option for us. Audit will be on our backs 🙂

[CaseyB]

This sounds like a familiar issue I had with a set of Palos. Looking at my old e-mails, I found this snippet:

These are the errors I have been seeing:

• Child SA exchange: Sending notification to peer: No proposal chosen MyMethods Phase2: AES-GCM-256 + HMAC-SHA2-256, No IPComp, No ESN, Group 19 (256-bit random ECP group)
• Initial exchange: Sending notification to peer: Invalid Key Exchange payload
• Child SA exchange: Exchange failed: timeout reached.

The Palo side was using this:

Phase 1 has these configured:

DH Groups 21, 20, 19, 14, 5, 2

Encryption AES-256-GCM, AES-256-CBC

Authentication sha512, sha384, sha256, non-auth

Lifetime: 24 hours

Phase 2 has these configured:

DH Group 19

Lifetime 1 hour

Lifesize 4608 MB

Encryption: aes-256-gc, aes-256-cbc, aes-192,cbc, aes-128-gcm, aes-128-ccm, aes-128-cbc

Authentication: sha512, sha384, sha256

The resolution: Palo side created a profile specifically for our tunnel to use the same encryption ciphers we were sending instead of using a global profile with several ciphers enabled.

[the_rock]

Thats an excellent point! I know PAN guy did that in our case (the post I referenced), though there, main issue was ID he needed from CP side to make it work fully. I believe things like what you advised @CaseyB are less relevant for say Cisco or Fortinet, but for PAN, I do know it matters more.

Andy

[BjornErichsen]

Thanks - yes I read somewhere that is solved the issue (maybe an older comment by you? ), and this is actually what I meant by "Also we have requested Palo side to first try with a IKEv2 proposal that exactly matches CP configuration" in OP
Specifically put this in top of global proposals or as specific for this tunnel:
IKEv2 proposal: VPN_XXXX_PROPOSAL_AES_GCM_CP
     Encryption : AES-GCM-256
     Integrity  : none
     PRF        :  SHA384
     DH Group   : DH_GROUP_384_ECP/Group 20

This gives me hope that it might actually solve the issue.

Its just sickens me that the issue apparently arose after an CP JHF deployment, because it leaves little confidence in the future stability of this VPN. Thankfully its mostly used for mgmt access, but still... 

[the_rock]

I think that may had been coincidental, I really do.

Andy

[the_rock]

Forgot to ask, did tunnel work constantly without any issues BEFORE jumbo install?

Andy

[BjornErichsen]

Yes - at least no users reported errors - and firewall log contains none of these Rejects prior to reload after JHF was applied. If there were any issues they were insignificant. Edit: or not logged...

[JozkoMrkvicka]

What JHF was installed on CP when all seemed to work ?

To which JHF you updated CP when you noticed the issue?

Did you uninstall the problematic JHF to confirm it is 100% related to JHF update ? If so, TAC should be involved and provide root cause

[BjornErichsen]

We upgraded from JHF take 129 to JHF take 152

We did not try to uninstall JHF take 152 since it took a long time before the issue was discovered (due to summer vacation) and we need security fixes in 152 more than we need this mgmt access to be stable. 

We just installed latest JHF take 158, but it does not appear to have solved anything. Next step is to have the Palo IKEv2 proposal changed.

[the_rock]

Hey @BjornErichsen 

Did you check out the post I referenced? I can even send you sreenshot of the ID Im referring to. Again, not sure if its 100% applicable in your situation, but worth checking.

Andy

[CaseyB]

FYI - I have an open TAC case with R&D, a lot, if not all my IKEv2 tunnels started generating IKE failures on re-keys in R81.10 JHF 152+. I had to revert to JHF 150 to resolve the errors.

Most of my tunnels are IKEv2: AES-256, SHA256, AES-GCM-256.

[Alex-]

We have solved a lot of VPN issues by going to R81.20 Take 65+ on both VSX and ClusterXL on all sort of VPN configurations.

The VPN feature seems really improved in that version of the OS.

[BjornErichsen]

Thanks - good to know.
Unfortunately I am stuck on R81.10 management servers a little while longer due to appliances - soon to be hardware refreshed.

[BjornErichsen]

ooh well - that sounds ominous.
You can probably add this post to the case then. Looks like VPN blade errors are not limited to Palo in my end either, but that specific VPN just has more error-logs than all the others combined, and we don't have user complains about the other VPNs (yet).
Currently running R81.10 JHF take 158 (I know JHF 156 is the recommended take for the time being).

Edit: All errors are Rejects - 94% of errors are inbound to checkpoint firewall

[Zolocofxp]

Having PA create a custom profile will most likely fix the issue. I've had great success limiting the encryption algorithm to CBC like the following example.

 

I'd also check the Traffic Selectors that are being proposed by both ends when tunnels come down. 

[the_rock]

Agree 100%