topic Re: R81.10 VPN site-2-site Palo Alto in Firewall and Security Management

R81.10 VPN site-2-site to Cisco C8500-12X IOS XE (not Palo Alto as previously stated)

BjornErichsen — Thu, 05 Sep 2024 12:47:16 GMT

EDIT:
Sorry guys. I was misinformed - it now proves that the remote peer is in fact cisco C8500-12X, not Palo Alto firewalls... They are not making it easy on me 🙂

History:
I am managing a CP R81.10 secure GW (VSX) with several VPNs to different vendors.
In late April we created yet another site-2-site VPN tunnel - towards Cisco IOS XE (for the first time), and it worked flawlessly.
In early July we deployed most recent (at that time) Jumbo Hotfix take 152.

Issue:
Since the JHF deployed in July it appears we have had problems when IPsec SA keys are renegotiated (at default time interval of 3660 seconds).
Note that the tunnel works for the vast majority of the time, and the tunneled subnets does reestablish communication eventually without manual intervention, but we do see traffic impact.

VPN Blade logs Rejects of various types - but generally in sequence:

From remote Cisco IOS XE to CP:
Child SA exchange: Ended with error
Initial exchange: Sending notification to peer: Invalid Key Exchange payload

Then from CP to remote Cisco IOS XE:
Child SA exchange: Received notification from peer: No proposal chosen MyMethods Phase2: AES-GCM-256 + HMAC-SHA2-384, No IPComp, No ESN, Group 20 (384-bit random ECP group)

And from Cisco IOS XE to CP
Informational exchange: Ended with error
Initial exchange: Sending notification to peer: Invalid Key Exchange payload

Actions:
We will be upgrading to latest Jumbo Hotfix (which claims to fix some VPN issues though none appear directly related) next week, but in case that does not solve the issue any help would be greatly appreciated.

We already have our eye on DPD @onfigured on Cisco IOS XE since the CP side has not been configured with the tunnel as "Permanent", but I doubt that would cause IPsec renegotiation to fail periodically.

Also we have requested Cisco IOS XE side to first try with a IKEv2 proposal that exactly matches CP configuration. This has not yet been implemented - as these proposals are "global" - but they are looking into it.

I'd really like to hear if anybody have fixed identical issues?

CP VPN community config:
We have a VPN community ("policy based") tunnel with verified encryption domains (subnets) at both ends.
Only allow encrypted traffic
IKEv2 only
Phase 1: AES-256,SHA384.Group 20
Phase 2: AES-GCM-256, PFS group 20
Not permanent and One VPN tunnel per subnet pair
Shared secret (which of course works)
Renegotiate IKE 1440 (minutes)
Renegotiate IPsec 3600 (seconds)
Anything not mentioned should be at default values for R81.10 (initial deployment for this VSX cluster was on R80.40)

Cisco IOS XE config (which I do not control):
#show crypto ikev2 proposal
IKEv2 proposal: VPN_XXXX_PROPOSAL_AES_CBC
     Encryption : AES-CBC-256 AES-CBC-192 AES-CBC-128
     Integrity : SHA512 SHA384 SHA256
     PRF        : SHA512 SHA384 SHA256
     DH Group   : DH_GROUP_2048_256_MODP/Group 24 DH_GROUP_521_ECP/Group 21 DH_GROUP_384_ECP/Group 20
IKEv2 proposal: VPN_XXXX_PROPOSAL_AES_GCM
     Encryption : AES-GCM-256 AES-GCM-128
     Integrity : none
     PRF        : SHA512 SHA384 SHA256
     DH Group   : DH_GROUP_2048_256_MODP/Group 24 DH_GROUP_521_ECP/Group 21 DH_GROUP_384_ECP/Group 20
IKEv2 proposal: default Disabled

#show crypto ipsec profile VPN_XXXX_PROFILE_10029
IPSEC profile VPN_XXXX_PROFILE_10029
        IKEv2 Profile: VPN_XXXX_PROFILE_10029
        Kilobyte Volume Rekey has been disabled.
        Security association lifetime:3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group: group20
        Mixed-mode : Disabled
        Transform sets={
                TS_XXXX_AES_GCM256: { esp-gcm 256 } ,
        }

#show crypto ikev2 profile VPN_XXXX_PROFILE_10029

IKEv2 profile: VPN_XXXX_PROFILE_10029
Ref Count: 5
Match criteria:
  Fvrf: INFRA
  Local address/interface:
   yyy.zzz.xxx.ooo
  Identities:
   address vvv.uuu.ddd.qqq 255.255.255.255
  Certificate maps: none
Local identity: none
Remote identity: none
Local authentication method: pre-share
Remote authentication method(s): pre-share
EAP options: none
Keyring: VPN_XXXX_KEYRING_10029
Trustpoint(s): none
Lifetime: 86400 seconds
no lifetime certificate
DPD: interval 10, retry-interval 5, periodic
NAT-keepalive: disabled
Ivrf: none
Virtual-template: none
mode auto: none
AAA AnyConnect EAP authentication mlist: none
AAA EAP authentication mlist: none
AAA Accounting: none
AAA group authorization: none
AAA user authorization: none
PPK Dynamic: 0 PPK Required : 0 PPK Instance ID:

Re: R81.10 VPN site-2-site Palo Alto

the_rock — Fri, 30 Aug 2024 14:26:47 GMT

See if my post below helps and how we got it working.

Andy

Might not be exact same issue, but you may find it useful.

VPN route based query - Check Point CheckMates

Re: R81.10 VPN site-2-site Palo Alto

CaseyB — Fri, 30 Aug 2024 14:43:08 GMT

This sounds like a familiar issue I had with a set of Palos. Looking at my old e-mails, I found this snippet:

These are the errors I have been seeing:

Child SA exchange: Sending notification to peer: No proposal chosen MyMethods Phase2: AES-GCM-256 + HMAC-SHA2-256, No IPComp, No ESN, Group 19 (256-bit random ECP group)
Initial exchange: Sending notification to peer: Invalid Key Exchange payload
Child SA exchange: Exchange failed: timeout reached.

The Palo side was using this:

Phase 1 has these configured:

DH Groups 21, 20, 19, 14, 5, 2

Encryption AES-256-GCM, AES-256-CBC

Authentication sha512, sha384, sha256, non-auth

Lifetime: 24 hours

Phase 2 has these configured:

DH Group 19

Lifetime 1 hour

Lifesize 4608 MB

Encryption: aes-256-gc, aes-256-cbc, aes-192,cbc, aes-128-gcm, aes-128-ccm, aes-128-cbc

Authentication: sha512, sha384, sha256

The resolution: Palo side created a profile specifically for our tunnel to use the same encryption ciphers we were sending instead of using a global profile with several ciphers enabled.

Re: R81.10 VPN site-2-site Palo Alto

the_rock — Fri, 30 Aug 2024 14:49:41 GMT

Thats an excellent point! I know PAN guy did that in our case (the post I referenced), though there, main issue was ID he needed from CP side to make it work fully. I believe things like what you advised @CaseyB are less relevant for say Cisco or Fortinet, but for PAN, I do know it matters more.

Andy

Re: R81.10 VPN site-2-site Palo Alto

BjornErichsen — Mon, 02 Sep 2024 06:35:29 GMT

Thanks - but IKEv1 (as I read your solution) is not really an option for us. Audit will be on our backs 🙂

Re: R81.10 VPN site-2-site Palo Alto

BjornErichsen — Thu, 05 Sep 2024 12:49:05 GMT

Thanks - yes I read somewhere that is solved the issue (maybe an older comment by you? ), and this is actually what I meant by "Also we have requested Cisco IOS XE side to first try with a IKEv2 proposal that exactly matches CP configuration" in OP
Specifically put this in top of global proposals or as specific for this tunnel:
IKEv2 proposal: VPN_XXXX_PROPOSAL_AES_GCM_CP
     Encryption : AES-GCM-256
     Integrity : none
     PRF        : SHA384
     DH Group   : DH_GROUP_384_ECP/Group 20

This gives me hope that it might actually solve the issue.

Its just sickens me that the issue apparently arose after an CP JHF deployment, because it leaves little confidence in the future stability of this VPN. Thankfully its mostly used for mgmt access, but still...

Re: R81.10 VPN site-2-site Palo Alto

the_rock — Mon, 02 Sep 2024 18:52:39 GMT

I think that may had been coincidental, I really do.

Andy

Re: R81.10 VPN site-2-site Palo Alto

the_rock — Mon, 02 Sep 2024 20:47:13 GMT

Forgot to ask, did tunnel work constantly without any issues BEFORE jumbo install?

Andy

Re: R81.10 VPN site-2-site Palo Alto

JozkoMrkvicka — Tue, 03 Sep 2024 04:27:29 GMT

What JHF was installed on CP when all seemed to work ?

To which JHF you updated CP when you noticed the issue?

Did you uninstall the problematic JHF to confirm it is 100% related to JHF update ? If so, TAC should be involved and provide root cause

Re: R81.10 VPN site-2-site Palo Alto

BjornErichsen — Tue, 03 Sep 2024 11:23:09 GMT

Yes - at least no users reported errors - and firewall log contains none of these Rejects prior to reload after JHF was applied. If there were any issues they were insignificant. Edit: or not logged...

Re: R81.10 VPN site-2-site Palo Alto

BjornErichsen — Thu, 05 Sep 2024 12:49:53 GMT

We upgraded from JHF take 129 to JHF take 152

We did not try to uninstall JHF take 152 since it took a long time before the issue was discovered (due to summer vacation) and we need security fixes in 152 more than we need this mgmt access to be stable.

We just installed latest JHF take 158, but it does not appear to have solved anything. Next step is to have the Cisco IOS XE IKEv2 proposal changed.

Re: R81.10 VPN site-2-site Palo Alto

the_rock — Tue, 03 Sep 2024 12:41:46 GMT

Hey @BjornErichsen

Did you check out the post I referenced? I can even send you sreenshot of the ID Im referring to. Again, not sure if its 100% applicable in your situation, but worth checking.

Andy

Re: R81.10 VPN site-2-site Palo Alto

CaseyB — Tue, 03 Sep 2024 15:24:34 GMT

FYI - I have an open TAC case with R&D, a lot, if not all my IKEv2 tunnels started generating IKE failures on re-keys in R81.10 JHF 152+. I had to revert to JHF 150 to resolve the errors.

Most of my tunnels are IKEv2: AES-256, SHA256, AES-GCM-256.

Re: R81.10 VPN site-2-site Palo Alto

Alex- — Tue, 03 Sep 2024 15:50:42 GMT

We have solved a lot of VPN issues by going to R81.20 Take 65+ on both VSX and ClusterXL on all sort of VPN configurations.

The VPN feature seems really improved in that version of the OS.

Re: R81.10 VPN site-2-site Palo Alto

Zolocofxp — Tue, 03 Sep 2024 23:16:57 GMT

Having PA create a custom profile will most likely fix the issue. I've had great success limiting the encryption algorithm to CBC like the following example.

I'd also check the Traffic Selectors that are being proposed by both ends when tunnels come down.

Re: R81.10 VPN site-2-site Palo Alto

the_rock — Tue, 03 Sep 2024 23:59:56 GMT

Agree 100%

Re: R81.10 VPN site-2-site Palo Alto

BjornErichsen — Wed, 04 Sep 2024 06:51:43 GMT

Thanks - good to know.
Unfortunately I am stuck on R81.10 management servers a little while longer due to appliances - soon to be hardware refreshed.

Re: R81.10 VPN site-2-site Palo Alto

BjornErichsen — Thu, 05 Sep 2024 12:51:50 GMT

ooh well - that sounds ominous.
You can probably add this post to the case then. Looks like VPN blade errors are not limited to Cisco IOS XE in my end either, but that specific VPN just has more error-logs than all the others combined, and we don't have user complains about the other VPNs (yet).
Currently running R81.10 JHF take 158 (I know JHF 156 is the recommended take for the time being).

Edit: All errors are Rejects - 94% of errors are inbound to checkpoint firewall

Edit-2: Ended up creating a TAC case of my own - will post results once its concluded.

Re: R81.10 VPN site-2-site Palo Alto

Radu_Ciobanu — Wed, 18 Sep 2024 06:40:05 GMT

Greetings,

We're facing the same thing after deploying the latest JHF on 81.10, what was the outcome of the ticket in your case?

The issue isn't restricted to Cisco peers, we can see tunnels with all sorts of vendors exhibiting the same behavior - full re-establishment every ~2 minutes due to "Invalid Key Exchange payload"

Thank you.

Re: R81.10 VPN site-2-site Palo Alto

VSX_Bernie — Wed, 18 Sep 2024 08:43:05 GMT

Hello @BjornErichsen,

Did you have any conclusion on your TAC case?
We are experiencing the exact same issue on all IKEv2 tunnels on a VSX cluster with 45 VS, after having upgraded from R81.10 Take 139 to Take 156.

It really seems very likely that a change to the behavior of phase 2 re-negotiations for IKEv2 tunnels have been made in the code, and this has not been amply (or at all) described in the release notes of Take 152.

The only thing I can find in the release notes that sound like something that could cause this, is the following:
"
PRJ-53366/PRHF-32706

VPN IKEv2 negotiation with a third party peer may fail when the peer offers multiple combined encryption algorithms in one proposal. For example, AWS by default offers AES-GCM and AES-GCM-256. The issue triggers an IKE failure log.
"

This though is noted as a fix in Take 139 - so it does not really add up.

I am quite confident that changing to IKEv1 will solve the issue.
I have not tried this out yet however - as it is not really the solution to the problem - only a "hotfix" so to speak.

I am contemplating creating my own TAC case because of this.