VPN Check Point - Palo Alto issue

[the_rock]

Hey everyone,

Hope someone can maybe give a good suggestion/idea about this. So I was helping a hospital with route based VPN tunnel from their CP cluster to Palo Alto and this tunnel had been there since 2020 I think, but always working intermittently.

PAN guy was saying that for some odd reason, when there is an issue, they see ID on their end as 0.0.0.0, though it should be 169.254.0.103, which is whats configured. Im not really sure why that would happen and if its something related to CP or Palo Alto.

For the context, all other vpn tunnels on CP side work just fine, just this one. Any clue as to why the ID would show different on peer side?

We did not have time to really debug, so simply ended up lowering the encryption methods and that brought up the tunnel.

Thanks as always for any ideas. Just for the context, we tried unnumbered VTI as well, that would send ID 0.0.0.0, so thats expected, but with numbered VTI, definitely should NOT.

Andy

[the_rock]

Just looked at all other debug files generated that TAC sent from the remote and absolutely cant find single line containing KEY_ID. Let me see if guy responds after they check if that file was even generated last night when debug was initiated after ikev2 was set.

Andy

[AmirArama]

sorry, for GAIA look in:

iked*.ikev2trace files 

KEY_ID is just something that existed on specific versions of Spark for ikev2 against 3rd parties. you can ignore it.

Usually it should be IPV4_ADDR instead.

[the_rock]

Funny enough, I checked this yesterday, but probably did not know exactly what to look for. Now we are getting somewhere : - )

Anywho, tx for being so persistent and helping with this, means a lot! Btw, I attached the file, peer is 10.66.52.18 and based on screenshots below, looks like IP of the peer is being sent indeed, right? Not sure what 2nd screencap indicates : - (. Dang, cant attach the file, does not support the extension, if you are curious to review it in ikeview, I can email it, just message me directly.

Andy

 

 

[the_rock]

Hey @AmirArama 

I just wanted to update one more thing for now. Maintenance window wont happen till 1st week of August, since couple of boys are on vacation, BUT, in case anyone may have doubts about the ID PAN guy was referring to, its what I pointed out below. I credted PAN VM with never expiring license, so it lets me test any feature available.

Thanks again so much for all your help, I cant thank you enough.

Have a nice weekend.

Andy

 

[Duane_Toler]

You also say this is a route-based VPN?  That generally implies universal tunnels ("tunnel per gateway") but DPD is still optional.  Is the peer's 10.x IP something you wrote for sanitization?  Just want to be sure so it's not being mixed up with the numbered VTI IPs.  Even with route-based VPN, the IKE peer IP should still be the IP of the interoperable device.

The debug shows "ikev2" here.  Search the debug and see if your gateway was acting as Initiator or Responder.  You can search for "Here's the IKE SA we dug up" and the stanza after that gives a concise output (assuming all went well enough up to this point) and shows "initator=..." ("me", or "peer").

To try to normalize some things, and keeping with route-based VPN, I'd suggest switching [back] to:

* Tunnel management: tunnel-per-gateway

* Edit the gateways list  in the VPN community and override VPN domain for both gateways to only be empty group objects

* Disable permanent tunnels

Bonus (if PAN can agree: 

* Encryption: IKEv2 Only, VPN Suite B GCM 256 (Phase1: AES-256/SHA-384, DH Group 20 [ecp384], Phase2: AES-GCM-256 (implies SHA-256), DH Group 20 [ecp 384]).   Otherwise, get it as close as you can.

You're welcome to share any other debug snippets if you'd like.

[the_rock]

Its actually not route-based at this time. We tried that about month ago, but it was way worse outcome, sadly...

At this time, its strictly domain based vpn tunnel.

Yes, PAN side agrees with those higher enc methods, but the issue is, when they are in place, tunnel never comes up.

We still find it odd that randomly, when those methods asre in place (ikev2, eas256/sha256), CP side shows as up, but their side is down? How in the world is that possible??!! 

Andy

[Duane_Toler]

Interesting.  Sounds like PAN might have a bug. 🙂

So if you're domain-based, then tunnel-per-subnet is correct with VPN domains defined (override in the community for the gateways).  Otherwise, if you are using a monolithic VPN domain like old days (pre-R80.40), then you might have something in your giant VPN domain group that is being sent when you're initiator that neither side expects.  Maybe you have an odd address range object in that group, or nested groups, or someone added the "All_Internet" object... who knows.

Since it's domain-based VPN, get rid of the VPNT interfaces in CLISH as well since they really won't be matched anymore and that might be compounding some issues.

Regardless of IKEv1/v2, I'd still suggest disable DPD "just because".

[the_rock]

Hey Duane,

Yes, so those VTI interfaces were removed last time we had the window when we could not get this working as route based tunnel. DPD is also disabled. 

Again, keep in mind, VPN encryption domain is all subnets now, ho individual hosts, but what really baffles me is that on CP, this is the ONLY tunnel that has these issues and on PAN side, guy said its also the ONLY tunnel they have problem with, all else works.

WTH 😂😂

I dont get it man, Im totally puzzled at this point how to make this work with ikev2 and higher enc methods. Thats why I was thinking of suggesting them to try "per gateway" option, but then again, since ONLY subnets are used, not so sure that would make a whole lot of difference...

Andy

[the_rock · Accepted solution]

Though we dont know for sure until next maintenance window, Im fairly positive this is NOT CP fw issue at this point. Thanks a lot @AmirArama for being so kind to get the ike trace file and confirm that ID being sent to PAN fw is not 0.0.0.0 as they claimed during the call. We also verified with TAC that correct auth methods were being used, so at this time, if they have issue next time, I will advise them to open the case with Palo Alto support and investigate further.

Appreciate everyone who responded.

Best,

Andy

[the_rock]

By the way, I feel this is IMPORTANT info to share, in case anyone else has this problem. As Amir pointed out, which I also got an email from TAc about it, what you need to look for are below lines (sections) of ike trace file, which you dump into ikeview utility.

IDi means ID of an initiator, and IDr is responder ID, in this case Check Point. I blurred it out, but its defnitely NOT 0.0.0.0 lol

Or, if you prefer, open the file in txt format (preferable just basic notepad) and you will see the same

Screencap:

 

Notepad:

<Payload Type="IDi" Next="Auth" Length="12" Critical="No">
<Type>IPV4_ADDR</Type>
<Data>10.66.52.18</Data>
</Payload>
<Payload Type="Auth" Next="Notify" Length="40" Critical="No">
<Method>Shared Secret</Method>
<ndata>91 1d 5f b7 47 b2 9a 43 61 61 5c 02 2d dd 9b ba bf 09 05 f0 8e c0 3f c2 83 97 c7 8b d5 60 db 4e</ndata>
</Payload>
<Payload Type="Notify" Next="SecurityAssociation" Length="8" Critical="No">
<Protocol>0</Protocol>
<Type>ESP TFC (Flow Confidentiality) padding not supported</Type>
<spisize>0</spisize>
</Payload>
<Payload Type="SecurityAssociation" Next="TSi" Length="44" Critical="No">
<prop ID="1">
<SPI>b6558b75</SPI>
<encr>AES-256</encr>
<integ>HMAC-SHA2-256</integ>
<esn>No ESN</esn>
</prop>
</Payload>
<Payload Type="TSi" Next="TSr" Length="24" Critical="No">
<Count>1</Count>
<TrafficSelector Type="TS_IPV4_ADDR_RANGE" Length="16">
<range>142.21.42.0 - 142.21.42.255</range>
</TrafficSelector>
</Payload>
<Payload Type="TSr" Next="None" Length="24" Critical="No">
<Count>1</Count>
<TrafficSelector Type="TS_IPV4_ADDR_RANGE" Length="16">
<range>205.189.236.32 - 205.189.236.47</range>
</TrafficSelector>
</Payload>
</Payloads>
</Message>
<Message Valid="Yes" Initiator="No" Response="Yes" higherVer="No">
<arrivalTime>2024-07-09T18:47:55</arrivalTime>
<MsgID>1</MsgID>
<initSPI>058549a770dcf5b2</initSPI>
<respSPI>5de9d809db058be0</respSPI>
<Next>Encr</Next>
<Version>2.0</Version>
<Type>Authentication</Type>
<Length>272</Length>
<Payloads>
<Payload Type="IDr" Next="Auth" Length="12" Critical="No">
<Type>IPV4_ADDR</Type>
<Data>2x.xx.xx.xx</Data>
</Payload>
<Payload Type="Auth" Next="Notify" Length="40" Critical="No">
<Method>Shared Secret</Method>
<ndata>a4 0b ee c6 a1 0c 82 1d 26 aa 3b 0a 23 28 b3 d1 80 e5 34 b1 90 27 40 e6 80 08 8c 74 64 95 24 2b</ndata>

 

[Duane_Toler]

Yeah that's kinda what I was thinking, too.  Good that you have confirmation, too! Definitely something in this configuration that PAN isn't liking.  Hopefully you can push their side to get some serious attention to fix it!

[the_rock]

I emailed the boys and guy that has access to PAN already asked us when we can do next remote , so they can change the ID to whats showing in the debug

Go figure 😉

Andy

[the_rock]

I love how our "friend" chat GPT answered my question 🙂

Andy

 Though, when I asked more precisely, its on point.

 

[the_rock]

@AmirArama Just wanted to let you know boys set maintenance window for August 28th after hours, so hopefully should be quick change and I will make sure to remember to update the thread to let you know if the change we discussed worked 100%. I am sure we will know as soon as its done, because so far, as soon as we would try higher/stronger enc. methods, tunnel would never work, so if its different this time, that would be a good sign.

Thanks again for all your help, just know I even emailed one of manegeers over there in R&D to tell her how awesome you are.

 

[AmirArama]

Thank you 🙂

Let me know your results 🙂

[the_rock]

Will do! I really have a good feeling about it, specially after all you showed me and also great guy from DTAC I worked with many times before, I always found him to be very knowledgable and patient, he explained it exactly the same way you did, so Im super confident it will be fine.

Either way, I made a reminder to update the thread after we have that call. It was supposed to happen next week, but one of the guys is on vacation, so has to be in 2 weeks.

Best,

Andy

[patrickpreuss]

Hello 

Maybe you can clarify "but always working intermittently". Does this mean that you can ping over the CP / PAN without other traffic?

Does this break a some point when traffic starts flowing or does this happen close the Phase 1 or 2 Timers or bytes configured? 

A few years back we had a case that the IPSec Packtes got reordered on the other side, so reply-protection kicked in. This was caused by hardware, two crypto processors and smaller packets where faster the larger packets. 

This can also happen when you have port channels in use along the path.  

   Patrick 

[the_rock]

Hi Patrick,

Im fairly sure we now have good idea the ID palo alto side needs to use on their end, based on the debugs, so that will be changed next time we have window, probably last week of July. I will upodate the thread then.

Andy

[the_rock]

Hey guys,

Wanted to say we got all this fixed. Tested with strongest enc. methods once PAN side put in the ID number from ike trace file Amir Arama provided from the debug, so THANKS TO ALL OF YOU FOR HELPING, I APPRECIATE IT!! 

Thanks again @AmirArama @patrickpreuss @Duane_Toler @CaseyB @Timothy_Hall

 

[Duane_Toler]

Dude that's awesome!  Nicely done!

[the_rock]

I appreciate all of you, but special thanks to @AmirArama 

Best,

Andy