R80.40 JHF 120 - S2S VPN issue

[Juan_]

Hey Lads,

A customer installed JHF 120 this morning and many S2S vpns didn't come up.

Solved it by reverting to JHF 118.

Am about to raise to TAC for them to have a look at the debugs but wanted to just give the heads up to the community.


Juan

[Garrett_DirSec]

Please provide updates on your investigations.   would be good to understand.

There are a ZILLION fixes in JFA take #119.

reference:    R80.40 Jumbo

[Ruan_Kotze]

We had exactly the same issue at a customer site.  All S2S tunnels failed with "local interface spoofing".  Had to revert back to T118.

Ironically, TAC had us install this take troubleshoot an issue with a single S2S that wouldn't come up.

[genisis__]

Could this issue be with T119 rather then T120?  According to the official list only two things have been added since T119 (If you believe that).  I did also find it strange that T119 did not go into GA and T120 in ongoing.

Above said, I have installed T119 on a gateway with a S2S VPN and we have had no reported issues.

[Kim_Moberg]

Would be interesting to hear if the condition are related to:

- CP2CP tunnels

- CP2 3rd party like Cisco

- IKEv1 or IKEv2

- Remote peer behind NAT?

thanks

[Juan_]

Yeah, the one with the VPN fixes is 119, but 120 went GA. That's why i mentioned 120.

[the_rock]

wow...thats unfortunate, but glad it was solved. It would be good to find out what is causing this with take 120.

[Ilya_Yusupov]

@Juan_ ,

can you please share more info about which tunnels exactly are impacted?

I'm interesting to understand the encryption suite of those tunnels and Ike version.

Thanks,

Ilya 

[Juan_]

Hi Ilya,

One tunnel is ikev2 using:

P1: aes2 sha2 group 20

P2:aes2 sha2 group20

Other 2 that we documented failing are ikev1:

P1:3des md5 group2

P2: 3des md5

More info:

• Remote peers are of different vendors, one of them Palo Alto. Not aware of the others
• Cluster is behind a fortigate that NATs the CPs IP.
  • Link selection is not configured
    • It's just "main ip address" which is a private
• One of the VPNs on JHF_118 goes over ESP, the other 2 over 4500
  • Didn't check it on JHF_120, we reverted quickly.

By responses of others doesn't seem to be so widespread though there seem to be a couple of issues. I can send you the SR number if you want.

Juan

[Kim_Moberg]

Hi Juancho

When running tcpdump do you see ESP packets intiated udp/500 and reply on udp/4500? or is it both ways sending udp/500 or udp/4500? should be the later.

For example

tcpdump -penni any host x.x.x.x 

x’s representative public ip of remote peer.

thanks

[smoraneng]

Hey all, I'm not sure this 100% relative, but we're seeing something else with ike v2 s2s vpn's that's different in 120 vs other versions.  GW general properties is set to a private IP, link selection is set to public IP.  In versions under 120, the peer side sees the General Properties IP, in 120, phase 2 is seeing the public IP.  

[the_rock]

Well, technically, lots of customer would have it that way and thats perfectly fine, as VPN would look for whats configured in link selection, so whatever is set there, would be what is taking as main public IP...what option do you have configured in that tab? 

[smoraneng]

So with no configuration change on the gw other than applying HF 120  The general properties is set to the 10 ip, the link selection is set to public IP.  

[the_rock]

Then Im really not sure...seems like take 120 problem, I would say.

[Brandon_Pace]

If after upgrade, the IP presented for the ID payload is the Link Selection address, then note that an issue was fixed and the peer's configuration should be updated to match the expected (link selection) IP. (There was an issue in IKEv2 where we would use the main IP instead of the link selection IP)

In the jumbo sk in Take 119 there is this note regarding that fixed issue:

• "Invalid ID information” message may be displayed when peer is 3rd party and Link selection is overridden.

[smoraneng]

We did get a custom patch made to reverse the invalid ID information issue.  

[RS_Daniel]

Hello,

Just wanted to share that we had problem with only one S2S vpn after installing JHA Take 120. In our case the vpn is completely down (checked with vpn tu options 3 and 4) and smartview monitor. However, the gateway is keeping and old SPI (seen with vpn tu tlis) and is trying to encrypt the traffic with that SPI. As this SPI is incorrect the peer rejects our traffic which is normal. The problem is the checkpoint gateway not deleting that SPI, tried to remove the gateway from the community but did not work, SPI is not deleted. Working with TAC to see how to fix this on Jumbo 120 as we waited months for a fix included on Take 119. 

Regards

[the_rock]

Hm, thats interesting...I know R&D told me before not to worry what it shows in SV monitor, as that could be misleading as far as vpn tunnel status and I do agree, as thats my experience as well, so vpn tu is way more reliable and more correct. So you are saying only 1 tunnel is broken...is it with cloud provider, cp to cp or cp to 3rd part, just wondering?

[Wolfgang]

Can someone please provide SR number or anything else regarding this issue?

Reading about this issue here stops us for going to take 120. We need 120 to solve some other problems but failing S2S tunnels is the more complicating problem.

Wolfgang

[idants · Accepted solution]

Hi all,

I am Idan Tsarfati, IPsec VPN R&D group manager.

Regarding the problem in take 120:

We changed a behavior which related to narrowing feature.

In case where CP GW connected with a 3rd party peer and there is a mismatch with the encryption domain configuration the tunnel  might stop transfer traffic.

Until this take, the narrowing feature fixed a situation where we have universal tunnel in CP side and 'tunnel per subnet' in the 3rd party side.

In order to make it work, the misconfiguration should be fixed - a SK will be published tomorrow on how to recognize it.

Basically if you see **narrowing** in vpn tu tlist in the relevant tunnel, this is the case.

I am not familiar with any other issue in this take - the known issue is relevant only where the environment has a configuration issue and it should not widely happen. I am familiar with only 2 cases.

I know about many customers who use take 120 (or 119 which is identical) and it works very well.

The take include many fixes and solved many issues.

[Garrett_DirSec]

hello @idants  -- thanks for the background and details.  very helpful.  

[RS_Daniel]

Hello @idants, thanks for the reply. in our case I think it is a differente problem. just FYI In our case the problem is not similar to what you describe. In vpn tu tlist we can see:

+-----------------------------------------+-----------------------+---------------------+
| Peer: Peer_IP - Peer_Name | MSA: ffffc9005ee299e0 | i: 0 ref: 10 |
| Methods: ESP Tunnel PFS AES-256 SHA256..| | i: 1 ref: 8 |
| My TS: Local_Domain | | |
| Peer TS: Remote_Domain | | |
| MSPI: 15 (i: 0, p: 0) | Out SPI: 990e0f27 | |
| Tunnel created: Aug 10 17:33:58 | | |
| Tunnel expiration: Aug 11 17:33:58 | | |

Check we have Out SPI: 990e0f27. But the tunnel is down, when we look for this peer using vpn tu,  nothing appears so it is not possible to delete it from there. The tunnel does not appear on smartview.

doing a tcpdump we can see that CheckPoint gateway is trying to use that SPI.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond2.73, link-type EN10MB (Ethernet), capture size 262144 bytes
11:22:48.296561 IP My_PublicIP > Peer_PublicIP: ESP(spi=0x990e0f27,seq=0xa799c), length 104
11:22:49.148963 IP My_PublicIP > Peer_PublicIP: ESP(spi=0x990e0f27,seq=0xa799d), length 104
11:22:49.298097 IP My_PublicIP > Peer_PublicIP: ESP(spi=0x990e0f27,seq=0xa799e), length 104

As this phase two SA is not present on the peer they reject our ESP traffic:

ESP request discarded from My_PublicIP. 

The SPI expires today, so i will check if it is deteted at that time. 

Regards

[Kim_Moberg]

I would have cleared vpn connection tabels and then restart all vpn’s

please be aware of this cmd will drop all tunnels.

from expert

1) fw tab -t orig_route_params -x -y

2) vpn tu del all

I have seen this issue multiple times that ‘vpn tu’ shows no active tunnels BUT ‘vpn tu tlist -p x.x.x.x’ shows an active tunnel med a SPI. It is not related to IKEv1 or IKEv2. I have seen this on both.

with the cmd all tunnels goes down but reestablish itself again when it sees traffic.

Do you also see traffic flowing between peers when doing tcpdump? Is it oneway or two-ways communications?

3) tcpdump -penni any host x.x.x.x

Thanks

[the_rock]

Good to know, thank you very much for the info!

[genisis__]

Was this behaviour change introduced in Take120?

[Vladimir]

Idan,

Thank you for the clarification. Just in case, I would like to have the access to the T118, as we are performing new cluster deployment with multiple VPNs. The JHFA Archive download results in unusable files (tried Chrome, Firefox, Edge for downloads from https://supportcenter.checkpoint.com/supportcenter/portal/role/supportcenterUser/page/default.psml/m... with hash verified) and am getting:

Import of package Check_Point_R80_40_JUMBO_HF_Bundle_T118_sk165456_FULL.tgz Failed
The imported package is not in the correct format. Please use a package file in the format of *.tar.
Contact Check Point Technical Services for further assistance.

[PhoneBoy]

The relevant SK @idants mentioned is: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[DennisJ]

Thanks for posting the SK.

As a follow-up to this, if I run the vpn tu tlist command and don't see either * * * Eclipsed * * * or * * * Narrow * * * in the output then applying JHF 120 should not cause an issue with our site-to-site VPN connections, correct?

[PhoneBoy]

There shouldn't be an issue, and if there is, the issue in this SK won't be the cause.

[Marcel_Gramalla]

I wonder if this helpful SK can be added to the "Important Notes" section on the JHF page. If I was not following this thread we would probably encounter this issue on two VPNs where we can find a "Narrow" flag. Now I can fix this or let it get fixed before we update the Gateways. 

I think it's great that we sometimes get big enhancements and new features within JHF and not only major versions. They are sometimes already documented in an SK but that's useless if we only get a hint after manual research or TAC. Many issues could be avoided if the SKs were directly mentioned on the relevant JHF page (sometimes they are, sometimes not).