R80.40 IPSEC VPN shows stuck at Phase I but is fully operational

[scottikon]

I have a R80.40 lab with three Check Point gateways. One distributed environment (managed by separate management server) and the other two are standalone (gateway and management on same device). 

I have site to site VPNs configured as follows: -

1. from Main GW to Remote GW1

2. from Main GW to Remote GW2

VPNs show in SmartView monitor as Up - Phase I but when I look at vpn tu on the cli I see phase II tunnels formed: -

SAs of all instances:

Peer 192.168.101.11 , RemoteGW1 SAs:

IKE SA <b026ba653a85f493,13cd8ad810ce962a>
INBOUND:
1. 0x97698d60 (i: 0)
OUTBOUND:
1. 0x5c3cf41e (i: 0)

Peer 192.168.101.12 , RemoteGW2 SAs:

IKE SA <c33a8776de4d53f1,62554189591c0af1>
INBOUND:
1. 0xa306b733 (i: 1)
OUTBOUND:
1. 0x53ff98e0 (i: 1)

the IKE.elg also shows three messages in quick mode. 

I have also cleared the tunnel down and brought it up by initiating traffic firstly from local network (issuing a ping from PC1 to remote PC 2) and then secondly from remote network (issuing ping from remote PC2 to local PC1). 

Has anyone seen this before?

[PhoneBoy]

Just to clarify, SmartView Monitor is not showing the VPN up but it actually is?
Please open a TAC case.

[scottikon]

That is correct.

This is in a lab environment so no support unfortunately. That's why I was wondering if anyone else had seen this or if it was something specific to my lab.

[HeikoAnkenbrand]

Hi @scottikon 

This issue always occurs if both gateways are in the same external network with the external interface.

Add a router instance beetween VPN gateways in your LAB:

internal network A <> Gateway A <> Router <> Gateway B <> internal network B)   
                                                           <---VPN--->

[scottikon]

Thanks Heiko, I will give this a try

[scottikon]

Unfortunately I put a virtual SRX in packet mode (router) in between the Check Point external interfaces and the VPN is still showing as Phase I. Even though tunnel utility shows full VPN established and the logs show encrypt/decrypt and quick mode key installs. 

[Narsimha_Rao_Ko]

I too observed this in our production environment with R80.40. I am installing latest Hotfix to see if it resolves the issue.

[scottikon]

Thanks Narsimha, be good to hear from you once you have done that.

[BoGa]

Hello Mates,

Does anybody have any update regarding this symptom? I can observe this behaviour after upgrading from R80.20 to R80.40 JHA48.

All of the S2S tunnels operate noramlly, but in SmartView Monitor all of them show State UP - Phase 1

Additionally it is production environment, not lab.

Thanks for all,

Gabor

[HristoGrigorov]

Same here. But I don't mind as long as VPN actually works.

[BoGa]

The problem with this cosmetical issue is, that it hides the actual and real status of the VPN tunnels.

The issue is being investigated by the R&D.

SmartView Monitor is a separate and legacy dashboard. Does anybody know if it will be integrated into the inified console?

Best regards,

Gabor

[Marcel_Gramalla]

We have the same problem with our R80.40 Clusters (JHF 67 and 77) and we opened a case and hopefully Check Point will fix this soon. For us this is not just a cosmetical issue as we rely on that VPN status for monitoring. It's not only the status in the SmartView but also directly via SNMP so we currently don't get any notification if a VPN goes down etc. 

Of course I can check the status manually (if something seems broken) but that's not the point. I wonder how other people monitor the status of their VPNs...

[BoGa]

Hello to Everybody,

The SR was closed - Check Point R&D provided a custom hotfix regarding this issue. Thanks for their development.

We haven't tried the custom hotfix as it is not a good practice imo. We are waiting for the custom hotfix will be integrated to a JHA GA.

[scottikon]

Thank you for letting us know. I agree, since this is cosmetic, better to wait for it to be included in the next JHF, 

[VictorPG]

We received the hotfix and I tried in a lab environment and it indeed solved the cosmetic issue 

[Florin_Dumitru]

Is the custom hotfix (?number?)  available ?

[BoGa]

fw1_wrapper_HOTFIX_R80_40_JHF_T48_605_MAIN_GA_FULL.tgz - so it is suitable to this exact software version & JHA combination imo.

[Florin_Dumitru]

Is there on that would apply to R80.40 Clusters with JHA T77 ? 

[BoGa]

I don't know, so I can't recommend it to use. The best way is to open a TAC SR for this purpose or wait for an upcoming JHA GA, which contains this fix.

[Darren_Fine]

Hi ,

We have the same issue on two clients R80.40 JHF T78.

One was a new install - conversion from ASA - waisted hours thinking the vpns were not coming up !!!! 😞

Opened TAC case.

[BoGa]

Hi Darren,

It is good to know that the fix wasn't integrated into the latest JHA GA - thank you for sharing this information.

Best regards,

BoGa

[Florin_Dumitru]

it's said that take 99 is going to include the fix

[BoGa]

That would be great, thank you for the information 🙂

[MatanYanay]

Hi @Florin_Dumitru , @BoGa 

A fix for this issue will be part of the next JHF ongoing take – the take number is not final yet (99 is not the correct take #).

[Darren_Fine]

 Hi ,

Just wanted to post an update here - that I experienced this issue at two other clients. 

1 x was an upgrade from R80.30 to R80.40 JHFT78 another was a fresh install on R80.40 JHF69 - then upgrade to JHF78.

It seems there is an actual issue with the OID - since external monitoring systems were also getting phase 1 up from the snmp queries against the tunnel status.

I was given a portfix by TAC and this fixed the issue.

So there is a custom fix available hope it goes main train soon.

(This is not just a simple "cosmetic issue" for some clients - their NOC's monitor these vpns since they are business critical - an incorrect output from SNMP disables their ability to accurate track the VPN's functionality and ensure they are meeting their SLA's)

Regards

[Kim_Moberg]

I have seen similar issues in R80.20, R80.30 and R80.40.

Started to monitor VPN connectivity via SNMP following the sk63663 

The SNMP Response should return an integer, whose value has the following meanings:

• 3 = VPN tunnel is active
• 4 = VPN tunnel is destroyed
• 129 = VPN tunnel is idle
• 130 = VPN tunnel is during Phase1
• 131 = VPN tunnel is down
• 132 = VPN tunnel is initializing

Right now I think it is due to timing issues with IKE and IPSEC rekeys.

Recommended settings are

IKE Rekey: 1440 (24 hours)
IPSEC rekey: 3600 seconds (1 hour)

so right now I am adjusting all my 3rd party site2site connection to comply to this recommendation and I keep monitoring the tunnel state and then monitoring the ping between a remote IP in the encryption domain.

As an emergency if I cannot get a vpn back online either because if the the following conditions.

While running vpn tu tlist -p <remote peer> it says there is an IPSEC SA but I still cannot ping remote ip inside the encryption domain

Next I would check vpn tu  using the CLI vpn tunnel client and check for IKE or IPSEC vpn tunnel has an active tunnel. 

Next I would do a tcpdump -penni any host <remote peer> to check if any traffic flows between the public ip and remote peer public ip.

I would as last try clear vpn connection tables. Please note! it will drop all vpn tunnels both s2s and vpn clients.

sk116453

• fw tab -t orig_route_params -x -y
• vpn tu del all

If one clears the vpn connection table one also need to clear the vpn tunnels afterwards.

[RobertSmeikal]

@scottikon Your issue is confirmed and there is a tailored hotfix available to remove it, see sk169121.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[_Val_]

Thanks for the update. 

[Kim_Moberg]

Hi Robert,

I don't see it fixes my issue.

The link to support center relates to SmarView are not viewing the vpn tunnel status correct.

I am using CLI with command "vpn tu" and there is it not active but it stills keeps the tunnel up and cannot be cleared.

[_Val_]

@Kim_Moberg do you have a TAC case for that?