IKEv2 issue with Cisco ASA

[StevePearson]

I'm trying to troubleshoot a strange issue that has hit a common problem of a failure with no changes.

The customer has around 20 site to site VPN's running, with about a third of them running IKEv2. The gateways were upgraded from 5400's to 3920's (R81.20 to R82.10 obviously too), after which, besides a stuck SA that was soon sorted, it's been running fine.

Everything was fine for about a week, then one VPN failed to renew phase 1, receiving a cookie response to the IKE INIT packet. We know nothing was changed our end, and the Cisco guys say nothing was changed their end.

Looking in detail at the key exchange, I would expect to see phase 1 renew every 24 hours (1440 minutes) and phase 2 every hour (3600 seconds), but instead, in the days leading up to the failure I see the Phase 1 at around 6.45am then repeating again every hour for anything from 2 to 5 hours, before it switches to just renewing phase 2, until 6.45 the next day when the process starts again. Each time we receive a delete request for the SA, which is obviously coming from the other end.

Another strange thing is that up to that point of failure we see the Phase 1 presenting and authenticating fine using DH group 14, but the Cisco guys are saying it should be group 19 or 20.

The cookie response, i believe (but i'm no expert!), happens to prevent from DDoS attacks. The ASA sees lots of incoming auth requests, sends a cookie to stem the flow and the other end should then respond including the cookie data to verify it's identity. A tcpdump from both ends simply shows an IKE INIT Initiator Request outbound from us followed by an IKE INIT Responder Response from them to us, and this pattern repeats. (Both ends show the same thing, so we're not flooding them).

My thought is that the ASA is not responding as expected and possibly something changed. TAC are saying that something appears wrong at the other end.

Can anyone throw any light on what may suddenly be happening, or the best way to troubleshoot it at all?

[PhoneBoy]

Which side is initiating the authentication with DH 14?

[StevePearson]

We initiate from the Checkpoint end, traffic only flows outbound on this connection.

One thing that maybe worth mentioning is that all outbound traffic from local source subnets to destination subnets is manually natted behind a specific external IP, not the gateway. But this has been working in the current configuration for years.

[PhoneBoy]

This is most likely going to require some debugging: https://support.checkpoint.com/results/sk/sk180488 
And, most likely, a TAC case.

[StevePearson]

Yes, I'm working with TAC and we've done quite a bit of debugging, I was just putting this out here to see if anyone else had come across similar issues.

At present, TAC are saying that the issue appears to be the Cisco responding in an unexpected way with a cookie

[StevePearson]

Additional info:

The peer is a Cisco Secure Firewall 3130 Threat Defense (80) Version 7.6.2 (Build 329).

Switching back to IKEv1 (AES256/SHA1) and the tunnel comes up, but obviously it can't stay like this.

So looks like a compatibility issue with IKEv2, but this worked for over a week after the Checkpoint upgrade, so I'm not sure it is related to this. 

[Hugo_vd_Kooij]

One interesting observation here might be that Both Check Point and Cisco have some interresting features (bugs) in regard to VPN's. 

I knew about the feature where Check Point will accept a set of encryptions domain that is not an exact match to your configuration but is within the definded encryption domains. This usually leades to the setup wher it works fire if the other side initiates the VPN but not if Check Point does the initiation.

This is usually a configuration mismatch.

The interresting one cam from a conference call with customer and Cisco TAC after I took over on our end and another Cisco TAC engineer took over a call that initialed at 01:30 in the morning. From the new TAC engineer I learned that Cisco has a sort of a similar issue. It will accept a proposal that is not matching the configured setting and bring up Phase 1 and consequently Phase 2. But as soon as Phase 1 needs to rekey the tunnel breaks. And it will remain broken until one of the 2 side to reset the tunnel and start a new tunnel where the whole issue will repeat itself.

Again the issue is with a mismatch in the configuration.

Lesson learned: Just beacuse you got it working once does'nt mean your VPN configuration is correct. 

So in each and every VPN related issue I insist on checking the configuration on both end to make sure we are 100% identical.