IPsec VPN Certificate-Based Authentication Failed – Authentication Failed

[Venue2185]

We are migrating existing IPsec VPN from Pre-Shared Key (PSK) authentication to Certificate-Based Authentication with 3rd Party Firewall.

Previous Configuration:

• Authentication Method: Pre-Shared Key (PSK)
• VPN tunnel: Working successfully

New Configuration:

• Authentication Method: Certificate
• VPN tunnel: Not established

We are receiving the following error in the logs:

Error: Seq: 4 Auth exchange: Sending notification to peer: Authentication failed MyAuthMethod: Certificate

[PhoneBoy]

Version/JHF of gateway?
What is the third party gateway in question?
What precise steps did you take to configure Certificate-Based Authentication on both ends?

[Venue2185]

Hello @PhoneBoy 

Please find the details below:

• Check Point Gateway Version: R82 Take 107
• Third-party Gateway: FortiGate with DAIP
• IKE Version: IKEv2

Certificate Configuration

• Installed the same external Root CA certificate on both the Check Point gateway and the FortiGate firewall.
• Generated a CSR on each device:
  • Check Point: CN=cpsg.test.com
  • FortiGate: CN=fw3.test.com
• Both CSRs were signed by the same external CA server.
  Installed the signed certificates on both devices.
• Configured the VPN to use certificate-based authentication with IKEv2.

Additional information:

• We tested in both production and lab environments.
• In production, we tested with both Check Point ICA and External CA certificates. The result was the same.
• We also tested in a R81.20 lab, and the issue is the same.

[PhoneBoy]

From this SK, the error message suggests the Fortinet end is not configured to authenticate with certificate correctly: https://support.checkpoint.com/results/sk/sk181787 

[spottex]

Also, tell
* Are you using Internal Check Point cert or public cert. 
* If Public are both sides using the same public CA?
* How may intermediate certs installed and which were installed in Trusted CA and which were installed in Subordinate CA.
* If you configured the interoperable device object for the peer's Matching Criteria with CA and Ip address or email address. (this restricts what Cert your GW will request  from the peer and is more secure)
* If the Peer has installed the same public cert Root and intermediate certs
* If you have installed the Peers Root and intermediate certs (to confirm the Peers cert is valid)

The error obviously saying that at this point of the auth your GW has a problem. (Once sorted it can still fail if there are more configuration errors)
If your interop Matching criteria is requesting a cert from a particular CA and the peer does not have that installed. It will send a random cert or all its certs. Then your gateway will fail the auth.
If the peer is sending a cert and you do not have their Root certs installed etc. Then your gateway will fail the auth. 

VPN debug can be used with ikeview if we can't get it going via here

[Venue2185]

Thanks for your response. @spottex 

• We tested with both Check Point ICA and private external CA certificates.
• Both devices use the same trusted CA.
• There are no intermediate CA certificates, only the Root CA.
• On the interoperable device object, the peer matching criteria is configured with DN only (no IP address or email).
• The peer device also has the same Root CA certificate installed.

Could you please provide the correct procedure or deployment guide for configuring certificate-based IKEv2 VPN between Check Point and FortiGate? We would like to verify that our configuration follows the recommended method.