Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Thomas_Eichelbu
Advisor

How the firewall selects its available certificates for VPN?

Hello Check Mates, 

a question for an issue i have seen several times so far
Somehow, out of a sudden a VPN gateway tries to establish a VPN tunnel with the "wrong" certificate ..


In the latest occurance the issue started without any plausible root cause.
The VPN was no longer working after a policy install. Nothing critical has been changed.
In the logs we saw "invalid certificate" messages
Iam very sorry i have no good screenshots or logs to share ... 

In Ikeview we saw the local ICA certificate and a third party IPsec certificate, nothing really strange.
the third  party certificate is used for remote access only! All VPN´s are made with PSK, only the VPN inside the own MGMT is of course using the interal CA.

most likey it was:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

and

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...
perhaps when the peer gateway has problems to reach the interal CA it falls back to any other CA? 

it seems easy to solve, and i can remember i fixed the issue once by running through this SK.

but i have some follow up questions, maybe you guys can enlighten me:

+  how the gateways determines what certificate it has to use? iam pretty sure it will offer all available certificates to its peer, and if one of those CA´s can be validated a tunnel can be formed... if the ICA is not reachable and the other certificates fails to be valdidated, no tunnel. But how is the process really?

 + if i have many certificates running on my gateway, is there a way to set a priority or set the certificate hard coded?
i know you can set the "match criteria" button on an external managed GW and the button "tradition mode" the SK 323648 is reffering too. But is there a better way?

+ since the SK refers to a "Traditional Mode configuration" setting, why is it still working?

i hope you have some good answers for this!


best regards
Thomas

0 Kudos
18 Replies
the_rock
Legend
Legend

I carefully read your description and I find this very odd, to say the least. Here is where I would start...are you having issue with just a particular tunnel or all more than one when this happens? For VPN cert, just make sure its valid under gateway object -> VPN...its valid for 5 years if Im not mistaken. If this happens with multiple sites and multiple gateways, then its possible is ICA issue.

0 Kudos
Thomas_Eichelbu
Advisor

Hello, 

well in this particular case it was only one tunnel out many ... all tunnels on this customer for its own peer have indepedent vpn communities. so one peer per community ... all tunnels UP, only one Check Point ot Check Point is down


And those peers use link selection, one link is MPLS and other is internet, in HA i think.
of course all certificates are valid, nothing has expired and we reissues the IPSec certificate on the remote peer too!

i dont think the CA is affected, its more likely a communication issue between remote peer and internal CA which might triggers this behavior... but i see prove for that ...
we will try to fetch the CRL via curl_cli then we will see.

best regards

0 Kudos
_Val_
Admin
Admin

Before answering your questions, some clarification:

  1. Which certificate, ICA or third party, was used for the tunnel on this GW? 
  2. Which part is reporting "Invalid certificate"?
  3. Did you change anything on your networks that would prevent CRL connectivity?
0 Kudos
Thomas_Eichelbu
Advisor

Hello Val, regarding your questions.

  1. Which certificate, ICA or third party, was used for the tunnel on this GW? 
    -> only the internal Check Point CA, 

  2. Which part is reporting "Invalid certificate"?
    -> well i have to less logs and debugs to make a qualified statement about that.
    i need to arrange the collection of more logs
    the remote peer says "invalid certifcate" it wants to validate the third party certificate, and fails.

  3. Did you change anything on your networks that would prevent CRL connectivity?
    -> since i know, not at all. as i said out of a sudden after a policy installing applying a "normal" access policy.
0 Kudos
_Val_
Admin
Admin

Ok, so no third party certs. Check you can reach CRL from the GW which refuses the cert. CRL is cashed fro 7 days, so it is normal if your VPN failure does not correlate with any recent changes

0 Kudos
Thomas_Eichelbu
Advisor

Hi, 

yes i asked the customer to check this ... if the ICA is reachable via MPLS and Internet as well ... 
i have only received logs from the central VPN gateway ... 
for example ... since it is so many ...

208.44.YY.ZZ -> is the remote external IP
192.168.254.XX -> is the remote internal MPLS IP
both are in link selection, in HA, 192.168.254.XX is set to primary.

[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] New TransportConnection (9765669 Total: 25)
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] UDPConnection::UDPConnection: Enter (copy ctor) peer: 192.168.254.62
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] UDPConnection::UDPConnection: conn.m_txSocket: 0x1b1ee458, 0x1b17fc20.
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] GetEntryIsakmpObjectsHash: received ipaddr: 192.168.254.XX as key, found fwobj: FW-USAM
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] extended_log_info_create, entered.
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_ROLE_START > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_ROLE_RESPONDER > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] FwIkeResponder: entering
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] FwIkeResponderOnEnter: idRanges NOT USED mine [0-0] peer's [0-0]
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] findSAByPeer: Find SA with cookies 5362e2e0f6a51e6b,30eb62504c4a471e from packet
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] findSAByPeer: ISAKMP SA was found
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] ResponderOnEnter: create new p1state
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] GetEntryIsakmpObjectsHash: received ipaddr: 208.44.YY.ZZ as key, found fwobj: FW-USAM
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] ResponderOnEnter: set peer ike port to: 500
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] ResponderOnEnter: client mode: 0
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] GetEntryIsakmpObjectsHash: received ipaddr: 208.44.YY.ZZas key, found fwobj: FW-USAM
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] GetEntryCommunityHashX: received ipaddr: ZZ.YY.44.208 as key, found community: vpn_USAM-1
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] FindCommonCommunity: Found common community (IPv4 addr=XX.YY.44.208) (vpn_USAM-1) for FW-USAM
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] FwIkeNewPhase2State: Community uses profile custom_profile
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_EXCH_START > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_EXCH_INFORMATION > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_PACKET_START > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_INFO_RESPONDER > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] ProcessInfoHeader: peer sent non-encrypted info-exchange while ISAKMP SA exists.
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] ProcessInfo: enter
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] fwIsakmp_ProcessInfoExc entering
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] -- updatePayloadMap: received payload PA_NOTIFY.
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] ProcessInfo: identifyPayloads succeeded.
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] processNotifyPayload: protocol: 1
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] processNotifyPayload: notify type: 20

[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] Peer d02c0b1e says: Received Notification from Peer: invalid certificate

[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] Received Notification from Peer: invalid certificate
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] extended_log_info_build_reason_from_list: list is empty,
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] isakmpd_log: calling isakmpd_log with original reason=(Phase1 Received Notification from Peer: invalid certificate)
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] GetDAGIP: ID d02c0b1e not in DAIP range
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] CFwdCommStreamLocal::Write called
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45] CFwdCommStreamLocal::Write sent 220 bytes
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_PACKET_END > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_EXCH_END > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] < FWIKE_ROLE_END > Id = 1699451
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] TalkToEngine: Engine RC is << FWIKE_RCV_NOTIFY >>
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] TalkToEngine: received Notification from peer
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: Killing negotiation 1699451 (0x1b1ca478) ...
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: p2state isakmp sess id:
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: machine state -exchange: 0
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: machine state -packet: 0
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: marcipan state: 0
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: status: 0
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: cookieI e0e262536b1ea5f6
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: cookieR 5062eb301e474a4c
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] KillNegotiation: fwisakmp error type: 0, code: 20
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] NegotiationTable::DeleteNegotiation: Invoked for:
[vpnd 23263 4092643232]@FW-BR-MB[8 Jul 13:56:45][tunnel] neg ptr: 1b1ca478 ass: 1b2ac088 wait4: 0

what key words i should search for?

0 Kudos
jeannotx
Explorer

same problem here, any ideas?

0 Kudos
flachance
Advisor

Did you find  a solution for this? I'm running into the same problem

the_rock
Legend
Legend

So glad you brought this up again, because I saw few other posts about it since as well. I keep checking support site for this, but cant find any SKs that would give any insight into this.

Andy

0 Kudos
spottex
Contributor

Certificate requests need to be adhered to so whatever one GW requests, needs to be fulfilled by the other gateway. If not the auth will be failed.
Cert Requests are created by which Trusted Root certs are installed and what is configured on the vendors gateway.

In Check Point Cert Requests can be configured (to send to your Peer) to be specific and is done in the interoperable device > IPSec VPN Matching Criteria.

e.g. if you want the peer to send a cert verified from a certain Verisign root cert. That Root cert needs to be installed on your CP and it's subordinates. Then configure matching criteria to the Verisign subordinate root cert.

If this is left blank the GW's cert request packet will hold all your Trusted Root certificates. 
The peer will try to match one and send the cert. If there is no match, the rfc states it needs to send one. If it has a default cert (in CP that is the CP Internal cert) and it will send that. 
 *If the peer has your GW Root Cert install as trusted the cert will be verified.

*quick note. Each gateway can request a different cert on one VPN connection i.e. you can request Verisign they can request CP internal cert. As long as both have the correct root cert to verify the cert. And the correct cert installed to send.


If the peer can be configured to modify its requests, it will need to be a cert installed in your GW VPN config. They will need to also trust it to verify. Other Vendors can install a cert without having to have the root cert installed first - make sure they confirm they have your root cert installed.
If a peer cannot change their requests, then you need to install the root cert into your GW and create a VPN cert in the GW VPN config. Sonicwalls are notorious for this and if the administrator is not sure on how to find out the cert request, you will have to debug the vpn on the CP.

CRL/OCSP URL needs to be accessible from a CP GW otherwise the cert will not be valid. You can turn it off in root/sub cert.  OCSP is the default now for CP, if that fails it should revert to https, but for R81.x this revert feature has only been fixed in recent JHF's. If you have an internal OSCSP server you need to make sure DNS routes rules and NATs capture and allow the traffic. You would see the issue in the vpnd.elg - something to do with cannot reach OCSP to verify etc.

So if the peer sends a cert request that does not match a cert installed on your GW VPN config it should send your default cert. But I have seen the GW send other certs. Looking in GuiDBedit I was trying to find the cert it would send - e.g. if it was the order in the database or if there was a priority number. I have not found this yet so it is still mystery on which cert it does choose. All but one VPN I have seen it send the CP internal.
I did harass TAC for approx 6 months about this but we never got to the resolution.

Viewing your vpn debugs with ikeview you will see the cert request packets and the certs sent packets for each GW. That will give you a good picture of what is happening if you compare requests to sent.

In your case your peers request has not been satisfied and they have failed the auth.
Could be certs expired either end (installed VPN certs or Trusted etc  will no longer be requested or sent) or they have changed their cert requests.

(1)
spottex
Contributor

Steps:
Capture VPN communications using debug
#vpn debug trunc
reset the VPN using 'vpn tu' or right-click reset in FWmonitor
Create traffic so the VPN authentication starts
Wait for 15 secs
turn off debug
#vpn debug ikeoff
#vpn debug off

Download the files from your GW using something like Winscp.exe location $FWDIR/log/
if ikev1 : legacy_ike.elg
if ikev2 : legacy_ikev2.xmll

Get your hands on ikeview.exe and open the debug file
look for the peer ip if it is not there redo the above steps.
Then find the cert request from the peer - it will be asking for a cert signed by a certain Root cert.
Compare it to the response that includes your cert from your GW
It may have multiple certs in the reply, they will be the cert chain certs Roots/Subordinates etc, so check them all and make sure the highest cert in the hierarchy is signed by the cert in the cert request.

0 Kudos
spottex
Contributor

Something I thought of later. If the peer is checking Alternative certificate details e.g. IP addresses, email address etc when the Cert arrives, and it doesn't match then adding an alternative IP to the cert when installing it on CP... i.e. the IP of the Interface that the VPN traffic leaves will help. It may need configuring on the peer to check IP instead of other things like email address or domain.
If multiple alternative IPs are added to your cert, vendors like Sonicwalls need the IP address of your external interface to be at the top. i.e. added first - guess how long that took to work out.

0 Kudos
the_rock
Legend
Legend

Not sure if its same with CP, but would make sense.

Best,

Andy

0 Kudos
spottex
Contributor

For the Alternative IP's etc, no the Check Points don't care, But we had to add the alternative IP to the cert when installing one on the CP to work with the Sonicwall VPN.
The VPN was leaving the internal interface (some legacy reason we inherited) but the link selection IP was set as external interface. The Sonic wall would drop the traffic with invalid cert until we matched it's checks by adding the IP to the cert .

0 Kudos
the_rock
Legend
Legend

Might be worth having TAC confirm all this on CP side.

Andy

0 Kudos
spottex
Contributor

I read the RFC's for PKI ikev1 and ikev2 then tested on the CP's.

I had a TAC case logged for the IP issue but they struggled massively I don't think they had yet encountered IPSEC cert use at all except for remote access. An email a week asking the same questions for two months. I worked it out in a lab then asked for access to the Sonicwall, as the Admin for that was also emailing me once a week explaining why he thinks his config "should" work and it is a Check Point issue. Technically it was a Sonic Wall issue that could be fixed from the CP.

The other issue where OSCP was unable to connect so needed alternative to verify certs as turning it off would not help. After 6 months and about 8 engineers it got to R&D which then only took them a week to find out that the code was only added to IkeV1 so they updated JHF about another 6 months later with the fix for Ikev2. Our workaround was to connect to an internal OCSP server for one customer, Others had to give internal GW's access to the internet or revert to PSK.

The third TAC case occasionally the CP would send the internal cert and the VPN would fail. Emails from TAC were KB articles that were unrelated, so I gave up on them.
I persevered with debugs, as the VPN failed... The peer had the same root cert but two different subordinates so the cert request could only send one request for all VPNs and didn't want to change it. It did not match any installed certs on the CP, therefore the CP would sometimes send the cert that could be verified then other times send the default CP internal cert.  There is a KB somewhere saying CP does not support multiple trusted certs from the same root certs. We tested it and it would still sometimes mismatch. So we uninstall the second cert.
The peer could have installed the CP internal root cert but the customer did not trust the CP cert (even though I provided all the tech details of the cert from a CP KB article) This VPN was reverted back to PSK.

I passed on the above details about cert request etc so TAC could create a KB and I have done this other times with issues that had no documentation, but nothing happens. I think they are just grateful they can close a long-winded case they probably get harassed daily by their manager to hurry up and close.

 

0 Kudos
the_rock
Legend
Legend

I was hoping there was an SK about it, but I could not find one myself.

It would be nice if this was actually documented in an article.

Best,

Andy

0 Kudos
the_rock
Legend
Legend

Great explanation.

Andy

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events