Certificate defaultCert cannot be validated; Reason: Could not retrieve CRL

[Kevin_Vargo]

Hi-

We are seeing this alert every two hours on an open server HA cluster (77.30).  Out mgmt server is 80.10 (VM) and other clusters in the environment are all 80.10, also open servers.  I've confirmed the cluster throwing the alert can reach the mgmt server.  We use this cluster, in part, to collect logs from two Windows Identity Collector servers and this seems to have started at the same time those were stood up.  Based on the alert log detail we enabled the VPN blade (it was not enabled previously), renewed the certificate, and disabled the VPN blade (pushing policy along the way).  This cluster has never been used as part of VPN proper.  Curious to know if folks have thoughts on a potential cause or what I may be able to collect to further investigate?  Thank you.  

emailed error message

HeaderDateHour: 13Jan2019 15:31:14; ContentVersion: 1; HighLevelLogKey: N/A; Uuid: {0x0,0x0,0x0,0x0}; SequenceNum: N/A; Action: keyinst; Origin: Firewall6; IfDir: >; InterfaceName: daemon; Alert: useralert; OriginSicName: N/A; OriginSicName: ; HighLevelLogKey: 18446744073709551615; scheme:: NA; Validation log:: Certificate defaultCert cannot be validated.; Reason:: Could not retrieve CRL.; Serial num:: ; DN:: CN=EProdCluster VPN Certificate,O=mgmt101.domainname.com.hppeee ; Instruction:: If this log persists, contact the CA administrator.; fw_subproduct: VPN-1; vpn_feature_name: IKE; ProductName: VPN-1 & FireWall-1; ProductFamily: Network; HeaderDateHour: 13Jan2019 15:31:17; ContentVersion: 1; HighLevelLogKey: N/A; Uuid: {0x0,0x0,0x0,0x0}; SequenceNum: N/A; Action: keyinst; Origin: Firewall7; IfDir: >; InterfaceName: daemon; Alert: useralert; OriginSicName: N/A; OriginSicName: ; HighLevelLogKey: 18446744073709551615; scheme:: NA; Validation log:: Certificate defaultCert cannot be validated.; Reason:: Could not retrieve CRL.; Serial num:: ; DN:: CN=EProdCluster VPN Certificate,O=mgmt101.domainname.com.hppeee ; Instruction:: If this log persists, contact the CA administrator.; fw_subproduct: VPN-1; vpn_feature_name: IKE; ProductName: VPN-1 & FireWall-1; ProductFamily: Network;

[PhoneBoy]

Specifically, how did you confirm reachability?

I believe CRL retrieval happens on TCP port 18264, which would be the port to confirm the gateways can reach on the management server.

[Kevin_Vargo]

The mgmt server sits behind this firewall cluster.  The mgmt server actually has an IP address within the scope of the cluster VIP, which is a /24. 

So basically the cluster VIP is 10.10.10.100/24, gateway A is .101 and gateway B is .102.  The mgmt server itself (the log server) is 10.10.10.20.  Thank you.

[Hugo_vd_Kooij]

NAT can be an issue.

How the NAT is done on traffic to the SmartCenter may depend on wether or not you have the optíon "apply to Control connections" enabled.

Check what the gateway thinks what should be the server to get the CRL. A tcpdump with "port 18264" as filter should tell you.

Then follow the breadcrumbs and see if they actually lead to the SmartCenter.

[Kevin_Vargo]

Thanks.  So what I've learned is that years ago we changed our mgmt VIP to an external IP address due to VPN issues.  I essentially cannot route to that external VIP address so we are going to change it back to an internal IP of that mgmt cluster.  This should not impact VPN as I am leaving that link/config alone. 

Regardless, are you saying I can configure NAT just for mgmt traffic to avoid this (but I'll probably change the VIP anyway).  thanks for the info!

[Peter_Lyndley]

Thats correct Damion - its TCP/18264 for CRL retrieval. If there are any firewalls between the gateway that cannot retrieve and the management server then worth checking the logs there.

Try a telnet from the gateway to the management server on port 18264 - does it connect ?

[Kevin_Vargo]

Hi, thank you.  Please see the reply to Dameon above as well.  Below is the output of my connection.  First is the telnet attempt and the second is the output from another putty session on the same gateway so that I could capture the traffic when I attempted the telnet connection.  Also added a traceroute, third.

[Expert@Firewall6:0]# telnet 10.10.10.20 18264
Trying 10.10.10.20...
Connected to 10.10.10.20.
Escape character is '^]'.

[Expert@Firewall6:0]# fw monitor -e "accept port(18264) and host(10.10.10.20) ;"
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_0] s0p4:o[60]: 10.10.10.101 -> 10.10.10.20 (TCP) len=60 id=29859
TCP: 45230 -> 18264 .S.... seq=856b7df8 ack=00000000
[vs_0][fw_0] s0p4:O[60]: 10.10.10.101 -> 10.10.10.20 (TCP) len=60 id=29859
TCP: 45230 -> 18264 .S.... seq=856b7df8 ack=00000000
[vs_0][fw_0] s0p4:i[60]: 10.10.10.20 -> 10.10.10.101 (TCP) len=60 id=0
TCP: 18264 -> 45230 .S..A. seq=e7a98a7c ack=856b7df9
[vs_0][fw_0] s0p4:I[60]: 10.10.10.20 -> 10.10.10.101 (TCP) len=60 id=0
TCP: 18264 -> 45230 .S..A. seq=e7a98a7c ack=856b7df9
[vs_0][fw_0] s0p4:o[52]: 10.10.10.101 -> 10.10.10.20 (TCP) len=52 id=29860
TCP: 45230 -> 18264 ....A. seq=856b7df9 ack=e7a98a7d
[vs_0][fw_0] s0p4:O[52]: 10.10.10.101 -> 10.10.10.20 (TCP) len=52 id=29860
TCP: 45230 -> 18264 ....A. seq=856b7df9 ack=e7a98a7d

[Expert@Firewall6:0]# traceroute 10.10.10.20
traceroute to 10.10.10.20 (10.10.10.20), 30 hops max, 40 byte packets
1 mgmt101.domainname.com (10.10.10.20) 0.735 ms 0.697 ms 0.698 ms
[Expert@Firewall6:0]#

[PhoneBoy]

I recommend getting the TAC involved to further troubleshoot this.

[Kevin_Vargo]

I should add I've already tried SK131112 as well (clearing cache)