topic Re: SIC issue in Firewall and Security Management

SIC issue

GUEYDON_Olivier — Thu, 09 Aug 2018 08:05:40 GMT

Hi all,

I have a cluster of 2 5000 Appliances, running R80.10.

My trooble is one of the member, the standby, has lost SIC with the SMS. The Active is running well, but i can't push new policies.

I tried 30 times to reset SIC between the standby and the SMS, but always got error (300, 148).

So about 30 revoked certs on the SMS ...

My question is : must i reset both gateways SIC (even the active) ?

If so, as i can't push policies, what would happen for the active GW ?

Thanks a lot for your help,

Re: SIC issue

Anthony_Joubai1 — Thu, 09 Aug 2018 08:11:09 GMT

Hello,

First question would be to determine why a standby member is not able to maintain SIC.

there is no basically no need to reset it.

Please elaborate architecture first: FULL HA or Distributed ?

Once SIC is establish, you can push once and then you are blocked?
New installation ?

regards,

Anthony

Re: SIC issue

GUEYDON_Olivier — Thu, 09 Aug 2018 08:43:39 GMT

Hello Anthony (french?)

The cluster was working well since February. There is no Nat (Lan firewalling), and is in Active/Passive mode with HA.

Both GW are on the same vlan but not on the same as SMS but as i said, active one works well. I can ping SMS from active and standby, and GW from SMS.

Ports 18191,18192 are Listen on the standby GW.

I started having problem when i pushed a modified policy. I ran into an error of services port conflict (Uncheck match for any checkbox ...).

Thanks,

Re: SIC issue

Anthony_Joubai1 — Thu, 09 Aug 2018 08:55:04 GMT

Hello,

Could you run the following commands,
fw ctl zdebug drop | grep <SmartcenterIP> on both firewalls. and try to check SIC status on the object.

We may have the answer.

On the cluster objects, which IP are used for cluster member (private/public)

I would be delighted to discuss about this case in the french section

CheckMates en Français

Re: SIC issue

GUEYDON_Olivier — Thu, 09 Aug 2018 09:21:00 GMT

Yes i'd also prefer move to french section. How can we do ?

Let's talk about my IP :

Active GW : 10.30.255.241

Standby GW : 10.30.255.243

SmartCenter : 10.33.1.130

The results of the command :

On the active GW :

Packet proto=6 10.33.1.130:55216 -> 10.30.255.243:18192 dropped by fw_tcp_state_update Reason: Illegal post SYN packet;

On the standby GW : nothing

Re: SIC issue

Anthony_Joubai1 — Thu, 09 Aug 2018 09:35:55 GMT

Solution could be on both member to

On the fly :
# fw ctl set int fwha_forw_packet_to_not_active 1

doesn't survive reboot

If it correct the problem, we will add it to the fwkern.conf.

Re: SIC issue

Jerry — Thu, 09 Aug 2018 09:47:30 GMT

If were in your shoes and have had such circumstance I would have first determine all the aspects of the routing between the CoreXL Cluster and SMS/MDS then troubleshoot the SIC issue but narrowing the protocol flow by fw monitor. It may occure sometimes that one of the HA memebers lost SIC but essentially when it does the HA has no Management capabilities at all as you cannot push policy to one member only can you.

See if the certs can be removed from the none-working member first and then try to re-establish a SIC with the Cluster instead of the none working member (cpconfig on Active member).

if not - seek sk in Support Center - there was something about it ... let me try to locate it myself just now ...

Re: SIC issue

Jerry — Thu, 09 Aug 2018 09:49:18 GMT

sk62873 then sk30579

Re: SIC issue

GUEYDON_Olivier — Thu, 09 Aug 2018 12:12:02 GMT

Ok, here are the latest news:

@Anthony : fwha_forw_packet_to_not_active 1 : didn't solved.

@Jerry : couldn't find a way to remove the GW certificates.

In the ICA WebGUI (Management Tool), are all the revoked certificates for the non-working GW. I think i can delete them (confirm ?). There is also the valid one for the active GW, which is valid.

I found this : Regenerate the Internal CA Without Breaking SIC

Could it be useful ?

Re: SIC issue

Vincent_Bacher — Thu, 09 Aug 2018 12:32:29 GMT

fw debug cpca on TDERROR_ALL_ALL=5

might be interesting as well

Re: SIC issue

Jerry — Thu, 09 Aug 2018 12:55:32 GMT

yes indeed you can (just check the issue date beforehand)

reg. Regenerate - indeed that may come helpful.

you’re the man, best of luck !

Jerry

Re: SIC issue

Kaspars_Zibarts — Fri, 21 Jun 2019 09:07:29 GMT

Judging from logs traffic from mgmt arrives on different interface (not 10.30.255.x) so you can try /32 routing solution provided here in case packets are not forwarded to standby member correctly. It's more to the end of the thread.

https://community.checkpoint.com/message/13561-re-problem-accessing-standby-cluster-member-from-non-local-network?commentID=13561

Re: SIC issue

JozkoMrkvicka — Thu, 09 Aug 2018 18:16:37 GMT

fw unloadlocal on problematic node during issue to see if this is rulebase related ?

Also, did you try "fw ctl zdebug + drop | grep <SMS_IP>" as was suggested ?

Re: SIC issue

GUEYDON_Olivier — Fri, 10 Aug 2018 07:22:04 GMT

Hi guys,

Seems to be endless problem. I've tried all you suggested me, but with no luck.

I think factory-default is the next step. As it's the first time for me, i'm a little bit confused

Will i be able to reattach the GW to the cluster after that ?

Are there some actions to do before factory-default ?

Re: SIC issue

Jerry — Fri, 10 Aug 2018 07:27:05 GMT

don’t do defaulting just re-do full HA SIC again (yes, from scratch) - unless you don’t have console/mgmt access to the ClusterXL HW (VM/APPL).

do the SIC again with the Management but first, make sure you’ve got IP connectivity with the MGMT server and HA memebers can individually reach out to the SMS host/s.

what about the routing table ? did you compare the routing tables from BOTH HA members?

I have had such case where both Members have had DIFFERENCES between themselves. It might be the case of simply connectivity or ... NIC associations?

Re: SIC issue

Anthony_Joubai1 — Fri, 10 Aug 2018 07:34:53 GMT

Hello,

Agree with Jerry, don't Reset the box, as long as the source problem is not identified.

Please contact your support

A quick remote and the problem will be solved without Reset the Box.

regards,

Anthony

Re: SIC issue

Jerry — Fri, 10 Aug 2018 07:42:26 GMT

agree with Anthony, call the Support, unless you’ve got no valid contract (valid support) with the Vendor. If you don’t then ... you may need to digg into the details of the design mate

Re: SIC issue

GUEYDON_Olivier — Fri, 10 Aug 2018 08:36:56 GMT

Jerry, what do you mean by "re-do full HA SIC again (yes, from scratch)" ?

Do you speak about Regenerate the Internal CA Without Breaking SIC ?

It's the only thing i haven't done yet (too scared to loose the working box !)

About connectivity, both GWs can ping the SMS, but they don't ping each other.

SMS can ping both GWs.

Routing table are the same on both GWs

Re: SIC issue

Jerry — Fri, 10 Aug 2018 11:49:30 GMT

in that case yes, I think this is the only way to regenerate a SIC with CA without breaking the communication entirely, however I would be careful though if something bad happen so that I can still securely access Management IPs from the HA members.

Re: SIC issue

GUEYDON_Olivier — Fri, 10 Aug 2018 12:08:18 GMT

Found something strange in the smartcenter logs :

from SMS to non-working GW : DROP/ CDP_amon (18192) / Data received before SYN was acknowledged. Stripping all packet data.