topic Cluster Full sync taking very long time R80.40 T161 in Firewall and Security Management

Cluster Full sync taking very long time R80.40 T161

Kaspars_Zibarts — Wed, 26 Oct 2022 06:27:47 GMT

Just wondering if anyone else has any thoughts on the subject..

We have a cluster of 28000 series running R80.40 T161 with IPS, APCL, URLF, AB, AV and HTTPS interception turned ON.

Yesterday we were forced to reboot standby member during day and observed that full sync took nearly half an hour which seemed quite excessive

Oct 25 09:55:42 2022 fw1 fwk: CLUS-120120-1: Fullsync started
Oct 25 10:20:21 2022 fw1 fwk: CLUS-120122-1: Fullsync completed successfully

Performance figures at that point:

total throughput ~15Gbps
internet ~4Gbps
HTTPS inspected ~2Gbps
Threat prevention applied to external traffic only
600,000 concurrent connections
10,000 new connections per second

It seemed that sync protocol was not able to keep up with new connection rate - we just saw from connections table size on the standby that it was growing very very slowly. An no obvious errors reported from cphaprob syncstat

It's a fairly new cluster and we are still in the "tuning" phase (new boxes and new functionality). So we disabled sync for DNS connections and delayed HTTP/S connection sync to 30secs. Which should help of course.

I just wanted to hear if anyone else is pushing high end appliances close to these numbers and have seen anything like that?

Has anyone noticed "performance" improvements after upgrading to R81.10 on gateways? I know management gets "faster" but gateways?

I realize that we are getting close to box MAX:

Re: Cluster Full sync taking very long time R80.40 T161

_Val_ — Wed, 26 Oct 2022 09:51:33 GMT

600K connections is A LOT. I would look into an option to set up delayed sync for at least some of the trafffic.

Re: Cluster Full sync taking very long time R80.40 T161

Kaspars_Zibarts — Wed, 26 Oct 2022 11:10:12 GMT

If it was a FW blade only, it would not be that much. Especially when you look at the datasheet of 28000 🙂

Re: Cluster Full sync taking very long time R80.40 T161

_Val_ — Wed, 26 Oct 2022 18:58:26 GMT

Full sync sends over all kernel tables for 600K connections. It is quite a chunk of data.

Re: Cluster Full sync taking very long time R80.40 T161

the_rock — Wed, 26 Oct 2022 19:47:07 GMT

I agree, thats way too much time. Personally, I would open TAC case to investigate more.

Re: Cluster Full sync taking very long time R80.40 T161

Alexander_Wilke — Fri, 16 Dec 2022 14:44:50 GMT

~400.000 concuirrent connections,

~6.000 new conns per sec

162000 appliance

r80.40 take 156

only Firewall Blade

Nov 2 09:51:34 2022 xxxxx fwk: CLUS-120120-1: Fullsync started
Nov 2 09:52:04 2022 xxxxx fwk: CLUS-120122-1: Fullsync completed successfully

You have many blades and perhaps much more to sync than a firewall only GW.
however it should not take so long.

check MTU size on both sync interfaces to match.

open a ticket.

Re: Cluster Full sync taking very long time R80.40 T161

Timothy_Hall — Fri, 16 Dec 2022 20:57:01 GMT

Sounds like an unhealthy or overloaded sync network, for both members can you post the output of cphaprob syncstat, along with fw ctl pstat in case the firewalls are experiencing other memory issues.

Re: Cluster Full sync taking very long time R80.40 T161

Kaspars_Zibarts — Fri, 16 Dec 2022 15:44:13 GMT

Sorry, Elvis has left the building.. I'm not longer with the company and can't get any logs. But I'm 101% sure that sync network was intact. It's a black fiber between DCs approx 1km apart running mearly 100Mbps from 1Gbps available from memory

Re: Cluster Full sync taking very long time R80.40 T161

the_rock — Fri, 16 Dec 2022 16:32:24 GMT

But come on, now that you work for CP, thats more pressure to fix the issue ; - )

Re: Cluster Full sync taking very long time R80.40 T161

Kaspars_Zibarts — Mon, 19 Dec 2022 11:07:18 GMT

it's fixed in T1543 😄

Re: Cluster Full sync taking very long time R80.40 T161

the_rock — Mon, 19 Dec 2022 11:08:40 GMT

🤣🤣🤣