topic Re: Cluster dropping packets in R80.30 unicast Load sharing Mode in Firewall and Security Management

Cluster dropping packets in R80.30 unicast Load sharing Mode

Sudip_Majee — Sat, 04 Jan 2020 10:03:15 GMT

Cluster is dropping packets in unicast Load sharing Mode after upgrading from R80.10 to R80.30, while in HA mode it is working fine.

below are the output of "fw ctl zdebug + drop"

@;825535;[cpu_8];[SIM-207416775];pkt_handle_stateless_checks: Packet dropped (cluster decision). conn: <xxx.xxx.xxx.xxx,46667,xxx.xxx.xxx.xxx,443,6>;

@;825535;[cpu_8];[SIM-207416775];do_packet_finish: SIMPKT_IN_DROP vsid=0, conn:<xxx.xxx.xxx.xxx,46667,xxx.xxx.xxx.xxx,443,6>;

@;825535;[cpu_0];[SIM-207416775];sim_pkt_send_drop_notification: (0,0) received drop, reason: cluster error, conn: <1xxx.xxx.xxx.xxx,46667,xxx.xxx.xxx.xxx,443,6>;

@;825535;[cpu_0];[SIM-207416775];sim_pkt_send_drop_notification: no track is needed for this drop - not sending a notificaion, conn: <xxx.xxx.xxx.xxx,46667,xxx.xxx.xxx.xxx,443,6>;

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

PhoneBoy — Sun, 05 Jan 2020 07:14:59 GMT

Probably best to open a TAC case on this so we can have a closer look.

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

_Val_ — Sun, 05 Jan 2020 16:11:03 GMT

To enable LS mode with R80.20 and R80.30, use the solution mentioned here: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk162637

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Sudip_Majee — Sun, 05 Jan 2020 16:22:41 GMT

yes, already followed the SK162637 to enable load sharing on R80.30, but post that we are facing issue in load sharing while HA mode is working fine.

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

_Val_ — Mon, 06 Jan 2020 08:23:55 GMT

Please open a case with TAC

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

_Val_ — Mon, 06 Jan 2020 08:25:06 GMT

One more note. Did you make sure to get the required HFA, as the case states? Just checking.

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Sudip_Majee — Tue, 07 Jan 2020 09:31:57 GMT

Yes, Take 111 is installed on GW.

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Yair_Shahar — Tue, 07 Jan 2020 11:47:16 GMT

Hi,

Those drops are known and are harmless, originated from debug being enabled while using +

these debug messages show cluster decision on load sharing while one member handles the connection and other drops it.

you can use "fw ctl zdebug drop" to avoid these debug messages and see FW drops only.

Yair

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Sudip_Majee — Thu, 30 Jan 2020 07:58:10 GMT

it drops all the connection on all GW and unable to communicate through firewall.

The issue resolved post installation of Custom hotfix provided by TAC.

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Yair_Shahar — Thu, 30 Jan 2020 09:53:09 GMT

Hi Sudip,

Can you share SR number so we can follow up on the issue observed by TAC?

Yair

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Pawel_Szetela — Mon, 10 Feb 2020 10:20:33 GMT

Hi,

We observed the same behaviour: 3 gateways, clean install, JH111 applied, ClusterXL Load Sharing Unicast, open server.

Is there any chance to integrate fix with JH in near future? Or we have to open service request?

Regards,

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Yair_Shahar — Mon, 10 Feb 2020 11:01:36 GMT

Hi,

I understand you observe real drops and not just drop messages as suggested above.

Did you follow sk162637?

specifically this note:

Starting from R80.20, the ccl_force_sticky kernel parameter must be set to 1 (in fwkern.conf) on all cluster members.

Yair

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Pawel_Szetela — Mon, 10 Feb 2020 11:32:43 GMT

Hi,

Yes, we've followed instructions in sk162637 and we observe not only drop messages mentioned above but real drops in traffic as well.

Regards

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Yair_Shahar — Mon, 10 Feb 2020 11:39:18 GMT

Thanks,

I'm not aware of any known drops in this configuration, if you can add more details on what kind of drops you observe, it can help pinpointing the root cause here.

Anyway, I would suggest to open Service Request to for proper handling and fixing if needed.

Yair

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Sudip_Majee — Mon, 10 Feb 2020 11:43:32 GMT

Yes.

you can disable vMAC and run the below command to make it working and later ask TAC to provide custom hotfix to resolve the issue.

fw ctl set int fwha_vmac_global_param_enabled 0

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Pawel_Szetela — Mon, 10 Feb 2020 11:53:52 GMT

Hi,

To be more precise: no traffic is going through the cluster, we can only connect (ssh) to cluster members from directly connected networks.

Yesterday we did some labs (in vmware) and we observed the same problem - no traffic in clusterxl load sharing unicast (vmac), only switching to HA mode resolves problem.

We'll check sugestion from Sudip_Majee (below) to disable vmac (probably next week in maintenance window).

Regards

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Pawel_Szetela — Mon, 10 Feb 2020 11:56:20 GMT

Hi,

Thanks a lot for Your advice. We'll try this next week.

Regards,

PS. Anyway, I don't like custom hotfixes because of future compatibility problems with JH accumulators.

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Yair_Shahar — Mon, 10 Feb 2020 12:04:59 GMT

Can you share the drop log you observe when using fw ctl zdebug drop ?

Yair

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Pawel_Szetela — Tue, 11 Feb 2020 11:18:01 GMT

Hi,

Sorry but we did "fw ctl zdebug + drop" and results were the same like in first post in this thread (we didn't save results). We reverted back to R80.10.

We're considering next try for next week.

Regards

Re: Cluster dropping packets in R80.30 unicast Load sharing Mode

Yair_Shahar — Tue, 11 Feb 2020 14:46:26 GMT

Hi,

We did additional lookup in regards to these drops.

Following drop message is as mentioned above and are harmless debug messages

@;825535;[cpu_8];[SIM-207416775];pkt_handle_stateless_checks: Packet dropped (cluster decision). conn: <xxx.xxx.xxx.xxx,46667,xxx.xxx.xxx.xxx,443,6>;

Following drop message is a real drop and is a bug reflected in Cluster Load Sharing mode - this has been addressed in R80.40 and is already planned to be addressed in R80.30 next Jumbo HF (above take 140)

@;825535;[cpu_0];[SIM-207416775];sim_pkt_send_drop_notification: (0,0) received drop, reason: cluster error, conn: <1xxx.xxx.xxx.xxx,46667,xxx.xxx.xxx.xxx,443,6>;

I hope this clear things out.

Yair