topic Re: Dynamic dispatcher issue with R80.30 in Firewall and Security Management

topic Re: Dynamic dispatcher issue with R80.30 in Firewall and Security Management https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91156#M10855 <P>If using kernel mode firewall, the command <STRONG>fw ctl multik print_heavy_conn</STRONG> will show all detected elephant flows (aka "heavy" connections) for the last 24 hours.</P> <P>If using USFW or VSX, you'll need to use the <STRONG>CPMonitor</STRONG> and <STRONG>connstat</STRONG> tools. </P> <P>See here for further reading: <A class="cp_link sc_ellipsis" href="https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk164215&partition=Advanced&product=CoreXL," target="_blank" rel="noopener">sk164215: How to Detect and Handle Heavy Connections</A></P> Fri, 10 Jul 2020 15:20:02 GMT Timothy_Hall 2020-07-10T15:20:02Z Dynamic dispatcher issue with R80.30 https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91012#M10852 <P>Hi again,</P><P> </P><P>I already posted a related question here: <A href="https://community.checkpoint.com/t5/Next-Generation-Firewall/Investigating-CPU-core-consumption-in-R80-30-kernel-3-10-UMFW/m-p/90746#M2203" target="_blank">https://community.checkpoint.com/t5/Next-Generation-Firewall/Investigating-CPU-core-consumption-in-R80-30-kernel-3-10-UMFW/m-p/90746#M2203</A></P><P> </P><P>Did some analyses and now realized that CoreXL connection distribution seems to not work properly anymore:</P><P> </P><P>[Expert@FW:0]# fw ctl multik stat</P><P>ID | Active  | CPU    | Connections | Peak</P><P>----------------------------------------------</P><P> 0 | Yes     | 15     |      <FONT color="#FF9900"> 78152</FONT> |    89916</P><P> 1 | Yes     | 7      |      <FONT color="#FF9900"> 64497</FONT> |    68297</P><P> 2 | Yes     | 14     |      <FONT color="#FF9900"> 57302</FONT> |    64495</P><P> 3 | Yes     | 6      |      <FONT color="#FF9900"> 36117</FONT> |    50217</P><P> 4 | Yes     | 13     |       13611 |    35532</P><P> 5 | Yes     | 5      |        3130 |    29384</P><P> 6 | Yes     | 12     |         767 |    26404</P><P> 7 | Yes     | 4      |         548 |    25565</P><P> 8 | Yes     | 11     |         618 |    25357</P><P> 9 | Yes     | 3      |         420 |    25236</P><P>10 | Yes     | 10     |         503 |    27162</P><P> </P><P>These are all CoreXL CPUs carrying a very different number of connections.</P><P> </P><P>[Expert@FW:0]# fw ctl multik dynamic_dispatching get_mode</P><P>Current mode is On</P><P> </P><P>[Expert@FW:0]# fw ctl affinity -l -r</P><P>CPU 0:</P><P>CPU 1:</P><P>CPU 2:</P><P>CPU 3:  fw_9</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 4:  fw_7</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 5:  fw_5</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 6:  fw_3</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 7:  fw_1</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 8:</P><P>CPU 9:</P><P>CPU 10: fw_10</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 11: fw_8</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 12: fw_6</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 13: fw_4</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 14: fw_2</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P>CPU 15: fw_0</P><P>        mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P><P> </P><P>Any ideas ?</P><P> </P><P>Regards Thomas</P><P> </P><P> </P> Thu, 09 Jul 2020 11:26:26 GMT https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91012#M10852 TomShanti 2020-07-09T11:26:26Z Re: Dynamic dispatcher issue with R80.30 https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91018#M10853 <P>Hi <a href="https://community.checkpoint.com/t5/user/viewprofilepage/user-id/34522">@TomShanti</a> </P> <P>Rather than statically assigning new connections to a CoreXL FW instance based on packet's IP addresses and IP protocol (static hash function), the new dynamic assignment mechanism is based on the utilization of CPU cores, on which the CoreXL FW instances are running.</P> <P>The dynamic decision is made for first packets of connections, by assigning each of the CoreXL FW instances a rank, and selecting the CoreXL FW instance with the lowest rank.</P> <P><STRONG>The</STRONG> <STRONG>rank for each CoreXL FW instance is calculated according to its CPU utilization</STRONG> (<STRONG>only for first packet)</STRONG>.</P> <P>The higher the CPU utilization, the higher the CoreXL FW instance's rank is, hence this CoreXL FW instance is less likely to be selected by the CoreXL SND.</P> <P>The CoreXL Dynamic Dispatcher allows for better load distribution and helps mitigate connectivity issues during traffic "peaks", as connections opened at a high rate that would have been assigned to the same CoreXL FW instance by a static decision, will now be distributed to several CoreXL FW instances.</P> <P>There are the following points which influence an asymmetrical distribution:</P> <P>- Elephant flows with high CPU utilization per CPU core<BR />- Other FW processes that increase the CPU usage of a core. <BR />   In your example these processes:<BR />     mpdaemon fwd pdpd lpd pepd dtpsd in.acapd dtlsd in.asessiond rtmd vpnd cprid cpd</P> <P>I have the suspicion that these are the following IA processes: <BR />pepd, pepd</P> <P> </P> Thu, 09 Jul 2020 12:36:38 GMT https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91018#M10853 HeikoAnkenbrand 2020-07-09T12:36:38Z Re: Dynamic dispatcher issue with R80.30 https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91108#M10854 <P>We have the same problem with the asymmetric distribution.<BR />How can I recognize elephant flows?</P> Fri, 10 Jul 2020 08:41:30 GMT https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91108#M10854 C-3PO 2020-07-10T08:41:30Z Re: Dynamic dispatcher issue with R80.30 https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91156#M10855 <P>If using kernel mode firewall, the command <STRONG>fw ctl multik print_heavy_conn</STRONG> will show all detected elephant flows (aka "heavy" connections) for the last 24 hours.</P> <P>If using USFW or VSX, you'll need to use the <STRONG>CPMonitor</STRONG> and <STRONG>connstat</STRONG> tools. </P> <P>See here for further reading: <A class="cp_link sc_ellipsis" href="https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk164215&partition=Advanced&product=CoreXL," target="_blank" rel="noopener">sk164215: How to Detect and Handle Heavy Connections</A></P> Fri, 10 Jul 2020 15:20:02 GMT https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91156#M10855 Timothy_Hall 2020-07-10T15:20:02Z Re: Dynamic dispatcher issue with R80.30 https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91298#M10856 <P>Interestingly the phenomena vanished when we switched cluster member.</P><P><BR />Regards Thomas</P><P> </P><P> </P> Mon, 13 Jul 2020 08:58:27 GMT https://community.checkpoint.com/t5/Firewall-and-Security-Management/Dynamic-dispatcher-issue-with-R80-30/m-p/91298#M10856 TomShanti 2020-07-13T08:58:27Z