Connection table Confusion

[Sorin_Gogean]

Hello everyone,

Like I was telling in the other thread Cluster Capacity - peak/concurrent connections we're facing some HUGE connections from time to time, and I'm in the process of identifying them.

So I worked a script, that gets triggered when the connection is HIGH - goes over the 150K that we defined as trigger.

[Expert@XxXx-FW01:0]# fw ctl pstat | grep Concurrent | awk '{print $3}'
97807
[Expert@XxXx-FW01:0]#

Now, when this gets triggered, we collect some reports, like the TOP x Sources HIT and TOP x Destinations HIT and from those TOP X, we're getting some additional reports on the first 3 IP's from each, in order to see what HUGE traffic happens against our appliances.

Now on the connections table, we currently use "fw tab -u -t connections -f" to get all sessions, but one of the problems is that this export and parse takes approx. 3-5 min or more. So I researched and ended up with "fw ctl conntab" that some state is better to use - as is more Human Readable format - and wayyy faster (under 30sec).  

[Expert@XxXx-FW01:0]#time (fw tab -u -t connections -f > fwtab.log ) Using cptfmt Formatting table's data - this might take a while... real 3m7.821s user 2m22.628s sys 0m7.075s [Expert@XxXx-FW01:0]# [Expert@XxXx-FW01:0]# wc -l fwtab.log 372903 fwtab.log [Expert@XxXx-FW01:0]#

[Expert@XxXx-FW01:0]# time ( fw ctl conntab > fwconntab.log ) real 0m2.351s user 0m1.129s sys 0m0.426s [Expert@XxXx-FW01:0]# [Expert@XxXx-FW01:0]# wc -l fwconntab.log 189586 fwconntab.log [Expert@XxXx-FW01:0]#

What is unclear for me right now, are those two "fw tab -u -t connections -f" and "fw ctl conntab" showing same connections or there could be differences ?

Which one is better to use and report connections at a certain point and work with?

Also can someone clarify it contains the "fw ctl multik gconn", as if I address this table, I get other value for number of connections 🤔 .

[Expert@XxXx-FW01:0]# fw ctl multik gconn -s
Summary:
Total number of global connections: 213982

[Expert@XxXx-FW01:0]#

[Expert@XxXx-FW01:0]# time ( fw ctl multik gconn > gconn.log )

real 0m1.377s
user 0m0.851s
sys 0m0.211s
[Expert@XxXx-FW01:0]# wc -l gconn.log
209247 gconn.log
[Expert@XxXx-FW01:0]#

Thank you,

PS: after I finish the script and have a final version, I'll share it....

[PhoneBoy]

fw tab -t connections -u and fw ctl conntab should show the same information about active connections since they're ultimately coming from the same location (kernel tables).
Not sure when fw ctl conntab was added, but fw tab (and friends) have been around since the earliest days of the product.

My understanding is that fw ctl multik gconn provides more details about existing connections (namely what cores they are being distributed onto).
As such, I'd expect it to include everything that's in fw ctl conntab. 

[Sorin_Gogean]

thank you @PhoneBoy ,

still I'm not so convinced, as I see some discrepancies in the number of lines because the "fw tab -u -t connections" returns double the amount of lines/connections if we compare with "fw ctl conntab" . Could it be because of the "accounting" being enabled on some rules? 

As you can observe, the reported Concurrent connections were 99K and we got from different commands, values like 373K (almost 4 times) or 190K (almost 2 times) or 209K (a bit more than 2 times) .

[Expert@XxXx-FW01:0]# fw ctl pstat | grep Concurrent | awk '{print $3}'
97807
[Expert@XxXx-FW01:0]#

Collecting "fw tab -u -t connections -f > fwtab.log" we get 373K lines in the file.....	Collecting "fw ctl conntab > fwconntab.log" we get 190K lines in the file.....
[Expert@XxXx-FW01:0]# wc -l fwtab.log 372903 fwtab.log [Expert@XxXx-FW01:0]#	[Expert@XxXx-FW01:0]# wc -l fwconntab.log 189586 fwconntab.log [Expert@XxXx-FW01:0]#

thank you,

[Timothy_Hall · Accepted solution]

The difference in counts is due to how connections are reported and to some degree output formatting.

fw tab -u -t connections -f doesn't really track connections as a singular entity; it tracks flows of packets.  For non-NATted connections there will be two lines shown for every "connection", a c2s (client to server) flow from the original initiator to the responder, and a s2c (server to client) return flow.  For connections that are NATted, there will be 4 total flow lines in the output: c2s pre-NAT, c2s post-NAT, s2c pre-NAT, and s2c post-NAT.  This is why the fw tab command shows 4X the true number of connections displayed by fw ctl pstat.

fw ctl conntab shows each connection and all its associated flows on just one line, however it does place a newline after each reported connection.  So if you are doing a straight line count this command will show 2X the amount reported by fw ctl pstat.

All of the above counts are tracked on the Firewall Worker/Instance cores.

On the other hand fw ctl multik gconn is showing what the Dynamic Dispatcher is doing on the SND cores, namely keeping track of which connections and their associated flows are assigned to which Firewall Worker/Instance.  It operates similarly to the fw tab command and will show 4 lines/flows for each NATted connection.  I believe the slightly elevated count for fw ctl multik gconn  over fw tab is due to the fact that a connection's worker core assignment will stick around for a bit in the Global Connections table, even after the connection has been closed and removed from the table on the Firewall Workers.  I assume this helps ensure that if a wayward late packet for what appears to be a closed connection happens to show up, it will still be sent to the original worker core who may "resurrect" it via something like this: sk24960: "Smart Connection Reuse" feature modifies some SYN packets.

[Sorin_Gogean]

Thank you @Timothy_Hall , it's exactly what I was looking for.

I was pretty convinced that "fw tab -u -t connections -f " would hold more information/lines and one reason I was considering, was Accounting, I didn't knew about NAT and others.

As for the "fw ctl conntab" - that is true, and therefore in my script I'm dropping the new-lines, so I will address this for future reportings.

" fw ctl multik gconn" was a newly found command, that would show connections, and I didn't researched further.

So I'll go and change from "fw tab -u -t connections -f " to "fw ctl conntab" and see what are the results.

Thank you,

[Daniel_Kavan]

Does rebooting a clusterXL member equate to also clearing its connection table?   Or is the connections table just replicated on each member?   If it is replicated, does clearing the connections table on one member clear it for all members?

[PhoneBoy]

Connections table is replicated on each member.
I do not know what will happen on the other member if you clear the connection table on one member.