Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
af0e2c12-4d24-3
Participant

HTTPS Inspection Limiting TCP receive window to 262144 bytes and limiting throughput of tcp stream

Hello Checkmates,

Am I the only one that has noticed that with HTTPS-inspected traffic, Checkpoint gateways always initiate an outbound tcp connection with TCP receive window scale factor of 8, and that the gateway's receive window never scales beyond 262144 bytes (32,768*8) in either direction (facing the client or facing the internet server)? This limits the throughput of a single tcp connection and is a problem if either bandwidth or latency is high.

For example, bandwidthplace.com test uses a single TCP connection. On our 500 Mbps link, with HTTPS inspection we get around 90 Mbps down. With HTTPS client bypass, we get 350 Mbps down (since windows controlled by client instead of gateway). This is on an otherwise idle gateway with no CPU usage. It's also not a matter of other active blades, it still happens with HTTPS-inspected traffic even if I turn off everything else. Of course, most web browsers use a single tcp connection for web downloads so we also see the same thing on https downloads. It is very clearly a problem of the gateways limiting the tcp receive window.

We have 6500 Gateways on R80.40 take 83. I only noticed since we have a higher bandwidth link now and also some remote users tunneling through high latency paths. I realize on the aggregate of many connections we still get the throughput, but it's odd to artificially limit HTTPS throughput of a single connection and make the gateway the bottleneck. Maybe they have to make some arbitrary choice for scalability of number of TCP connections or something, but it would sure be better to simply follow what the client presents in the TCP Syn packet so the gateway doesn't throttle the throughput.

I have an SR open and we're running this up the tree, but I would be interested to know if anyone else has noticed this limitation or could confirm their gateways do the same thing. I did it by using cppcap to a file then looking at the connection that the gateway makes to the internet server after the client does, then looking at the maximum receive window it gets. In all cases HTTPS inspection makes the connections as described above, and gateway's receive windows never scales past 262144 bytes, whereas the client on its own without inspection would go up to tcp receive windows like 8 MB and thus get much better throughput.

Also, this has nothing to do with the OS settings such as rmem tcp_rmem and so on. If from the gateway you initiate a connection directly (telnet to some port), it honors the rmem settings and in my case puts window scaling factor of 1024 which matches the OS rmem setting. No change in OS tcp settings affects what the HTTPS-inspected connections initiated from the gateway do.

Thanks for any confirmation. I'm just curious if checkpoint is really artificially limiting single-connection TCP throughput for all their customers in similar latency/bandwidth scenarios that use HTTPS inspection and didn't bother to tell us.

0 Kudos
16 Replies
Timothy_Hall
Champion
Champion

This limit would appear to be controlled by the following kernel variables:

cpas_tcp_xq_default_size = 262144
cpas_tcp_xq_max_limit = 1048576

However I'm not exactly sure what "xq" means in this context, so this could be a red herring.  These variables do not seem to be documented anywhere, so I'd advise against changing them without consulting TAC.

New 2021 IPS/AV/ABOT Immersion Self-Guided Video Series
now available at http://www.maxpowerfirewalls.com
_Val_
Admin
Admin

@Timothy_Hall Those parameters are related to active streaming and define the buffer size per stream. There are plenty of reasons why they are not documented. They are internal settings of the engine, and should only be changed in specific and very limited cases. 

@af0e2c12-4d24-3 The issue is, with HTTPSi and in some other scenarios, you terminate connections on your security GW and open new ones, so data stream could be decrypted and extracted as clear text for further analysis. Receive window parameters should not cause you too much issues, unless you are routinely transfer huge files over HTTPS. If that is your case, consult with support. I would also challenge your need to decrypt such transfers, before anything else.

Window size is indeed lower than those maximum you can get with a browser. The reason is, it helps avoiding a situation when your HTTPS streams become "heavy connections" aka "elephant flows", thus draining overall performance of your security GW. 

0 Kudos
af0e2c12-4d24-3
Participant

I appreciate the responses. I will continue with the SR and see where it lands.

I'm really having a hard time believing that after the industry worked hard to give us TCP window scaling beyond 64KB , that decades later a major firewall vendor would arbitrarily limit TCP receive windows for HTTPS traffic to only 256K. Hopefully there is something odd about our installation leading to this behavior and CheckPoint hasn't effectively put in a per-connection throughput limit that varies with latency. Of course customers want a normal TCP situation, let congestion control algorithms do their job during bandwidth contention, and get whatever performance the gateway hardware allows. We wouldn't want some arbitrary variable throughput throttling that no one ever told us about, nor would we want yet another reason we have to put in HTTPSi bypasses, as it's enough of a pain already. 

256K is clearly inadequate for everyday bandwidth/delay situations and it is indeed throttling throughput for users and leading to complaints. For a US west-coast based company, it is effectively saying that US east-coast server HTTPSi connections will be limited to 32 Mbps, Europe 14 Mbps, India 8 Mbps, not to mention that widely varying latencies can be found even geographically close that will affect the throughput. Allow normally-scaled TCP windows and we get many many times the throughput.

0 Kudos
_Val_
Admin
Admin

@af0e2c12-4d24-3 Caleb, if you are 100% the current settings are problematic for you, please follow your support process.

0 Kudos
af0e2c12-4d24-3
Participant

I've had an SR open for a week and investigated other avenues with no meaningful response so far. Frankly this is a bad design and should be embarrassing for Check Point. Firewall vendor throttles throughput without telling customers. Idle gateways with 25% peak CPU and 25% peak memory usage have TCP throughput throttled 75% for no valid reason. Customers shell out money for hardware but don't get the performance.

A better default design would be to dynamically adjust windows within some min/max range depending on gateway resources.

0 Kudos
_Val_
Admin
Admin

Please send me SR number via PM. Thanks.

0 Kudos
_Val_
Admin
Admin

@af0e2c12-4d24-3 Message received and answered. This case is actually documented in sk98871, but without any further details. I have asked R&D to comment. It may take a bit of time, though.

af0e2c12-4d24-3
Participant

Thanks. I did see sk98871 in my research but I found that even if I turned off all threat prevention blades, the results were still the same. Of course it's not just on speedtest sites but on any HTTPS traffic. In all cases the data I'm seeing matches the throughput math based upon the latency to target server with a 256K receive window.

Anyway, we'll see what R&D says.

0 Kudos
af0e2c12-4d24-3
Participant

After all this time, R&D said I could change cpas_tcp_xq_default_size to 1 MB.

Timothy_Hall
Champion
Champion

Interesting, thanks for the follow-up.

New 2021 IPS/AV/ABOT Immersion Self-Guided Video Series
now available at http://www.maxpowerfirewalls.com
0 Kudos
Thomas_Eichelbu
Advisor

@af0e2c12-4d24-3,
after changing this parameter  "cpas_tcp_xq_default_size" from  262144 to 1048576 did you encounter any significant improvement?

would be good to know 

0 Kudos
Chris_Atkinson
Employee
Employee

Please investigate more recent jumbo hotfix takes that include the following enhancement as a first step.

UPDATE: Check Point Active Streaming (CPAS) TCP Window scale factor is now increased up to 6.

0 Kudos
Thomas_Eichelbu
Advisor

OK i see it .. in R80.40 Take 150.

PRJ-32072,
STRM-737
Security Gateway UPDATE: Check Point Active Streaming (CPAS) TCP Window scale factor is now increased up to 6.


but not yet for R81+ ?

 

0 Kudos
Matt_Ricketts
Employee
Employee

MatanYanay
Employee
Employee

Hi @Thomas_Eichelbu 

For R81 version it will be part of our next ongoing jumbo 

For R81.10 as @Matt_Ricketts mention its already release as part of Take 38 with PRJ-32074

Thanks 

Matan.

 

0 Kudos
kbleb
Explorer

Hi - I believe that while there was a significant improvement from 256K to 1 MB in test cases, it wasn't enough for the specific scenario we were looking at (some VPN users with tunnel-all through headquarters who had high-bandwidth connections, but with high or variable latency).

Seems like the active streaming update could help such scenarios though.

0 Kudos