Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
af0e2c12-4d24-3
Participant

HTTPS Inspection Limiting TCP receive window to 262144 bytes and limiting throughput of tcp stream

Hello Checkmates,

Am I the only one that has noticed that with HTTPS-inspected traffic, Checkpoint gateways always initiate an outbound tcp connection with TCP receive window scale factor of 8, and that the gateway's receive window never scales beyond 262144 bytes (32,768*8) in either direction (facing the client or facing the internet server)? This limits the throughput of a single tcp connection and is a problem if either bandwidth or latency is high.

For example, bandwidthplace.com test uses a single TCP connection. On our 500 Mbps link, with HTTPS inspection we get around 90 Mbps down. With HTTPS client bypass, we get 350 Mbps down (since windows controlled by client instead of gateway). This is on an otherwise idle gateway with no CPU usage. It's also not a matter of other active blades, it still happens with HTTPS-inspected traffic even if I turn off everything else. Of course, most web browsers use a single tcp connection for web downloads so we also see the same thing on https downloads. It is very clearly a problem of the gateways limiting the tcp receive window.

We have 6500 Gateways on R80.40 take 83. I only noticed since we have a higher bandwidth link now and also some remote users tunneling through high latency paths. I realize on the aggregate of many connections we still get the throughput, but it's odd to artificially limit HTTPS throughput of a single connection and make the gateway the bottleneck. Maybe they have to make some arbitrary choice for scalability of number of TCP connections or something, but it would sure be better to simply follow what the client presents in the TCP Syn packet so the gateway doesn't throttle the throughput.

I have an SR open and we're running this up the tree, but I would be interested to know if anyone else has noticed this limitation or could confirm their gateways do the same thing. I did it by using cppcap to a file then looking at the connection that the gateway makes to the internet server after the client does, then looking at the maximum receive window it gets. In all cases HTTPS inspection makes the connections as described above, and gateway's receive windows never scales past 262144 bytes, whereas the client on its own without inspection would go up to tcp receive windows like 8 MB and thus get much better throughput.

Also, this has nothing to do with the OS settings such as rmem tcp_rmem and so on. If from the gateway you initiate a connection directly (telnet to some port), it honors the rmem settings and in my case puts window scaling factor of 1024 which matches the OS rmem setting. No change in OS tcp settings affects what the HTTPS-inspected connections initiated from the gateway do.

Thanks for any confirmation. I'm just curious if checkpoint is really artificially limiting single-connection TCP throughput for all their customers in similar latency/bandwidth scenarios that use HTTPS inspection and didn't bother to tell us.

0 Kudos
10 Replies
Timothy_Hall
Champion
Champion

This limit would appear to be controlled by the following kernel variables:

cpas_tcp_xq_default_size = 262144
cpas_tcp_xq_max_limit = 1048576

However I'm not exactly sure what "xq" means in this context, so this could be a red herring.  These variables do not seem to be documented anywhere, so I'd advise against changing them without consulting TAC.

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com
_Val_
Admin
Admin

@Timothy_Hall Those parameters are related to active streaming and define the buffer size per stream. There are plenty of reasons why they are not documented. They are internal settings of the engine, and should only be changed in specific and very limited cases. 

@af0e2c12-4d24-3 The issue is, with HTTPSi and in some other scenarios, you terminate connections on your security GW and open new ones, so data stream could be decrypted and extracted as clear text for further analysis. Receive window parameters should not cause you too much issues, unless you are routinely transfer huge files over HTTPS. If that is your case, consult with support. I would also challenge your need to decrypt such transfers, before anything else.

Window size is indeed lower than those maximum you can get with a browser. The reason is, it helps avoiding a situation when your HTTPS streams become "heavy connections" aka "elephant flows", thus draining overall performance of your security GW. 

0 Kudos
af0e2c12-4d24-3
Participant

I appreciate the responses. I will continue with the SR and see where it lands.

I'm really having a hard time believing that after the industry worked hard to give us TCP window scaling beyond 64KB , that decades later a major firewall vendor would arbitrarily limit TCP receive windows for HTTPS traffic to only 256K. Hopefully there is something odd about our installation leading to this behavior and CheckPoint hasn't effectively put in a per-connection throughput limit that varies with latency. Of course customers want a normal TCP situation, let congestion control algorithms do their job during bandwidth contention, and get whatever performance the gateway hardware allows. We wouldn't want some arbitrary variable throughput throttling that no one ever told us about, nor would we want yet another reason we have to put in HTTPSi bypasses, as it's enough of a pain already. 

256K is clearly inadequate for everyday bandwidth/delay situations and it is indeed throttling throughput for users and leading to complaints. For a US west-coast based company, it is effectively saying that US east-coast server HTTPSi connections will be limited to 32 Mbps, Europe 14 Mbps, India 8 Mbps, not to mention that widely varying latencies can be found even geographically close that will affect the throughput. Allow normally-scaled TCP windows and we get many many times the throughput.

0 Kudos
_Val_
Admin
Admin

@af0e2c12-4d24-3 Caleb, if you are 100% the current settings are problematic for you, please follow your support process.

0 Kudos
af0e2c12-4d24-3
Participant

I've had an SR open for a week and investigated other avenues with no meaningful response so far. Frankly this is a bad design and should be embarrassing for Check Point. Firewall vendor throttles throughput without telling customers. Idle gateways with 25% peak CPU and 25% peak memory usage have TCP throughput throttled 75% for no valid reason. Customers shell out money for hardware but don't get the performance.

A better default design would be to dynamically adjust windows within some min/max range depending on gateway resources.

0 Kudos
_Val_
Admin
Admin

Please send me SR number via PM. Thanks.

0 Kudos
_Val_
Admin
Admin

@af0e2c12-4d24-3 Message received and answered. This case is actually documented in sk98871, but without any further details. I have asked R&D to comment. It may take a bit of time, though.

af0e2c12-4d24-3
Participant

Thanks. I did see sk98871 in my research but I found that even if I turned off all threat prevention blades, the results were still the same. Of course it's not just on speedtest sites but on any HTTPS traffic. In all cases the data I'm seeing matches the throughput math based upon the latency to target server with a 256K receive window.

Anyway, we'll see what R&D says.

0 Kudos
af0e2c12-4d24-3
Participant

After all this time, R&D said I could change cpas_tcp_xq_default_size to 1 MB.

Timothy_Hall
Champion
Champion

Interesting, thanks for the follow-up.

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com
0 Kudos