UPDATE: I have been able to successfully recreate this issue using a lab client to the live service and compare r80.20 to r77.30 behaviour.
As we still have our work-a-round in place to protect our client wher the web server MTU is set at 1400, the numbers are reduced but the effect is the same.
The Lab is an Ubuntu client with an MTU of 1500 and a Cisco router with the LAN interface MTU set at 1360 (this is significant as it is this interface that doesn't want to transmit anything larger than 1360).
The Web server MTU is set at 1400 and is behind a CheckPoint r80.20 15600 Cluster.
During tests, the packet anaysis shows the SYN and SYN-ACK negotiate their preferred Maximum Segment Size (MSS) and that when the Cisco router receives a packets larger than it likes (due to the MTU set at 1360) , the packets are dropped and ICMP TYPE 3 CODE 4 messages are created and transmitted. CheckPoint r80.20 appears to ignore these ICMP messages and packets are re-transmitted until the HTTPs connection times out.
As we also have another CheckPoint 15600 cluster still running r77.30, the same lab test was run against a live web service NAT'd behind that cluster (the MTU of the live web server is 1500 here but the effect is the same).
In this test, the packet anaysis shows the SYN and SYN-ACK again negotiate their preferred Maximum Segment Size (MSS) and that when the Cisco router receives a packet larger than it likes (due to the MTU set at 1360) , the packet is dropped and an ICMP TYPE 3 CODE 4 message is created and transmitted. The difference being is that, from that point on, packets sent are lower than the Cisco's 1360 MTU and the HTTPs transaction completes successfully.
These results have being passed to CheckPoint support and we are scheduling further debug and testing with CheckPoint earl next week in an out of hours window.
Will keep this thread updated.