HTTP POST fragments not re-assembled, only first packet forwarded

[David_Herselman]

Hi,

I've been battling through Check Point support (6-0001758421) and was hoping someone here may be able to provide a work around regarding R80.30 Gaia 2.6 on a 6500 appliance not re-assembling a fragmented HTTP POST request and sending the first segment as the one and only frame.

The following is comparative packet capture between the ingress (bond1.316) and egress (bond1.2020) interfaces. The first packet fragment is selected in both views, frame 28 on the left and 29 on the right. The problem is with frame 29 not being re-assembled with 28 and its data not being included in the packet that Gaia sends out:

The first couple of things to raise suspicion were:

• Data appears to have been aggressively fragmented.

These connections are to data centre monitoring gateways that collect telemetry from probes in each rack and other points of interest. The connections appear to use the ulxmlrpcpp library, which explains the packetisation size.

• URI appears to be malformed

The only plausible answer I have here is that developers at Vigilent may have miss interpreted the XML RPC library and defined their connection string as proxy and port.

I'll get in to those in a moment. Check Point support provided R80.30 on-going hotfix accumulator take 50 which we installed on the standby cluster member and then switched over to. We have tried to ensure that established connections were reset by:

• temporarily defining a 'block suspicious activity' rule in SmartView Monitor (c:\Program Files (x86)\CheckPoint\SmartConsole\R80.30\PROGRAM\SmartViewMonitor.exe) to force connections between the IPs to be removed from the connection tracking tables:

Then loading a reject rule and removing it a minute later:

• Installing policy
• Temporarily defining the service object not to sync state, restarting the standby cluster member, waiting for sync to be current and then transferring role:

These are the layers:

• Network (Firewall blade only):

• Application (Applications & URL Filtering blade only):

Custom TCP service object which does not have protocol defined:

Process logs warn about the categorisation failure but the connection is allowed:

Probably simply due to there being no IP, hostname or FQDN to categorise.

Some things to consider:

• This is monitoring data for environmental control in several data centres in 5 geographically distributed buildings. The telemetry notifies about failures, hot zones and humidity problems and co-ordinates staff and equipment. This is a critical service.
• The Check Point allows the connection, following the desired state.

The problem is easily reproducible, monitoring requests arrive every 8 seconds so I constructed a tcpdump filter to only show those packets where the first four characters of the payload matched 'POST':

tcpdump -nn -i bond1.316 -s 0 -A 'host 192.168.131.33 and host 192.168.204.67 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354'
tcpdump -nn -i bond1.2020 -s 0 -A 'host 192.168.131.33 and host 192.168.204.67 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354'

PS: Wireshark and tcpdump both combine fragmented packets automatically. The result is that people often miss-interprit the displayed output as being packet duplication whereas the packet in memory is simply appended with each fragment and the content subsequently displayed. The section high-lighted in the ingress view (top window) is the content of the second fragment, which now includes the header transmitted in the first fragment.

The packet transmitted (bottom) is very clearly exclusively the content of the first fragment.

* I'll edit this post to provide details around the library used to generate these packets.