File Transfers over port 22 are failing

[David_Chau]

Hello,

We have a number of automated file transfers to different vendors that are failing.  The large files have to be retried and the large number of files(100+) have to be restarted.  Files are transferred via port 22.  Someone has to babysit these file transfer jobs to ensure they get completed.  This is happening on both inbound and outbound file transfers. 

This is affecting every vendor we send files to and any host we send or receive from so this is looking like it's the firewall that might be the bottleneck.  

Any ideas?  What can I check on the gateway or settings I can change to fix this?

I am running R80.30.

Thanks!

[the_rock]

First thing I would check is logs in dashboard on port 22 and see why it might be failing. So say if remote iP is 1.2.3.4, you can do filter like this -> dst:1.2.3.4 OR port:22 OR action:Drop

You can also use AND instead of OR. From ssh, just run something like this -> fw ctl zdebug + drop | grep 1.2.3.4 | grep 22

Andy

[David_Chau]

Hi Andy,

There are no drops in traffic since we have a specific rule to allow this.  Resource utilization on the gateways are also low.

[the_rock]

Ok, fair enough...do we see anything in the logs at all or it shows traffic being accepted?

[David_Chau]

Logs show accepted.  I am thinking about enabling log accounting to gather more details on this traffic.

[the_rock]

That, but you can also try do fw monitor filters, both below (assuming IP is 1.2.3.4)

fw monitor -e "accept host(1.2.3.4) and port(22);"

fw monitor -F '0,1.2.3.4,0,22,0' 

-F flag lets you do in order 'src IP, dst IP, src port, dst port, protocol'

Andy

[Sorin_Gogean]

morning,

anything in SSH logs, that shows why session was dropping ?

could it be that it was taking too much and for a while nothing happened so it was closed ?

you could also try and look to enable some keepalive probing

thank you,

PS: look on port 22 what timeout it has set - mine is 3600 - so the ssh connection would be kept for up to 1 hour before it would be dropped from the CKP FWL if not traffic happens

[Timothy_Hall]

Enable Accounting on the rule(s) matching the transfers and also try enabling TCP state logging to get more information about what is going on when the connections terminate: sk101221: TCP state logging

[abihsot__]

Try using plain tcp/22 without protocol selected. We had nasty connection drops, but with other ports though.

[David_Chau]

Found the issue.  It was the IPS Protection, "Malicious Payload Encoding Remote Code Execution", that are intermittently blocking these transfers.  

https://threatpoint.checkpoint.com/ThreatPortal/threat?threatType=protection&threatId=PAYLOAD_ENCODE...

Any reason why this IPS protection signature would block the file then allow the file to transfer when reattempted?

[Sorin_Gogean]

That's weird, as it should be quite visible in the FWL logs while looking for SSH traffic from A to B.

Good that you found what was impacting your traffic that randomly.

[the_rock]

Can you see any logs on that IPS protection?

[David_Chau]

Yes, I see the block in the logs and then the traffic is allowed after the file transfer is restarted.

[the_rock]

If I were you, I would add an exception for that, to be on the safe side.

[David_Chau]

Just added the exception.  The business is not very happy this was the issue.  Curious why this block was so intermittent.  These are daily transfers containing the same type of data.  These transfers are also encrypted so not sure how this signature flagged this as malicious payload, maybe because of the high volume?  Anybody from Check Point able to provide additional details on this IPS protection?

[Tobias_Moritz]

We also see exact these protection hitting on SSH traffic flowing over our gateways from time to time. Could never reproduce it, it just happens from time to time. So I'm also interested about the some background info regarding that protection.

[Timothy_Hall]

The Confidence rating of that "Malicious Payload Encoding Remote Code Execution" protection is Medium, if it were Low I'd be more willing to accept it going off randomly like that.  My guess is that this signature is looking for a very short sequence of bytes (<5) without any further context information, and every now and then an encrypted stream happens to match it and get blocked.  Seems unlikely but certainly not impossible.  Would need to see packet captures of a few separate instances to try to determine what it is seeing that is causing the false positive.

This vulnerability does not appear to have a CVE number and is something Check Point came up with on their own (CPAI-2013-3606) so only R&D will be able to provide further details.  Based on how it is behaving and that multiple posters have seen this behavior, I'd argue that the Confidence level of this protection should be changed to Low.  Who knows if this attack method is even relevant anymore; this is an advisory from almost 10 years ago.