topic Re: File Transfers over port 22 are failing in Firewall and Security Management

File Transfers over port 22 are failing

David_Chau — Thu, 14 Jul 2022 18:48:14 GMT

Hello,

We have a number of automated file transfers to different vendors that are failing. The large files have to be retried and the large number of files(100+) have to be restarted. Files are transferred via port 22. Someone has to babysit these file transfer jobs to ensure they get completed. This is happening on both inbound and outbound file transfers.

This is affecting every vendor we send files to and any host we send or receive from so this is looking like it's the firewall that might be the bottleneck.

Any ideas? What can I check on the gateway or settings I can change to fix this?

I am running R80.30.

Thanks!

Re: File Transfers over port 22 are failing

the_rock — Thu, 14 Jul 2022 19:50:30 GMT

First thing I would check is logs in dashboard on port 22 and see why it might be failing. So say if remote iP is 1.2.3.4, you can do filter like this -> dst:1.2.3.4 OR port:22 OR action:Drop

You can also use AND instead of OR. From ssh, just run something like this -> fw ctl zdebug + drop | grep 1.2.3.4 | grep 22

Andy

Re: File Transfers over port 22 are failing

David_Chau — Thu, 14 Jul 2022 19:59:40 GMT

Hi Andy,

There are no drops in traffic since we have a specific rule to allow this. Resource utilization on the gateways are also low.

Re: File Transfers over port 22 are failing

the_rock — Thu, 14 Jul 2022 20:05:39 GMT

Ok, fair enough...do we see anything in the logs at all or it shows traffic being accepted?

Re: File Transfers over port 22 are failing

David_Chau — Thu, 14 Jul 2022 20:10:46 GMT

Logs show accepted. I am thinking about enabling log accounting to gather more details on this traffic.

Re: File Transfers over port 22 are failing

the_rock — Thu, 14 Jul 2022 20:13:47 GMT

That, but you can also try do fw monitor filters, both below (assuming IP is 1.2.3.4)

fw monitor -e "accept host(1.2.3.4) and port(22);"

fw monitor -F '0,1.2.3.4,0,22,0'

-F flag lets you do in order 'src IP, dst IP, src port, dst port, protocol'

Andy

Re: File Transfers over port 22 are failing

Sorin_Gogean — Fri, 15 Jul 2022 05:12:35 GMT

morning,

anything in SSH logs, that shows why session was dropping ?

could it be that it was taking too much and for a while nothing happened so it was closed ?

you could also try and look to enable some keepalive probing

thank you,

PS: look on port 22 what timeout it has set - mine is 3600 - so the ssh connection would be kept for up to 1 hour before it would be dropped from the CKP FWL if not traffic happens

Re: File Transfers over port 22 are failing

Timothy_Hall — Fri, 15 Jul 2022 11:28:00 GMT

Enable Accounting on the rule(s) matching the transfers and also try enabling TCP state logging to get more information about what is going on when the connections terminate: sk101221: TCP state logging

Re: File Transfers over port 22 are failing

abihsot__ — Fri, 15 Jul 2022 19:09:29 GMT

Try using plain tcp/22 without protocol selected. We had nasty connection drops, but with other ports though.

Re: File Transfers over port 22 are failing

David_Chau — Thu, 21 Jul 2022 19:27:57 GMT

Found the issue. It was the IPS Protection, "Malicious Payload Encoding Remote Code Execution", that are intermittently blocking these transfers.

https://threatpoint.checkpoint.com/ThreatPortal/threat?threatType=protection&threatId=PAYLOAD_ENCODER

Any reason why this IPS protection signature would block the file then allow the file to transfer when reattempted?

Re: File Transfers over port 22 are failing

Sorin_Gogean — Thu, 21 Jul 2022 20:01:25 GMT

That's weird, as it should be quite visible in the FWL logs while looking for SSH traffic from A to B.

Good that you found what was impacting your traffic that randomly.

Re: File Transfers over port 22 are failing

the_rock — Thu, 21 Jul 2022 20:09:10 GMT

Can you see any logs on that IPS protection?

Re: File Transfers over port 22 are failing

David_Chau — Thu, 21 Jul 2022 20:11:34 GMT

Yes, I see the block in the logs and then the traffic is allowed after the file transfer is restarted.

Re: File Transfers over port 22 are failing

the_rock — Thu, 21 Jul 2022 20:13:19 GMT

If I were you, I would add an exception for that, to be on the safe side.

Re: File Transfers over port 22 are failing

David_Chau — Thu, 21 Jul 2022 20:26:17 GMT

Just added the exception. The business is not very happy this was the issue. Curious why this block was so intermittent. These are daily transfers containing the same type of data. These transfers are also encrypted so not sure how this signature flagged this as malicious payload, maybe because of the high volume? Anybody from Check Point able to provide additional details on this IPS protection?

Re: File Transfers over port 22 are failing

Tobias_Moritz — Fri, 22 Jul 2022 06:51:53 GMT

We also see exact these protection hitting on SSH traffic flowing over our gateways from time to time. Could never reproduce it, it just happens from time to time. So I'm also interested about the some background info regarding that protection.

Re: File Transfers over port 22 are failing

Timothy_Hall — Sat, 23 Jul 2022 21:07:36 GMT

The Confidence rating of that "Malicious Payload Encoding Remote Code Execution" protection is Medium, if it were Low I'd be more willing to accept it going off randomly like that. My guess is that this signature is looking for a very short sequence of bytes (<5) without any further context information, and every now and then an encrypted stream happens to match it and get blocked. Seems unlikely but certainly not impossible. Would need to see packet captures of a few separate instances to try to determine what it is seeing that is causing the false positive.

This vulnerability does not appear to have a CVE number and is something Check Point came up with on their own (CPAI-2013-3606) so only R&D will be able to provide further details. Based on how it is behaving and that multiple posters have seen this behavior, I'd argue that the Confidence level of this protection should be changed to Low. Who knows if this attack method is even relevant anymore; this is an advisory from almost 10 years ago.