topic Content Awareness Blade misbehaving (silently blocking/stalling) in General Topics

Content Awareness Blade misbehaving (silently blocking/stalling)

Johan_Hillstrom — Thu, 28 Feb 2019 14:16:41 GMT

We have had some problems recently with aborted FTP transfers and also (unrelated, or so we thought) delayed/stalled HTTP downloads.

On the FTP transfers we found that sometimes we got an alert on the logs, stating "Content Awareness - Error: Internal system error (1000)"
The Fail Mode for Content Awareness is set to Allow all requests (fail-open) but apparently it interferes with traffic anyway.

The second issue, with stalled HTTP downloads, we at first suspected was due to Threat Emulation.

Files would download almost completely and then stall for 1 to 4 minutes.

However, there were no logs from TE blade indicating these files were uploaded and emulated, nor were there any files stuck in TE queue.

We made exceptions in the policy to disable all Threat Prevention blades for this traffic, but that did not help.

But I remembered something from about a year ago with CA doing strange stuff, so we tried disabling it completely, unchecking it on the gateways, not just removing protocols from CA settings.

And lo and behold, downloads started to complete without delay!

Has anyone experienced similar issues?

In the case of HTTP downloads, they would eventually complete and files were correct, but no signs of anything wrong in the logs.

We really want to be able to have CA active, to block clients downloading .EXEs etc, but currently we need to have it off.

Re: Content Awareness Blade misbehaving (silently blocking/stalling)

Alessandro_Marr — Thu, 28 Feb 2019 15:50:29 GMT

Did you check your disk space?

Re: Content Awareness Blade misbehaving (silently blocking/stalling)

Mark_Mitchell — Thu, 28 Feb 2019 15:51:29 GMT

Hi Johan,

What environment are you working with at the minute? Hardware/software version, hotfix version etc?

Regards

Mark

Re: Content Awareness Blade misbehaving (silently blocking/stalling)

Johan_Hillstrom — Thu, 28 Feb 2019 16:48:30 GMT

Sorry, I tagged it with R80.20 only.

We are running two clustered 15600 with separate mgmt.

All are latest and greatest R80.20 maintrain, no custom HFs

Re: Content Awareness Blade misbehaving (silently blocking/stalling)

Johan_Hillstrom — Thu, 28 Feb 2019 16:50:19 GMT

Yes, no problems there.

Both GWs and mgmt and SmartLog have plenty of free soace.

Re: Content Awareness Blade misbehaving (silently blocking/stalling)

Alessandro_Marr — Thu, 28 Feb 2019 17:20:13 GMT

Could you try reenable CA and use a only one rule passing packets just your computer?

Re: Content Awareness Blade misbehaving (silently blocking/stalling)

Mark_Mitchell — Thu, 28 Feb 2019 21:13:42 GMT

Hi Johan,

Thanks for the additional information. I think as you are not seeing anything in the logs that pertains to a reason as to why connections are being terminated (FTP) a debug session is likely needed.

ATRG: Content Awareness (CTNT) <- will detail the debug process. However I would would proceed with extreme caution due to the additional load that the debug will put on the box. As per the following statement from the ARTG.

"Note: Kernel debug increases load on the Security Gateway's CPU. Schedule a maintenance window during a low traffic time. In cluster environment, this procedure must be performed on all members of the cluster."

I have had to run debugs in the past at a time that has low traffic and still managed to max all CPU's at 100%, needless to say things weren't great at this point.

Personally I think it would be best getting a TAC case raised as I would not like to advise on running the debug's without knowing the affected environment in detail.

Regards

Mark