Error logs in Threat Emulation

[Baggy]

An event occurred in which an error log in threat emulation was output and files could not be downloaded or uploaded.
Emulation is running on ThreatCloud.

In the log, the reason for the error is explained as follows
Reason：Timeout was exceeded

I think one of the causes is that the emulation connnection handling mode of Threat Emulation is set to Maximum Prevention in the Profile setting of the Threat Prevention Policy.

Do you know the main cause?

[the_rock]

Would you mind please send what you have configured under settings in smart console (last tab on the left menu) for that blade? I will send screenshot in a bit.

Andy

[the_rock]

This is what I was referring to.

Andy

[Baggy]

@the_rock 
Thank you for your response.

Here is the Screenshot. It is default.
The fail-mode is set to fail-open.

[the_rock]

I selected wrong tab, I meant to choose general...is it set to allow?

Andy

[Baggy]

@the_rock 
Yes Allow all connections Fail-open

In profile, Maximum Prevention is set and the file cannot be downloaded until the file Emulation is complete.
If this setting is set to Rapid Delivery, it improves, but I would like to know the cause of the error.

[the_rock]

See if below sk helps.

https://support.checkpoint.com/results/sk/sk114806

If not, I would try change to background option and install policy. If still no improvement, would certainly open TAC case.

Best,

Andy

[Baggy]

@the_rock 

I know about this SK.

If i set Rapid Delivery will improve this.
The question is why the Timeout was exceeded error occurs.

[the_rock]

I have no clue, sorry. Thats why I sugegsted TAC case.

Andy

[Baggy]

I am asking here because the TAC says they don't know.

[Baggy]

I am asking here because the TAC says they don't know.

When I checked the result of tecli show cloud queue at the time of the event, it showed “Cloud Connectivity Problem” and confirmed that there were many files waiting for emulation (inspection) in the queue.

Does this mean that a connection problem with ThreatCloud caused the timeout?
However, some files could be downloaded and uploaded.
Please let me know what you find out.

------------------------
tecli show cloud queue
------------------------

|file's sha1 |file's event_id |file type |insert time |status
|----------------------------------------|----------------------------------------|----------|------------------|----------------------------------------------------------------------
|pdf |203 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |197 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |96 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |94 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |88 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |85 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |82 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |81 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |77 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |75 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |16 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |7 Minutes |Cloud Connectivity Problem, waiting to resend.
|pdf |205 Minutes |Uploading to Cloud (resend).

[the_rock]

Hm, thats what it seems like, possibkly connectivity issue to threat cloud...why would that heppen, that Im really not sure. Lets see if someone else may have an idea.

Any other relevant logs that you can send that might be bit more helpful?

Andy

[Baggy]

@the_rock 

I can't share the details because it is a customer's log, but the Firewall log also recorded an error with File exceeded size limit. However, we verified that if the following settings are set to 15MB and fail-open, files exceeding the size limit can be downloaded.

SmartConsole > Manage & Settings > Blades > Threat Prevention > Advanced Settings >
Threat Emulation > Emulation Limits > Maximum file size for emulation

The maximum processing time in queue is 720 minutes, so I don't think it will be a Fail-open download, but if the error is Timeout was exceeded due to a connection problem with ThreatCloud, why can't the file be downloaded in Fail-open?

[the_rock]

Well, here is the way I look at it. If you are saying TAC could not help, the more we get here, better chances we can try solve it. I get you might not be allowed to send certain things, but if you blur out any sensitive data from the log entry, that would help big time.

Best,

Andy

[Baggy]

@the_rock 

Thank you for your support.
It is possible to blur the log and share it.

[the_rock]

Ok, please do share it once ready.

Best,

Andy

[Baggy]

@the_rock 

I apologize for the delayed response.

After further investigation, I found that at the same time the Firewall log recorded a “Timeout was exceeded” message under Threat Emulation, the tp_failures.elg file also showed evidence of a timeout occurring in the Anti-Virus component, as shown below.

The number of logs, timestamps, source ports, and destination ports all match those of the “Timeout was exceeded” events.

We looked further into the Anti-Virus component, but could not find any information indicating the values for Hold Timeout or Anti-Virus Timeout.

If you have any information regarding this, please let me know.

tp_failures.elg - The failures tp_collector finds (in clear data, you can print this file).
--------------------

Time:6/24 08:47:01; Inst:0; Conn:dir 0, IPP 6; Session:1539566; App:CI_AV; TE Async Response Time:08:37:01; TE Verdict:NONE; RAD Async Response Time:08:37:01; RAD Verdict:ACCEPT; Hold Timeout:08:39:14; AV Record Timeout:08:47:01; Fail-Open:1; Session ended with error:1; av_handler:7fcf5af46208;

Time:6/24 08:48:56; Inst:1; Conn:dir IPP 6; Session:1553737; App:CI_AV; TE Async Response Time:08:38:56; TE Verdict:NONE; Hold Timeout:08:42:43; AV Record Timeout:08:48:56; Fail-Open:1; Session ended with error:1; av_handler:7fcf51ab9c08;

Time:6/24 08:49:21; Inst:0; Conn:dir 0, IPP 6; Session:1540164; App:CI_AV; AV Record Timeout:08:49:21; Fail-Open:1; Session ended with error:1; av_handler:7fcf5af3e608;

Time:6/24 08:51:25; Inst:1; Conn:dir 0, IPP 6; Session:1554664; App:CI_AV; AV Record Timeout:08:51:25; Fail-Open:1; Session ended with error:1; av_handler:7fcf51aba208;

--------------------

[the_rock]

Maybe check these things:

• Threat Emulation / cloud queues & status

  • tecli show cloud queue — shows cloud queue and files stuck in cloud emulation.

  • tecli show emulator emulations (or tecli s e e) — status of emulation queue and VM workers.

  • tecli show remote queue — if you use an on-prem TE appliance, check the remote queue.
    These were suggested in Check Point docs and community posts for identifying TE backlog/latency.

• Check TE/AV statistics and engine health

  • tecli show statistics — general TE throughput/latency counters.

  • tecli show emulator vm detailed — see VM usage and if VMs are overloaded or unavailable.

  • cpview or top — look for high CPU, memory or I/O on the gateway that could delay engine responses. (CPView is in the CLI reference).

• Check logs and timestamps correlation

  • Inspect tp_failures.elg and the Security logs for the same session IDs/timestamps to correlate TE/AV timeouts with the user traffic.

  • If using cloud, check whether tecli show cloud queue shows long upload/processing times (SKs and KBs note cloud latency of minutes in some cases)

• Threat Prevention settings to review in SmartConsole

  • Manage & Settings → Blades → Threat Prevention → Advanced Settings (Threat Emulation / Anti-Virus). There you can view emulation timeouts, emulation environment (cloud vs on-prem), and rapid-delivery / fail-open behavior. (Docs show where to find these engine settings

[the_rock]

Also, under engine settings, I would put it to background instead of hold and install policy, test.

[Baggy]

@the_rock 

Thank you for your advice.
But I understand the settings you have explained to me.

Based on that, I would like to know under what conditions “Hold Timeout” and “Anti-Virus Timeout” are recorded in tp_failures.elg, and whether there are specific timeout values defined for them.

[the_rock]

I believe they are both located in $FWDIR/conf/scrub_debug.conf file. One is 60s and other one is 30s. But regardless, setting I mentioned relates to requests being ALLOWED if there is an issue with the engine, thats why I suggested it.

[Baggy]

@the_rock 

$FWDIR/conf/scrub_debug.conf is the configuration file for Threat Extraction, so I don't believe it is relevant.

$FWDIR/conf/malware_config. contains two settings related to timeouts. Could these be relevant?
・http_request_timeout
・url_entry_timeout

[the_rock]

So sorry mate, I mixed up the two. Yes, that definitely might be relevant. Can you provide the values you have, I can check in my lab as well.

[Baggy]

@the_rock 

The timeout settings in $FWDIR/conf/malware_config have not been changed.

[tp_conf_service]
port=12872
http_request_timeout=60 ★

[strict_hold_fail_open_config]
strict_hold_fail_open_flag=1
url_entry_timeout=30 ★
url_key_type=1
compare_second_try_md5=0

“Strict Hold” and url_entry_timeout=30 are settings that seem to be effective not only for Threat Extraction but also for Anti-Virus and Threat Emulation. Is this correct?

Anti-Virus Blade - Strict hold is not possible failure - Write to other side occured

https://community.checkpoint.com/t5/Threat-Prevention/Anti-Virus-Blade-Strict-hold-is-not-possible-f...

Solution ID: sk183840
Threat Prevention "Strict Hold" mode on the Security Gateway
https://support.checkpoint.com/results/sk/sk183840

regards,
Baggy

[the_rock]

You are correct. By the way, I would try change those values and see if it helps.

[Baggy]

@the_rock 

Thank you for your cooperation. I will try it as well.

[the_rock]

Of course, we always work as a team here to find a solution. Apologies it may take some time, as some issues are more complicated, but Im hopeful for positive results.

[the_rock]

Just to be on the safe side, I would take backup and also show configuration from clish as well. You can also output the config into a file -> clish -c "show configuration" > /var/log/fwconfig.txt

[the_rock]

@Baggy Just curious, did not ask you this, but wondering...when did this issue happen originally? Any idea if any changes may had been done to cause the problem?

[Baggy]

@the_rock 

It occurred on 24 June this year.

I didn't make any particular changes to the settings, but as I mentioned earlier,
one of the causes is that the emulation connnection handling mode of Threat Emulation is set to Maximum Prevention in the Profile setting of the Threat Prevention Policy.