topic Re: RAD issues? in Firewall and Security Management

RAD issues?

Mattias_Jansson — Mon, 17 Jun 2024 06:53:41 GMT

Hi!
We experience very slow internetaccess this morning and I suspect RAD being the root cause.

Feels like RAD service is not working well hosted by Check point?
Anyone else having the same issue this morning?

Flow ID = flow_8960_9582165
Flow Termination Status:Failed!
Flow Started (08:48:25)
Flow Ended (08:50:31)
Flow Total Time:126 seconds (Timed out)

Flow Steps:
Generic Trap Flow ( Queue Wait=0 / RunTime=0.001413 Seconds )
Cloud HTTP Access(IO) ( Queue Wait=46 / RunTime=0.118964 Seconds )
Handling Decrypted Response ( Queue Wait=1 / RunTime=0.108883 Seconds )
Cloud HTTP Access(IO) ( Queue Wait=79 / RunTime=0 Seconds )
End Of Flow Steps

Flow Items:
_indicator@trapper:vsid=2
_indicator@trapper:version=0
_indicator@trapper:session=
_indicator@trapper:service=malware
_indicator@trapper:resource=google.com/complete/search?output=chrome&q=test
_indicator@trapper:key_len=10
_indicator@trapper:is_ipv6=0
_indicator@trapper:flags=0
_indicator@cpradus:resend=1
Service=malware
Resource=google.com/complete/search?output=chrome&q=test
Malware-Restart=TRUE
FlowError=RAD request exceeded maximum handing time
FetchUrl=http://cws.checkpoint.com:80/Malware/malware/6.0?resource=Z29vZ2xlLmNvbS9jb21wbGV0ZS9zZWFyY2g/b3V0cHV0PWNocm9tZSZxPWthcmF0ZSBraWQgZw==&key=123456&full-url=1
ActiveFlows=941
End of Flow Items

Re: RAD issues?

sascham — Mon, 17 Jun 2024 07:48:42 GMT

Hello,

I can confirm this behavior. We have many logs entries with the reason:

"Failed to fetch Check Point resources. Timeout was reached, check /opt/CPsuite-R81.20/fw1/log/rad_events/Errors/flow_.... For more details"

Those logs ends with:

...
[rad_curl_task.cpp:163] CRadCurlTask::performCurl: [INFO] enter to ...
[rad_curl_task.cpp:180] CRadCurlTask::performCurl: [INFO] Retry mode - pefrom curl totoal retries: 82
[rad_curl_task.cpp:209] CRadCurlTask::performCurl: [INFO] Maxumum retries reached - give up
[rad_curl_task.cpp:114] CRadCurlTask::run: [ERROR] handle: 0xf133ffc0 curl_easy_perform() failed: Timeout was reached

Kind regards

Sascha Mommert

Re: RAD issues?

Mattias_Jansson — Mon, 17 Jun 2024 07:57:32 GMT

Can anyone from Check Point confirm if our problems are caused by issues on cws.checkpoint.com ?

Re: RAD issues?

SWBW_Florian — Mon, 17 Jun 2024 08:19:51 GMT

Same issue here. We can see dozens, hundreds of "rad_curl Maximum retries reached - give up" lines inside the flow logfiles as well.

CPU is climbing up on 2 cores to 90% again and again, but just for small peaks

Re: RAD issues?

Marcel_Gramalla — Mon, 17 Jun 2024 08:33:46 GMT

We also see this behavior. It startet at around 9am (germany) and the RAD daemon also restarted right before that. This also shows in cpview as timeouts. The 48 days before the counter was at 0.

I also have the feeling that the issue startet on Thursday or Friday already but not that bad and maybe just slow response times. But I don't see that in the logs.

Re: RAD issues?

Mattias_Jansson — Mon, 17 Jun 2024 08:36:15 GMT

I just got information that our users also noted that the behaviour started on friday

Re: RAD issues?

Marcel_Gramalla — Mon, 17 Jun 2024 08:46:36 GMT

I just notices that all "System Alert" logs are related to Google searches like: Error occurred while accessing:google.com/complete/search?client=chrome-XXXXXXX

Can you confirm that? Maybe that is causing the issue.

Re: RAD issues?

SWBW_Florian — Mon, 17 Jun 2024 08:49:08 GMT

for our site its also a lot of google, but besides that we have also kleinenzeigen.de and other sites.

But yes, google is (as always if its about RAD processes) a prime process here as well

Re: RAD issues?

Mattias_Jansson — Mon, 17 Jun 2024 08:53:33 GMT

99% google.com for us too (not only searches) and always service=malware

Re: RAD issues?

gg_fga — Mon, 17 Jun 2024 09:45:49 GMT

Hello,

Same issue that impact CPU of the gateway and so all Internet Traffic.

Re: RAD issues?

Marcel_Gramalla — Mon, 17 Jun 2024 10:32:54 GMT

TAC just confirmed the issue to us. They are working on it with high prio.

Re: RAD issues?

Damian_Red — Mon, 17 Jun 2024 11:13:29 GMT

We too have this issue. I created ticket and waiting for response.

Re: RAD issues?

Alex_Hofstee — Mon, 17 Jun 2024 11:56:55 GMT

We are having the same issues, users started reporting Google not working this morning. For us it's also mostly Google searches but also other websites (but a lot less). google.nl seems to be working.

It also seems we get a lot more Blocks / Alerts with various "Internal system error" messages since upgrading from R81.20 T41 to T53. I'm not sure this is related to this issue but it's something we noticed.

Errors like:
Application Control - HTTPS Inspection error occurred (2)
Internal system error in HTTPS Inspection due to categorization service timeout

and:

URL Filtering - Internal system error. (0)

We got the first reports of sites not working last week (June 11th).

Re: RAD issues?

TJ_Aus — Mon, 17 Jun 2024 13:08:42 GMT

We have an active case open for this too since the 5th of June. Mostly google.com not working. No progress to date.
We have only seen it on the SMBs. Firmware version doesn't seem to matter

We too seemed to notice it around the 30th of May we think but we're not sure

Precise Error: Internal system error in HTTPS Inspection due to categorization service timeout

Reason: Application Control - HTTPS Inspection error occurred (2)

Action Reason: Blocking request as configured in engine settings of Application Control

Action: Block

Re: RAD issues?

the_rock — Mon, 17 Jun 2024 13:20:09 GMT

Mine shows updated, but EXACT same date/time as yours.

Andy

Re: RAD issues?

the_rock — Mon, 17 Jun 2024 13:22:24 GMT

What do you see from cpview?

Andy

Re: RAD issues?

gg_fga — Mon, 17 Jun 2024 13:25:29 GMT

Workaround

caution lowers security level

In the meantime, it's possible to improve the situation by means of small workarounds.

Go to the Smartconsole's Manage and Settings menu, then to the "Blades" menu. You'll find several sub-menus, including "Threat Prevention" and "CheckPoint Online Web Service".

- Threat Prevention :

Fail Mode -> Switch to "Fail-open".

- CheckPoint Online Web Service:

App Control & URL Filtering -> switch to "Background

Fail Mode -> Switch to "Fail-open

This would temporarily relieve the flow processing and make it run more smoothly.

Re: RAD issues?

the_rock — Mon, 17 Jun 2024 13:29:17 GMT

But as you said, definitely lowers security and considering we all security oriented here, lets hope there is more permanent solution soon.

Andy

Re: RAD issues?

Alex_Hofstee — Mon, 17 Jun 2024 13:36:30 GMT

We also have that exact version.

Re: RAD issues?

dunkleyc — Mon, 17 Jun 2024 20:43:23 GMT

We were experiencing this too as of last Wednesday, then it started working, and then presented itself again sometime Thurs/Friday into today. It looks like it *might* be working again now...but our checkpoint ticket hasn't given any details. Anyone else have any information worth sharing?