RAD issues?

[Mattias_Jansson]

Hi!
We experience very slow internetaccess this morning and I suspect RAD being the root cause.

Feels like RAD service is not working well hosted by Check point?
Anyone else having the same issue this morning?

Flow ID = flow_8960_9582165
Flow Termination Status:Failed!
Flow Started (08:48:25)
Flow Ended (08:50:31)
Flow Total Time:126 seconds (Timed out)

Flow Steps:
Generic Trap Flow ( Queue Wait=0 / RunTime=0.001413 Seconds )
Cloud HTTP Access(IO) ( Queue Wait=46 / RunTime=0.118964 Seconds )
Handling Decrypted Response ( Queue Wait=1 / RunTime=0.108883 Seconds )
Cloud HTTP Access(IO) ( Queue Wait=79 / RunTime=0 Seconds )
End Of Flow Steps

Flow Items:
_indicator@trapper:vsid=2
_indicator@trapper:version=0
_indicator@trapper:session=
_indicator@trapper:service=malware
_indicator@trapper:resource=google.com/complete/search?output=chrome&q=test
_indicator@trapper:key_len=10
_indicator@trapper:is_ipv6=0
_indicator@trapper:flags=0
_indicator@cpradus:resend=1
Service=malware
Resource=google.com/complete/search?output=chrome&q=test
Malware-Restart=TRUE
FlowError=RAD request exceeded maximum handing time
FetchUrl=http://cws.checkpoint.com:80/Malware/malware/6.0?resource=Z29vZ2xlLmNvbS9jb21wbGV0ZS9zZWFyY2g/b3V0cH...
ActiveFlows=941
End of Flow Items

[sascham]

Hello,

I can confirm this behavior. We have many logs entries with the reason:

"Failed to fetch Check Point resources. Timeout was reached, check /opt/CPsuite-R81.20/fw1/log/rad_events/Errors/flow_.... For more details"

Those logs ends with:

...
[rad_curl_task.cpp:163] CRadCurlTask::performCurl: [INFO] enter to ...
[rad_curl_task.cpp:180] CRadCurlTask::performCurl: [INFO] Retry mode - pefrom curl totoal retries: 82
[rad_curl_task.cpp:209] CRadCurlTask::performCurl: [INFO] Maxumum retries reached - give up
[rad_curl_task.cpp:114] CRadCurlTask::run: [ERROR] handle: 0xf133ffc0 curl_easy_perform() failed: Timeout was reached

Kind regards

Sascha Mommert

[Mattias_Jansson]

Can anyone from Check Point confirm if our problems are caused by issues on cws.checkpoint.com ?

[SWBW_Florian]

Same issue here. We can see dozens, hundreds of "rad_curl Maximum retries reached - give up" lines inside the flow logfiles as well.

CPU is climbing up on 2 cores to 90% again and again, but just for small peaks

[Marcel_Gramalla]

We also see this behavior. It startet at around 9am (germany) and the RAD daemon also restarted right before that. This also shows in cpview as timeouts. The 48 days before the counter was at 0. 

I also have the feeling that the issue startet on Thursday or Friday already but not that bad and maybe just slow response times. But I don't see that in the logs.

[Mattias_Jansson]

I just got information that our users also noted that the behaviour started on friday

[Marcel_Gramalla]

I just notices that all "System Alert" logs are related to Google searches like: Error occurred while accessing:google.com/complete/search?client=chrome-XXXXXXX

Can you confirm that? Maybe that is causing the issue.

[SWBW_Florian]

for our site its also a lot of google, but besides that we have also kleinenzeigen.de and other sites.

But yes, google is (as always if its about RAD processes) a prime process here as well

[Mattias_Jansson]

99% google.com for us too (not only searches) and always service=malware

[gg_fga]

Hello,

Same issue that impact CPU of the gateway and so all Internet Traffic.

[Marcel_Gramalla]

TAC just confirmed the issue to us. They are working on it with high prio.

[Damian_Red]

We too have this issue. I created ticket and waiting for response.

[Alex_Hofstee]

We are having the same issues, users started reporting Google not working this morning. For us it's also mostly Google searches but also other websites (but a lot less). google.nl seems to be working.

It also seems we get a lot more Blocks / Alerts with various "Internal system error" messages since upgrading from R81.20 T41 to T53. I'm not sure this is related to this issue but it's something we noticed. 

Errors like: 
Application Control - HTTPS Inspection error occurred (2)
Internal system error in HTTPS Inspection due to categorization service timeout

and:

URL Filtering - Internal system error. (0)

We got the first reports of sites not working last week (June 11th).

[TJ_Aus]

We have an active case open for this too since the 5th of June. Mostly google.com not working. No progress to date.
We have only seen it on the SMBs. Firmware version doesn't seem to matter

We too seemed to notice it around the 30th of May we think but we're not sure

t

Precise Error: Internal system error in HTTPS Inspection due to categorization service timeout

Reason:                 Application Control  -  HTTPS Inspection error occurred (2)

Action Reason:          Blocking request as configured in engine settings of Application Control

Action:                 Block

[the_rock]

Mine shows updated, but EXACT same date/time as yours.

Andy

 

[Alex_Hofstee]

We also have that exact version. 

[the_rock]

What do you see from cpview?

Andy

 

[gg_fga]

Workaround

caution lowers security level

In the meantime, it's possible to improve the situation by means of small workarounds.

Go to the Smartconsole's Manage and Settings menu, then to the "Blades" menu. You'll find several sub-menus, including "Threat Prevention" and "CheckPoint Online Web Service".

- Threat Prevention :

Fail Mode -> Switch to "Fail-open".

- CheckPoint Online Web Service:

App Control & URL Filtering -> switch to "Background

Fail Mode -> Switch to "Fail-open

This would temporarily relieve the flow processing and make it run more smoothly.

[the_rock]

But as you said, definitely lowers security and considering we all security oriented here, lets hope there is more permanent solution soon.

Andy

[dunkleyc]

We were experiencing this too as of last Wednesday, then it started working, and then presented itself again sometime Thurs/Friday into today. It looks like it *might* be working again now...but our checkpoint ticket hasn't given any details. Anyone else have any information worth sharing?

[the_rock]

Maybe  @PhoneBoy or @_Val_ can share some info about it, if they have any.

Andy

[Marcel_Gramalla]

I don't know why there is no official statement but the issue should be resolved now. We don't see any issues anymore and everything is fast as it should be.

[Alex_Hofstee]

For us it still isn't working and google.com / searches from the Chrome address bar are still giving problems. So not so sure if it's resolved for everyone.

[PhoneBoy]

If people want to PM me SRs they have open with TAC on this issue, I can investigate.

[SWBW_Florian]

similar situation here. Last warning was yesterday 21:12. Until now no new alert. But even no update by checkpoint

[TJ_Aus]

This now appears to be fixed

[the_rock]

In the spirit of the community, would you mind sharing how it got fixed? It can definitely help others in the future.

Andy

[the_rock]

@TJ_Aus informed me they are still awaiting for final response from TAC, so will update once thats provided.

Andy