Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Mattias_Jansson
Collaborator

RAD issues?

Hi!
We experience very slow internetaccess this morning and I suspect RAD being the root cause.

Feels like RAD service is not working well hosted by Check point?
Anyone else having the same issue this morning?

Flow ID = flow_8960_9582165
Flow Termination Status:Failed!
Flow Started (08:48:25)
Flow Ended (08:50:31)
Flow Total Time:126 seconds (Timed out)

Flow Steps:
Generic Trap Flow ( Queue Wait=0 / RunTime=0.001413 Seconds )
Cloud HTTP Access(IO) ( Queue Wait=46 / RunTime=0.118964 Seconds )
Handling Decrypted Response ( Queue Wait=1 / RunTime=0.108883 Seconds )
Cloud HTTP Access(IO) ( Queue Wait=79 / RunTime=0 Seconds )
End Of Flow Steps

Flow Items:
_indicator@trapper:vsid=2
_indicator@trapper:version=0
_indicator@trapper:session=
_indicator@trapper:service=malware
_indicator@trapper:resource=google.com/complete/search?output=chrome&q=test
_indicator@trapper:key_len=10
_indicator@trapper:is_ipv6=0
_indicator@trapper:flags=0
_indicator@cpradus:resend=1
Service=malware
Resource=google.com/complete/search?output=chrome&q=test
Malware-Restart=TRUE
FlowError=RAD request exceeded maximum handing time
FetchUrl=http://cws.checkpoint.com:80/Malware/malware/6.0?resource=Z29vZ2xlLmNvbS9jb21wbGV0ZS9zZWFyY2g/b3V0cH...
ActiveFlows=941
End of Flow Items

 

28 Replies
sascham
Participant

Hello,

I can confirm this behavior. We have many logs entries with the reason:

"Failed to fetch Check Point resources. Timeout was reached, check /opt/CPsuite-R81.20/fw1/log/rad_events/Errors/flow_.... For more details"

Those logs ends with:

...
[rad_curl_task.cpp:163] CRadCurlTask::performCurl: [INFO] enter to ...
[rad_curl_task.cpp:180] CRadCurlTask::performCurl: [INFO] Retry mode - pefrom curl totoal retries: 82
[rad_curl_task.cpp:209] CRadCurlTask::performCurl: [INFO] Maxumum retries reached - give up
[rad_curl_task.cpp:114] CRadCurlTask::run: [ERROR] handle: 0xf133ffc0 curl_easy_perform() failed: Timeout was reached

Kind regards

Sascha Mommert

Mattias_Jansson
Collaborator

Can anyone from Check Point confirm if our problems are caused by issues on cws.checkpoint.com ?

SWBW_Florian
Contributor

Same issue here. We can see dozens, hundreds of "rad_curl Maximum retries reached - give up" lines inside the flow logfiles as well.

CPU is climbing up on 2 cores to 90% again and again, but just for small peaks

regards
Marcel_Gramalla
Advisor

We also see this behavior. It startet at around 9am (germany) and the RAD daemon also restarted right before that. This also shows in cpview as timeouts. The 48 days before the counter was at 0. 

I also have the feeling that the issue startet on Thursday or Friday already but not that bad and maybe just slow response times. But I don't see that in the logs.

 

 

Mattias_Jansson
Collaborator

I just got information that our users also noted that the behaviour started on friday

0 Kudos
Marcel_Gramalla
Advisor

I just notices that all "System Alert" logs are related to Google searches like: Error occurred while accessing:google.com/complete/search?client=chrome-XXXXXXX

Can you confirm that? Maybe that is causing the issue.

0 Kudos
SWBW_Florian
Contributor

for our site its also a lot of google, but besides that we have also kleinenzeigen.de and other sites.

But yes, google is (as always if its about RAD processes) a prime process here as well

regards
0 Kudos
Mattias_Jansson
Collaborator

99% google.com for us too (not only searches) and always service=malware

0 Kudos
gg_fga
Contributor

Hello,

Same issue that impact CPU of the gateway and so all Internet Traffic.

 

0 Kudos
Marcel_Gramalla
Advisor

TAC just confirmed the issue to us. They are working on it with high prio.

Damian_Red
Participant

We too have this issue. I created ticket and waiting for response.

0 Kudos
Alex_Hofstee
Participant

We are having the same issues, users started reporting Google not working this morning. For us it's also mostly Google searches but also other websites (but a lot less). google.nl seems to be working.

It also seems we get a lot more Blocks / Alerts with various "Internal system error" messages since upgrading from R81.20 T41 to T53. I'm not sure this is related to this issue but it's something we noticed. 

Errors like: 
Application Control - HTTPS Inspection error occurred (2)
Internal system error in HTTPS Inspection due to categorization service timeout

and:

URL Filtering - Internal system error. (0)

We got the first reports of sites not working last week (June 11th).

0 Kudos
TJ_Aus
Contributor

We have an active case open for this too since the 5th of June. Mostly google.com not working. No progress to date.
We have only seen it on the SMBs. Firmware version doesn't seem to matter

We too seemed to notice it around the 30th of May we think but we're not sure

Capture.JPGt

Precise Error: Internal system error in HTTPS Inspection due to categorization service timeout

Reason:                 Application Control  -  HTTPS Inspection error occurred (2)

Action Reason:          Blocking request as configured in engine settings of Application Control

Action:                 Block

0 Kudos
the_rock
Legend
Legend

Mine shows updated, but EXACT same date/time as yours.

Andy

Screenshot_1.png

 

0 Kudos
Alex_Hofstee
Participant

We also have that exact version. 

0 Kudos
the_rock
Legend
Legend

What do you see from cpview?

Andy

 

Screenshot_2.png

0 Kudos
gg_fga
Contributor

Workaround

caution lowers security level

In the meantime, it's possible to improve the situation by means of small workarounds.

Go to the Smartconsole's Manage and Settings menu, then to the "Blades" menu. You'll find several sub-menus, including "Threat Prevention" and "CheckPoint Online Web Service".

 

- Threat Prevention :

Fail Mode -> Switch to "Fail-open".

 

- CheckPoint Online Web Service:

App Control & URL Filtering -> switch to "Background

Fail Mode -> Switch to "Fail-open

 

This would temporarily relieve the flow processing and make it run more smoothly.

0 Kudos
the_rock
Legend
Legend

But as you said, definitely lowers security and considering we all security oriented here, lets hope there is more permanent solution soon.

Andy

0 Kudos
dunkleyc
Explorer

We were experiencing this too as of last Wednesday, then it started working, and then presented itself again sometime Thurs/Friday into today. It looks like it *might* be working again now...but our checkpoint ticket hasn't given any details. Anyone else have any information worth sharing?

0 Kudos
the_rock
Legend
Legend

Maybe  @PhoneBoy or @_Val_ can share some info about it, if they have any.

Andy

0 Kudos
Marcel_Gramalla
Advisor

I don't know why there is no official statement but the issue should be resolved now. We don't see any issues anymore and everything is fast as it should be.

0 Kudos
Alex_Hofstee
Participant

For us it still isn't working and google.com / searches from the Chrome address bar are still giving problems. So not so sure if it's resolved for everyone.

0 Kudos
PhoneBoy
Admin
Admin

If people want to PM me SRs they have open with TAC on this issue, I can investigate.

SWBW_Florian
Contributor

similar situation here. Last warning was yesterday 21:12. Until now no new alert. But even no update by checkpoint

regards
TJ_Aus
Contributor

This now appears to be fixed

the_rock
Legend
Legend

In the spirit of the community, would you mind sharing how it got fixed? It can definitely help others in the future.

Andy

0 Kudos
the_rock
Legend
Legend

@TJ_Aus informed me they are still awaiting for final response from TAC, so will update once thats provided.

Andy

0 Kudos
LST
Contributor

same here

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events