Repeated debug error messages in fwk.elg

[kamilazat]

Hi all!

I'm seeing a lot of messages like below in the fwk.elg file.

FW-1: stopping debug messages for the next 13 seconds. To disable this suppression see sk74580
[28 Sep 20:42:47][fw4_3];[vs_0];[10.x.x.x:32638 -> 10.y.y.y:10250] [ERROR]: up_calc_service_id_key_list: num_of_service_clobs (58) reached limit of entry key

Kernel debug parameters are set to default (as in we do fw ctl debug 0), and tracing options for routed daemon is off. I tried looking up the parts of the messages but found nothing. 

Apparently there is a problem that has been going on for a while. Where can I be getting these messages from? 

Thanks!

[PhoneBoy]

From the TAC cases that mention this error, it appears that there are too many matched services for a single connection.
In this case, the error refers to port 10250.
If you have multiple services that mention this port (either directly or as part of a range), reduce/eliminate them.
These messages will show even if you've disabled debugs.

These errors are happening as part of a caching function that be disabled with: fw ctl set int up_rulebase_use_compound_matching_cache 0
(To permanently disable this, see: https://support.checkpoint.com/results/sk/sk26202 )
However, this can negatively impact performance and is thus not recommended.

[kamilazat]

@PhoneBoy Thank you for the information. 

There are a lot of lines that show other ports as well, however I can see a pattern in those ports. So do I understand you correctly that these messages occur when there are too many connections that correspond to one port?

For example, I see a lot of entries that correspond to port 10250, and only a few IP addresses that are associated with that port. What kind of reduction/elimination do you suggest? We cannot prevent that traffic from happening, and obviously a performance impacting parameter modification is not feasible. 

[PhoneBoy]

What I'm suggesting is you have multiple service objects in your policy configuration that reference port 10250, either directly or as a result of a range.
Reduce/eliminate these objects.

[kamilazat · Accepted solution]

Cleaning my knowledge base, I noticed that I haven't posted the solution we received from TAC about this post.

For anyone having this issue, the solution is to remove "Match for Any" from the all custom created service objects unless absolutely necessary.

Throughout the years I've noticed that administrators create their custom objects even for basic TCP ranges. And without noticing, or knowing exactly what "Match for Any" does, they just enable it on the objects that they create. If this is a case with just several service objects, no problem. But once you start having hundreds of these objects and hundreds of different rules containing these objects, you'll start having a lot of num_of_service_clobs (58) reached limit of entry key messages in fwk.elg file. The gateways will still seem like they're working, but once these error messages become too much, you'll start having weird and seemingly irrelevant issues with overall stability, no matter how buffy the appliances are.

@PhoneBoy's recommendation was on spot. Keep your gateways clean guys!