topic Re: "policyCleanUp.py" - OverflowError in API / CLI Discussion

"policyCleanUp.py" - OverflowError

Maik — Fri, 01 Mar 2019 12:16:55 GMT

Hello guys,

This thread is especially targeted to the python community and also to the employees who wrote the policyCleanUp python script. Yesterday I found out that this script exists, so I wanted to give it a try within a lab environment with test data. The rules here actually have a hit count present, some of them do not have any hits at all. So it seems to be a good place to get a first perspective of the output and actual result.

However; I noticed a few things:

1. A shebang line for the respective GAiA version - if you run the script locally on the SMS - would help to run it more easily. That's no problem and could be added.

2. The first execution threw the following error:

admin@myLabSMS:~/Scripts/policyCleanUp# ./policyCleanUp.py
File "./policyCleanUp.py", line 324
SyntaxError: Non-ASCII character '\xe2' in file ./policyCleanUp.py on line 324, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for deta ils

This issue could be solved by specifying a different coding type like for example utf-8.

3. After that the execution - unrelated to any given parameters - ends with the following error:

admin@myLabSMS:~/Scripts/policyCleanUp# ./policyCleanUp.py -k "POL_Test_Package" -u admin -p <MySecretPassword>
[2019-03-01 12:41:08] Plan
[2019-03-01 12:41:09] Validate targets
[2019-03-01 12:41:14] Package POL_Test_Package
[2019-03-01 12:41:14] Layer POL_Test_Package Network
Traceback (most recent call last):
File "./policyCleanUp.py", line 1120, in <module>
main()
File "./policyCleanUp.py", line 1050, in main
if rule_should_be_disabled(rule, global_thresholds) is True:
File "./policyCleanUp.py", line 853, in rule_should_be_disabled
if rule_last_hit_time + datetime.timedelta(days=final_disable_threshold) < DATETIME_NOW:
OverflowError: date value out of range

This could also be resolved by someone who did not write the script, but my guess was that I am not the only one who experiences this issue. In addition to that; I am using the latest Python SDK from here and changed the OS variable as stated there. The GAiA version is R80.20 GA (not a special M Release).

@Community; is there anyone who already used this script? Did you run into the above (or other) errors and if yes - how did you solve them?

@Check Point; maybe someone from the actual script authors can verify the above error in order to verify the script functionality.

Thanks and regards,

Maik

Re: "policyCleanUp.py" - OverflowError

Joshua_Hatter — Fri, 01 Mar 2019 15:42:22 GMT

What is your DEFAULT_DISABLE_THRESHOLD set to, default(180)?

Best I can do is to simply debug this, can you modify the rule_should_be_disabled function to include the following.

Potentially Amiad Stern‌ might have a better idea.

def rule_should_be_disabled(rule, global_thresholds):      global DATETIME_NOW      # Calculate final thresholds     final_disable_threshold = get_rule_final_threshold(rule, 'field-1', 'disable', global_thresholds)     if final_disable_threshold is None:         return False      # Get rule last hit time     rule_last_hit_time = get_rule_last_hit_time(rule)      # Disable rule - no hits too long     try:         if rule_last_hit_time + datetime.timedelta(days=final_disable_threshold) < DATETIME_NOW:             return True     except OverflowError:         print_msg("Last Hit: {}".format(rule_last_hit_time))         print_msg("Disable Threshold: {}".format(datetime.timedelta(days=final_disable_threshold)))         raise      return False

Re: "policyCleanUp.py" - OverflowError

Amiad_Stern — Fri, 01 Mar 2019 18:53:51 GMT

Hi Maik Dummer‌ ,

Thanks for your feedback.

We will address items #1 and #2 on Sunday.

As for item #3 we didn't encounter it during our tests (and it was tested on real large setups), i would like to track this issue fast, can you please contact me on amiads@checkpoint.com so we can continue investigate it offline?

Regards,

Amiad.

Re: "policyCleanUp.py" - OverflowError

Maik — Wed, 06 Mar 2019 13:13:49 GMT

Hello,

First of all; thanks for getting back to me Joshua & Amiad.

Amiad, initially I wanted to answer you via mail as we discussed but I think this information is useful for the whole community. After a little research I found out what the initial cause of the bug is. I was not aware that your script takes use of the custom field option that each customer can configure individually. So to summarize it, the script has default values for disabling and/or deleting rules. Those values are set to 180 days per default. In addition to that it is possible to override the values via parameters, that can be included at the execution time (--disable-after and --delete-after). This is absolutely fine and a very good idea.

But there is more... the third override possibility is checked per each rule. Here the script looks at the value of custom-fields, field 1 to be precise. If the value of this field is an integer it uses this value instead of the "default 180 days" AND instead of the parameter values of --disable-after and --delete-after:

Beginning at line 212 as per current GitHub version:
# Get rule's thresholds.
# Disable and deleted thresholds are calculated by global thresholds or overrides thresholds (local override).
# Local thresholds- user can override the global thresholds by adding number to custom-fields per rule.
def get_rule_final_threshold(rule, field, threshold_type, global_thresholds😞
# Custom-field1 contains override for disable threshold
local_threshold = rule.get('custom-fields', {}).get(field)
# -1 represents a rule that should be skipped all time.

The interesting part about this procedure is that there aren't any ways to avoid the override by custom field option (maybe option is the wrong word, as it really is not optional). In the environment where I tested the script the custom field contains the date when the rule was checked via dual control principle. The content of the custom field for a rule which was checked on the 1st of February 2019 would contain the value "20190201" - this translated to the integer reading process of the script would lead to 20.190.201 days, resulting in the OverFlow error.

In addition to that it seems like the script does not read all the rules from a given policy package, but this could be a result due to the just mentioned threshold override.

Maybe I am blind, but is there a possibility to run this script just with parameter or default values regarding the disable/delete time? If not, is there any reason to not include one? And why is the script referencing to custom-fields at all? Don't misunderstand me, this can be very convinient to handle each rule on its own, but it is simply not a script that can be shared with the community in such a case.

Thanks in advance for any reply and hint.

Best regards,

Maik

Re: "policyCleanUp.py" - OverflowError

Maik — Wed, 13 Mar 2019 12:22:44 GMT

Hello Amiad,

Did you already have time to check my response?

Thanks and best regards,

Maik