Disable/Delete Rules with a Zero Hit Count (MDS or SMS)

[Adam_Forester]

**v3 and above now allows you to pick a specific access layer** 

**v4 added new functions thanks to user feedback. Now has the ability to navigate around section title headers and to handle of any size**

**v5 with a lot of work by Vincent Bacher‌ he determined that some larger policies need a time specified to search. This version added in a 6 month limit on hits prior to the day you run it (Today - 6Months.)**

** v6 combined MDS & SMS into a single script. Added the ability to disable or delete rules based on UID or NAME. The disable script will add a commend 'Disabled by Zero Hits'

This is a simple shell script that will allow you to parse a specific rulebase for rules with a ZERO hit count. The results will be output into a single file of mgmt_cli commands to disable or delete those rules.

The script is setup to run on the Mgmt station itself and uses the 'mgmt_cli -r true' function and uses the -d DOMAIN flag to support SMS and MDS in a single script

It is highly recommended to run the 'DISABLE' version prior to running a 'DELETE' it will treat it as a staging for full deletion

How to Use

• Move script to the management station
• ./cleanup-zero-hits.sh
• Enter IP address of SMS or CMA you wish to check
• Follow remaining prompts for options
  • uid or name
    • The script will ask if you want to export with uid or name. UID is more accurate as it does not change with position. This will prevent a situation where another admin is adding/removing rules from the rulebase before you are able to run the output file.

You can take the delete/disable command file and run it.

• chmod 755 Output-Filename.txt
• ./Output-Filename.txt

Original files on github: GitHub - cpmidsouth/Delete-or-Disable-Zero-Hit-Rules: This script is designed to search a specifed r... 

NOTE: If you use inline layers within the rulebase you will need to search those as a separate layer. This script is not effective in a rulebase where multiple targets within the same rulebase. I am working on that one. Thanks to Vincent Bacher‌ for being my QA and spending way too much time testing with me. 

Feedback welcome this was a simple project that came out of a client request.

[Tomer_Sole]

Very nice, and from field knowledge I know it's useful 

And I like how outside of input/output text it's essentially one line of code. So that's kind of pretty on the inside

Few comments in case you are interested..

- The fact that you added "[PolicyName] + Network" means it will only work for Firewall-only users who upgraded their policies. With R80.10, users can create multiple layers in the policy, and then the names of those layers are determined by the users' choice. Also, users with FW+APPI who upgrade their policies will have a policy with 2 ordered layer: "[PolicyName] Network" and "[PolicyName] Applications" (and they can rename those layers as they wish)

- Deleting rules based on their number can be problematic in case someone has just published new rules above them. I'm not sure how common this scenario, but when we designed R80 we had multiple users in mind. So taking the "uid" of the rule rather than "rule-number" is safer.

[Adam_Forester · Accepted solution]

Tomer thanks for the input. I did write it specifically for an account that had upgraded, I didn't take into account the creation of policies in 80.10. I will work on that specific for the next version.

Agree on the safeness of UID, I chose rule number as a way for the customer to manually review before letting the API do the deleting, call it controlled paranoia. But I did tweak a couple things this morning to use UID tested it and it works just fine, I think I will upload that as a second option.

mgmt_cli -r true show access-rulebase name "$POL_NAME Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true delete access-rule uid " + (.uid|tostring) + " layer")' > $POL_NAME-tmp.txt; sed "s,$, '$POL_NAME Network'," $POL_NAME-tmp.txt > $POL_NAME-delete-unused-uid.txt; rm *tmp.txt

[Ram1]

Hi Adam,

Good day, I need few clarifications and help from you.

1) to get full file bases which are disabled using a script.

2) how alert rules that are going to expire through email.

3) how to get the details of expired rule bases

Please let me know if any clarifications.

Regards

Ram

[Tomer_Sole]

Awesome. Also I'm glad that you managed to workaround a customer's request by yourself. Imagine what you had to do if we didn't have this

[Adam_Forester]

Tomer, I've now updated it to list out the Access Layers and allow the users to pick instead of just assuming the addition of Network. This should allow it to work for all packages upgraded or not.

[Jill_Sanders]

Great job!

[Vincent_Bacher]

Hi Adam,

nice useful script, good job.

One question:

I have just tested the script but unfortunately get no result, that means the utput file is empty.

Started the script:

./delete-zero-hits-rulenumber-v3.sh
This script will search a specific policy package for rules with a ZERO hit count.
Use with caution for deleting rules..
If for any reason you make a typo and need to exit use CTRL+C.
Press ENTER  to continue
Listing Access Policy Package Names.
Applications
************-FW-POLICY Network
What is the Policy Package Name?

I tried the policy Name with or without "Network"

************-FW-POLICY Network

************-FW-POLICY

With or without quotation marks. Output file is always empty. Hopefully i find some time to have a look into the script and at the mgmt_cli commands.
Cheers

Vincent

[Adam_Forester]

Interesting Vincent. Is it possible you don't have rules with a zero hit count? This is tested on 80.10.

I'm going to have you run the raw command to look for the output;

mgmt_cli -r true show access-rulebase name "************-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[] | select(.hits.value == 0)'

See what that outputs. If it's nothing then run this;

mgmt_cli -r true show access-rulebase name "************-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[]'

Then look for the output of each rule and the hits value;  which should look like the below (I formatted the text you would look for)

"rule-number": 1,

  "track": {

    "type": "598ead32-aa42-4615-90ed-f51a5928d41d",

    "per-session": false,

    "per-connection": true,

    "accounting": false,

    "alert": "none"

  },

  "source": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "source-negate": false,

  "destination": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "destination-negate": false,

  "service": [

    "97aeb470-9aea-11d5-bd16-0090272ccb30"

  ],

  "service-negate": false,

  "vpn": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "action": "6c488338-8eec-4103-ad21-cd461ac2c472",

  "action-settings": {

    "enable-identity-captive-portal": false

  },

  "content": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "content-negate": false,

  "content-direction": "any",

  "time": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "hits": {

    "percentage": "1%",

    "level": "low",

    "value": 24946,

    "first-date": {

      "posix": 1518945364000,

      "iso-8601": "2018-02-18T03:16-0600"

    },

    "last-date": {

      "posix": 1527527490000,

      "iso-8601": "2018-05-28T12:11-0500"

    }

  },

  "custom-fields": {

    "field-1": "",

    "field-2": "",

    "field-3": ""

  },

  "meta-info": {

    "lock": "unlocked",

    "validation-state": "ok",

    "last-modify-time": {

      "posix": 1497028032827,

      "iso-8601": "2017-06-09T12:07-0500"

    },

    "last-modifier": "admin",

    "creation-time": {

      "posix": 1497028010878,

      "iso-8601": "2017-06-09T12:06-0500"

    },

    "creator": "admin"

  },

  "comments": "",

  "enabled": true,

  "install-on": [

    "6c488338-8eec-4103-ad21-cd461ac2c476"

  ]

}

{

  "uid": "b11cd15c-f55b-4450-8f96-09ccda45f1bc",

  "type": "access-rule",

  "domain": {

    "uid": "41e821a0-3720-11e3-aa6e-0800200c9fde",

    "name": "SMC User",

    "domain-type": "domain"

  },

[Vincent_Bacher]

Hi Adam, 

in fact I have rules with zero hitcount. Did not have time yet to try running the raw commands, will do that tomorrow.

Cheers 

Vincent 

[Adam_Forester]

Sounds good. I’ll be interested to see. I’m going to send you my check Point email as well so we can work offline and report back

[Adam_Forester]

I had another thought... run the UID version of the script and check it’s output as well. 

[Vincent_Bacher]

Did not receive your mail 😉

Just ran the commands you suggested manually:

# mgmt_cli -r true show access-rulebase name "*******-INT-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[] | select(.hits.value == 0)'

--> nothing

running:

mgmt_cli -r true show access-rulebase name "********-INT-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[]'

Shows a list of round about 50 no-hits rules. Should be correct amount.

So question is why select(.hits.value == 0)' has no match

[Vincent_Bacher]

Empty as well.

[Adam_Forester]

Wanted to drop a note here as well for anyone following along. Vincent's testing helped me determine a few things that I did not test in my lab;

1. The original script was not setup to handle section title headers. *RESOLVED*

2. I was unaware of a the default limit of "50" being used. This was causing the script to only search the first 50 rules of a rulebase. This was not an issue in my lab as I was only testing with 25. I've since expanded the script to determine the rule count and to loop until it searches all rules. *RESOLVED*

3. The script does not take into account a rulebase that has multiple targets in it. *WORKING TOWARDS RESOLUTION*

Tomer Sole

[Tomer_Sole]

if you want to be extra awesome you can try multi-domain policies...

[Adam_Forester]

Like this? https://community.checkpoint.com/docs/DOC-2813 

Now im still working on updating that one to take the limit into account. Should be done tonight or tomorrow. 

[Robert_Decker]

Hi Adam,

please take a look at our Management API Python SDK on GitHub - 

https://github.com/CheckPointSW/cp_mgmt_api_python_sdk/blob/master/lib/mgmt_api.py

and go the the method called "api_query" on line 318.

this method shows how to query ALL objects, bypassing 50 objects limit.

Robert.

[Luciano_Miguel]

Hello

Its possible change the action of this script from DELETE HitCount zero Rule to DISABLE RULE ? And Put one comment to identification of rule after script execution ?

Thank you 

[Adam_Forester]

Thanks Robert. I’ll be honest I do t speak python all that well, I’m an old basher. 🙂 but I do need to expand and learn it. 

I was able to achieve the limit in bash by doing a total then setting my limit to 500 but looping the offset in a seq with it incrementing by 500 until it hits $total. 

[Adam_Forester]

Luciano Miguel‌ sure can. Sorry for posting it here fully but I’m traveling and did this from my cell phone and couldn’t upload a file. Was able to test on my sms and confirmed it worked out just fine. Just copy and paste the below. It will make it where you can disable the rules and adds a Comment that says “disabled by API Zero Hits

today="$(date +%Y-%m-%d)"

from="$(date --date="6 months ago" +%Y-%m-%d)"

timestamp="$(date +%Y-%m-%d-%H-%M-%S)"

printf  "This script will search a specific policy package for rules with a ZERO hit count.\nUse with caution for deleting rules..\nIf for any reason you make a typo and need to exit use CTRL+C.\nPress ENTER  to continue"

read ANYKEY

printf "\nListing Access Policy Package Names\n"

mgmt_cli -r true show access-layers limit 500 --format json | jq --raw-output '."access-layers"[] | (.name)'

printf "\nWhat is the Policy Package Name?\n"

read POL_NAME

POL2=$(echo $POL_NAME | tr -d ' ')

printf "\nDetermining Rulesbase Size\n"

total=$(mgmt_cli -r true show access-rulebase name "$POL_NAME" --format json |jq '.total')

printf "There are $total rules in $POL_NAME\n"

printf "\nDoes Your Policy Contain Section Title Headers?[y/n]\n"

read SECHEAD

if [ "$SECHEAD" = "y" ]; then

printf "\nCreating Deletion Scripts. This may take a minute depending on Rulebase size.\n"

for I in $(seq 0 500 $total)

do

 mgmt_cli -r true show access-rulebase name "$POL_NAME" details-level "standard" offset $I limit 500 use-object-dictionary true show-hits true hits-settings.from-date $from hits-settings.to-date $today --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | .rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true set access-rule uid  " + (.uid|tostring) + " enabled false layer")' >> $POL2-tmp.txt

done

elif [ "$SECHEAD" = "n" ]; then

printf "\nCreating Deletion Scripts. This may take a minute depending on Rulebase size.\n"

for I in $(seq 0 500 $total)

do

 mgmt_cli -r true show access-rulebase name "$POL_NAME" details-level "standard" offset $I limit 500 use-object-dictionary true show-hits true hits-settings.from-date $from hits-settings.to-date $today --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true set access-rule uid " + (.uid|tostring) + " enabled false layer")' >> $POL2-tmp.txt

done

fi

sed "s,$, '$POL_NAME' comments ‘disabled by API Zero Hit’," $POL2-tmp.txt > $POL2-delete-unused.txt; rm *tmp.txt

printf "\nDelete commands for zero hit count rules are now located in $POL2-delete-unused.txt\n"

[Adam_Forester]

Watch out for weird returns when you copy and paste. 🙂 cheers!

[Luciano_Miguel]

Adam Thank you , I will test .

[Fabrizio_De_Car]

Thank you. Great work!

[Simon_Garay]

Hi Adam,

Could you explain me this syntax? 

"mgmt_cli -r true set access-rule uid " + (.uid|tostring) + " enabled false layer"

Is this disable the rule?

Thank you for your support. 

[Adam_Forester]

Simon,

thanks for the message. Yes the "enabled false" is to disable. the rest is built around building a full command with jq

[Amiad_Stern · Accepted solution]

Just want to share a repository with similar functionality GitHub - CheckPointSW/PolicyCleanUp   

[mmethw2003]

Thanks for the share ....

Just to clarify is this code only for R80+ ?

[Ram1]

Hi Adam/Team

Good day, I need few clarifications and help from you.

1) to get full file bases which are disabled using a script.

2) how alert rules that are going to expire through email.

3) how to get the details of expired rule bases

Please let me know what areathe options to accomplish above tasks

Regards

Ram

[Jin_Zhou]

I am not sure if the thread is too old. But I am running into thesame issue that select(.hits.value == 0) returns nothing while it actually supposed to have 31 rules with zero hit count. Here is my situation. The following command returns 31,

mgmt_cli -r true show access-rulebase name "Internet Network" show-hits true use-object-dictionary true limit 50 offset 0 -d Internet -f json | jq -r '.rulebase[] ' |grep "\"value\": 0" | wc –l

while this command returns nothing.

mgmt_cli -r true show access-rulebase name "Internet Network" show-hits true use-object-dictionary true limit 50 -d Internet -f json | jq -r '.rulebase[] | select(.hits.value == 0) '

Any idea?