Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Adam_Forester
Ambassador
Ambassador
Jump to solution

Disable/Delete Rules with a Zero Hit Count (MDS or SMS)

**v3 and above now allows you to pick a specific access layer** 

**v4 added new functions thanks to user feedback. Now has the ability to navigate around section title headers and to handle of any size**

**v5 with a lot of work by Vincent Bacher‌ he determined that some larger policies need a time specified to search. This version added in a 6 month limit on hits prior to the day you run it (Today - 6Months.)**

** v6 combined MDS & SMS into a single script. Added the ability to disable or delete rules based on UID or NAME. The disable script will add a commend 'Disabled by Zero Hits'

This is a simple shell script that will allow you to parse a specific rulebase for rules with a ZERO hit count. The results will be output into a single file of mgmt_cli commands to disable or delete those rules.

The script is setup to run on the Mgmt station itself and uses the 'mgmt_cli -r true' function and uses the -d DOMAIN flag to support SMS and MDS in a single script

It is highly recommended to run the 'DISABLE' version prior to running a 'DELETE' it will treat it as a staging for full deletion

How to Use

  • Move script to the management station
  • ./cleanup-zero-hits.sh
  • Enter IP address of SMS or CMA you wish to check
  • Follow remaining prompts for options
    • uid or name
      • The script will ask if you want to export with uid or name. UID is more accurate as it does not change with position. This will prevent a situation where another admin is adding/removing rules from the rulebase before you are able to run the output file.

You can take the delete/disable command file and run it.

  • chmod 755 Output-Filename.txt
  • ./Output-Filename.txt

Original files on github: GitHub - cpmidsouth/Delete-or-Disable-Zero-Hit-Rules: This script is designed to search a specifed r... 

NOTE: If you use inline layers within the rulebase you will need to search those as a separate layer. This script is not effective in a rulebase where multiple targets within the same rulebase. I am working on that one. Thanks to Vincent Bacher‌ for being my QA and spending way too much time testing with me. 

Feedback welcome this was a simple project that came out of a client request.

2 Solutions

Accepted Solutions
Adam_Forester
Ambassador
Ambassador

Tomer thanks for the input. I did write it specifically for an account that had upgraded, I didn't take into account the creation of policies in 80.10. I will work on that specific for the next version.

Agree on the safeness of UID, I chose rule number as a way for the customer to manually review before letting the API do the deleting, call it controlled paranoia. But I did tweak a couple things this morning to use UID tested it and it works just fine, I think I will upload that as a second option.

mgmt_cli -r true show access-rulebase name "$POL_NAME Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true delete access-rule uid " + (.uid|tostring) + " layer")' > $POL_NAME-tmp.txt; sed "s,$, '$POL_NAME Network'," $POL_NAME-tmp.txt > $POL_NAME-delete-unused-uid.txt; rm *tmp.txt

View solution in original post

Amiad_Stern

Just want to share a repository with similar functionality GitHub - CheckPointSW/PolicyCleanUp   

View solution in original post

37 Replies
Tomer_Sole
Mentor
Mentor

Very nice, and from field knowledge I know it's useful 

And I like how outside of input/output text it's essentially one line of code. So that's kind of pretty on the inside Smiley Happy

Few comments in case you are interested..

- The fact that you added "[PolicyName] + Network" means it will only work for Firewall-only users who upgraded their policies. With R80.10, users can create multiple layers in the policy, and then the names of those layers are determined by the users' choice. Also, users with FW+APPI who upgrade their policies will have a policy with 2 ordered layer: "[PolicyName] Network" and "[PolicyName] Applications" (and they can rename those layers as they wish)

- Deleting rules based on their number can be problematic in case someone has just published new rules above them. I'm not sure how common this scenario, but when we designed R80 we had multiple users in mind. So taking the "uid" of the rule rather than "rule-number" is safer.

Adam_Forester
Ambassador
Ambassador

Tomer thanks for the input. I did write it specifically for an account that had upgraded, I didn't take into account the creation of policies in 80.10. I will work on that specific for the next version.

Agree on the safeness of UID, I chose rule number as a way for the customer to manually review before letting the API do the deleting, call it controlled paranoia. But I did tweak a couple things this morning to use UID tested it and it works just fine, I think I will upload that as a second option.

mgmt_cli -r true show access-rulebase name "$POL_NAME Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true delete access-rule uid " + (.uid|tostring) + " layer")' > $POL_NAME-tmp.txt; sed "s,$, '$POL_NAME Network'," $POL_NAME-tmp.txt > $POL_NAME-delete-unused-uid.txt; rm *tmp.txt

Ram1
Participant

Hi Adam,

 

Good day, I need few clarifications and help from you.

1) to get full file bases which are disabled using a script.

2) how alert rules that are going to expire through email.

3) how to get the details of expired rule bases

 

Please let me know if any clarifications.

Regards

Ram

0 Kudos
Tomer_Sole
Mentor
Mentor

Awesome. Also I'm glad that you managed to workaround a customer's request by yourself. Imagine what you had to do if we didn't have this Smiley Happy

Adam_Forester
Ambassador
Ambassador

Tomer, I've now updated it to list out the Access Layers and allow the users to pick instead of just assuming the addition of Network. This should allow it to work for all packages upgraded or not.

Jill_Sanders
Participant

Great job!

Vincent_Bacher
Advisor
Advisor

Hi Adam,

nice useful script, good job.

One question:

I have just tested the script but unfortunately get no result, that means the utput file is empty.

Started the script:

./delete-zero-hits-rulenumber-v3.sh
This script will search a specific policy package for rules with a ZERO hit count.
Use with caution for deleting rules..
If for any reason you make a typo and need to exit use CTRL+C.
Press ENTER  to continue
Listing Access Policy Package Names.
Applications
************-FW-POLICY Network
What is the Policy Package Name?

I tried the policy Name with or without "Network"

************-FW-POLICY Network

************-FW-POLICY

With or without quotation marks. Output file is always empty. Hopefully i find some time to have a look into the script and at the mgmt_cli commands.
Cheers

Vincent

and now to something completely different - CCVS, CCAS, CCTE, CCCS, CCSM elite
0 Kudos
Adam_Forester
Ambassador
Ambassador

Interesting Vincent. Is it possible you don't have rules with a zero hit count? This is tested on 80.10.

I'm going to have you run the raw command to look for the output;

mgmt_cli -r true show access-rulebase name "************-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[] | select(.hits.value == 0)'

See what that outputs. If it's nothing then run this;

mgmt_cli -r true show access-rulebase name "************-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[]'

Then look for the output of each rule and the hits value;  which should look like the below (I formatted the text you would look for)

"rule-number": 1,

  "track": {

    "type": "598ead32-aa42-4615-90ed-f51a5928d41d",

    "per-session": false,

    "per-connection": true,

    "accounting": false,

    "alert": "none"

  },

  "source": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "source-negate": false,

  "destination": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "destination-negate": false,

  "service": [

    "97aeb470-9aea-11d5-bd16-0090272ccb30"

  ],

  "service-negate": false,

  "vpn": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "action": "6c488338-8eec-4103-ad21-cd461ac2c472",

  "action-settings": {

    "enable-identity-captive-portal": false

  },

  "content": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "content-negate": false,

  "content-direction": "any",

  "time": [

    "97aeb369-9aea-11d5-bd16-0090272ccb30"

  ],

  "hits": {

    "percentage": "1%",

    "level": "low",

    "value": 24946,

    "first-date": {

      "posix": 1518945364000,

      "iso-8601": "2018-02-18T03:16-0600"

    },

    "last-date": {

      "posix": 1527527490000,

      "iso-8601": "2018-05-28T12:11-0500"

    }

  },

  "custom-fields": {

    "field-1": "",

    "field-2": "",

    "field-3": ""

  },

  "meta-info": {

    "lock": "unlocked",

    "validation-state": "ok",

    "last-modify-time": {

      "posix": 1497028032827,

      "iso-8601": "2017-06-09T12:07-0500"

    },

    "last-modifier": "admin",

    "creation-time": {

      "posix": 1497028010878,

      "iso-8601": "2017-06-09T12:06-0500"

    },

    "creator": "admin"

  },

  "comments": "",

  "enabled": true,

  "install-on": [

    "6c488338-8eec-4103-ad21-cd461ac2c476"

  ]

}

{

  "uid": "b11cd15c-f55b-4450-8f96-09ccda45f1bc",

  "type": "access-rule",

  "domain": {

    "uid": "41e821a0-3720-11e3-aa6e-0800200c9fde",

    "name": "SMC User",

    "domain-type": "domain"

  },

Vincent_Bacher
Advisor
Advisor

Hi Adam, 

in fact I have rules with zero hitcount. Did not have time yet to try running the raw commands, will do that tomorrow.

Cheers 

Vincent 

and now to something completely different - CCVS, CCAS, CCTE, CCCS, CCSM elite
0 Kudos
Adam_Forester
Ambassador
Ambassador

Sounds good. I’ll be interested to see. I’m going to send you my check Point email as well so we can work offline and report back

0 Kudos
Adam_Forester
Ambassador
Ambassador

I had another thought... run the UID version of the script and check it’s output as well. 

0 Kudos
Vincent_Bacher
Advisor
Advisor

Did not receive your mail 😉

Just ran the commands you suggested manually:

# mgmt_cli -r true show access-rulebase name "*******-INT-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[] | select(.hits.value == 0)'

--> nothing

running:

mgmt_cli -r true show access-rulebase name "********-INT-FW-POLICY Network" details-level "standard" use-object-dictionary true show-hits true --format json | jq --raw-output '.rulebase[]'

Shows a list of round about 50 no-hits rules. Should be correct amount.

So question is why select(.hits.value == 0)' has no match Smiley Happy

and now to something completely different - CCVS, CCAS, CCTE, CCCS, CCSM elite
0 Kudos
Vincent_Bacher
Advisor
Advisor

Empty as well.

and now to something completely different - CCVS, CCAS, CCTE, CCCS, CCSM elite
0 Kudos
Adam_Forester
Ambassador
Ambassador

Wanted to drop a note here as well for anyone following along. Vincent's testing helped me determine a few things that I did not test in my lab;

1. The original script was not setup to handle section title headers. *RESOLVED*

2. I was unaware of a the default limit of "50" being used. This was causing the script to only search the first 50 rules of a rulebase. This was not an issue in my lab as I was only testing with 25. I've since expanded the script to determine the rule count and to loop until it searches all rules. *RESOLVED*

3. The script does not take into account a rulebase that has multiple targets in it. *WORKING TOWARDS RESOLUTION*

Tomer Sole

Tomer_Sole
Mentor
Mentor

if you want to be extra awesome you can try multi-domain policies...

0 Kudos
Adam_Forester
Ambassador
Ambassador

Like this? https://community.checkpoint.com/docs/DOC-2813 

Now im still working on updating that one to take the limit into account. Should be done tonight or tomorrow. 

0 Kudos
Robert_Decker
Advisor

Hi Adam,

please take a look at our Management API Python SDK on GitHub - 

https://github.com/CheckPointSW/cp_mgmt_api_python_sdk/blob/master/lib/mgmt_api.py

and go the the method called "api_query" on line 318.

this method shows how to query ALL objects, bypassing 50 objects limit.

Robert.

Luciano_Miguel
Contributor

Hello

Its possible change the action of this script from DELETE HitCount zero Rule to DISABLE RULE ? And Put one comment to identification of rule after script execution ?

Thank you 

0 Kudos
Adam_Forester
Ambassador
Ambassador

Thanks Robert. I’ll be honest I do t speak python all that well, I’m an old basher. 🙂 but I do need to expand and learn it. 

I was able to achieve the limit in bash by doing a total then setting my limit to 500 but looping the offset in a seq with it incrementing by 500 until it hits $total. 

Adam_Forester
Ambassador
Ambassador

Luciano Miguel‌ sure can. Sorry for posting it here fully but I’m traveling and did this from my cell phone and couldn’t upload a file. Was able to test on my sms and confirmed it worked out just fine. Just copy and paste the below. It will make it where you can disable the rules and adds a Comment that says “disabled by API Zero Hits

today="$(date +%Y-%m-%d)"

from="$(date --date="6 months ago" +%Y-%m-%d)"

timestamp="$(date +%Y-%m-%d-%H-%M-%S)"

printf  "This script will search a specific policy package for rules with a ZERO hit count.\nUse with caution for deleting rules..\nIf for any reason you make a typo and need to exit use CTRL+C.\nPress ENTER  to continue"

read ANYKEY

printf "\nListing Access Policy Package Names\n"

mgmt_cli -r true show access-layers limit 500 --format json | jq --raw-output '."access-layers"[] | (.name)'

printf "\nWhat is the Policy Package Name?\n"

read POL_NAME

POL2=$(echo $POL_NAME | tr -d ' ')

printf "\nDetermining Rulesbase Size\n"

total=$(mgmt_cli -r true show access-rulebase name "$POL_NAME" --format json |jq '.total')

printf "There are $total rules in $POL_NAME\n"

printf "\nDoes Your Policy Contain Section Title Headers?[y/n]\n"

read SECHEAD

if [ "$SECHEAD" = "y" ]; then

printf "\nCreating Deletion Scripts. This may take a minute depending on Rulebase size.\n"

for I in $(seq 0 500 $total)

do

 mgmt_cli -r true show access-rulebase name "$POL_NAME" details-level "standard" offset $I limit 500 use-object-dictionary true show-hits true hits-settings.from-date $from hits-settings.to-date $today --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | .rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true set access-rule uid  " + (.uid|tostring) + " enabled false layer")' >> $POL2-tmp.txt

done

elif [ "$SECHEAD" = "n" ]; then

printf "\nCreating Deletion Scripts. This may take a minute depending on Rulebase size.\n"

for I in $(seq 0 500 $total)

do

 mgmt_cli -r true show access-rulebase name "$POL_NAME" details-level "standard" offset $I limit 500 use-object-dictionary true show-hits true hits-settings.from-date $from hits-settings.to-date $today --format json | jq --raw-output --arg RBN "$POL_NAME" '.rulebase[] | select(.hits.value == 0) | ("mgmt_cli -r true set access-rule uid " + (.uid|tostring) + " enabled false layer")' >> $POL2-tmp.txt

done

fi

sed "s,$, '$POL_NAME' comments ‘disabled by API Zero Hit’," $POL2-tmp.txt > $POL2-delete-unused.txt; rm *tmp.txt

printf "\nDelete commands for zero hit count rules are now located in $POL2-delete-unused.txt\n"

Adam_Forester
Ambassador
Ambassador

Watch out for weird returns when you copy and paste. 🙂 cheers!

0 Kudos
Luciano_Miguel
Contributor

Adam Thank you , I will test .

0 Kudos
Fabrizio_De_Car
Explorer

Thank you. Great work!

Simon_Garay
Contributor

Hi Adam,

Could you explain me this syntax? 

"mgmt_cli -r true set access-rule uid " + (.uid|tostring) + " enabled false layer"

Is this disable the rule?

Thank you for your support. 

0 Kudos
Adam_Forester
Ambassador
Ambassador

Simon,

thanks for the message. Yes the "enabled false" is to disable. the rest is built around building a full command with jq

Amiad_Stern

Just want to share a repository with similar functionality GitHub - CheckPointSW/PolicyCleanUp   

mmethw2003
Explorer

Thanks for the share ....

Just to clarify is this code only for R80+ ?

0 Kudos
Ram1
Participant

Hi Adam/Team

 

Good day, I need few clarifications and help from you.

1) to get full file bases which are disabled using a script.

2) how alert rules that are going to expire through email.

3) how to get the details of expired rule bases

 

Please let me know what areathe options to accomplish above tasks

Regards

Ram

0 Kudos
Jin_Zhou
Contributor

I am not sure if the thread is too old. But I am running into thesame issue that select(.hits.value == 0) returns nothing while it actually supposed to have 31 rules with zero hit count. Here is my situation. The following command returns 31,

mgmt_cli -r true show access-rulebase name "Internet Network" show-hits true use-object-dictionary true limit 50 offset 0 -d Internet -f json | jq -r '.rulebase[] ' |grep "\"value\": 0" | wc –l

while this command returns nothing.

mgmt_cli -r true show access-rulebase name "Internet Network" show-hits true use-object-dictionary true limit 50 -d Internet -f json | jq -r '.rulebase[] | select(.hits.value == 0) '

Any idea?

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events