Export of rules with zero hits in dashboard

[the_rock]

 Hi everyone,

I saw some posts about this before, but there was never a confirmation if this was ever available. I am trying to export rules in excel format for a customer that requested list of rules with zero hits, but does not seem its possible. I exported all the rules and can filter for example for any disabled rules, but I dont see column anywhere in excel file for hits, though hits column is enabled in dashboard.

Any idea if this is possible in R81 at all? It is cloud mgmt, but I dont think that makes any difference. 

Also, another thing I noticed, though this could be pure cosmetic is that all 100 some NAT rules show zero hits, which also makes no sense, since we know bunch of them are getting hit for sure.

Thanks in advance!

[Bob_Zimmerman]

The level of possibility depends on how much effort you're willing to expend. 😉

It's relatively easy to handle sectionless rules (that is, rules above any section header) and rules within sections. It's a lot harder to handle inline layers, as you have to run a separate API call to get their contents. Try this:

mgmt_cli -r true \
--format json \
show access-rulebase \
uid "<UUID>" \
show-hits true \
use-object-dictionary false \
| jq -c '.rulebase[]|if .rulebase then {section:.name,rule:.rulebase[]|{name:.name,hits:.hits.value}} else {name:.name,hits:.hits.value} end'

The 'if .rulebase then ... else ... end' structure gives you separate output for items which have a rulebase (read: rule sections) and objects which don't (sectionless rules). For one of the access layers on my development box, it returns this:

{"name":"Sectionless","hits":0}
{"name":"Bad browsing","hits":0}
{"section":"WebApp-1","rule":{"name":"Internet access in","hits":0}}
{"section":"WebApp-1","rule":{"name":"Web to App","hits":0}}
{"section":"WebApp-1","rule":{"name":"App to DB","hits":0}}
{"section":"WebApp-1","rule":{"name":"Admin access","hits":0}}
{"section":"Some Other Web App","rule":{"name":"Internet access in","hits":0}}
{"section":"Some Other Web App","rule":{"name":"Web to App","hits":0}}
{"section":"Some Other Web App","rule":{"name":"App to DB","hits":0}}
{"section":"Some Other Web App","rule":{"name":"Admin access","hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Cleanup","rule":{"name":"Cleanup rule","hits":0}}

You can then grep for "hits":0 and get the rules you're interested in. The jq filter should be relatively easy to expand to cover whatever fields you want.

[the_rock]

I hear ya, but I wanted to do this via dashboard, not cli, as its cloud server, so there is no ssh available to us, only from backend.

[Bob_Zimmerman]

You should still be able to make API calls. mgmt_cli even has a way to log in to a remote management server (though the '-r true' won't work; you'll need to provide credentials).

[PhoneBoy]

This was in SmartDashboard back in the day but had not been added to R8x SmartConsole.
I don’t know what the precise plan is to add it back.
Current workaround is to use the API and there are several examples in the community on how to do this.

[the_rock]

I tried running below, but no luck, command did work, but I cant see anything that shows hits values at all...

mgmt_cli show access-rulebase offset 0 limit 20 name "Network" details-level "standard" use-object-dictionary true show-hits true hits-settings.from-date "2021-04-30" hits-settings.to-date "2021-05-30"

[Bob_Zimmerman]

Without the 'hits-settings' part, I get this on R80.40 jumbo 114:

uid: "b406b732-2437-4848-9741-6eae1f5bf112"
name: "Network"
rulebase: 
- uid: "f8206d78-7c5f-4995-b519-865e2bf0730c"
  name: "Sectionless"
  type: "access-rule"
  domain: 
    uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
    name: "SMC User"
    domain-type: "domain"
  rule-number: 1
  track: 
    type: "29e53e3d-23bf-48fe-b6b1-d59bd88036f9"
    per-session: false
    per-connection: false
    accounting: false
    enable-firewall-session: false
    alert: "none"
  source: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  source-negate: false
  destination: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  destination-negate: false
  service: 
  - "97aeb405-9aea-11d5-bd16-0090272ccb30"
  service-negate: false
  vpn: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  action: "6c488338-8eec-4103-ad21-cd461ac2c472"
  action-settings: 
    enable-identity-captive-portal: false
  content: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  content-negate: false
  content-direction: "any"
  time: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  hits: 
    percentage: "0%"
    level: "zero"
    value: 0
  custom-fields: 
    field-1: ""
    field-2: ""
    field-3: ""
  meta-info: 
...

Do you get the hits section if you leave the 'hits-settings' off?

[the_rock]

Tried that, but no luck...TAC said they will run web visualization script on backend to see if they can parse it that way, but sounds like a lot of work though. I really wanted to give customer excel spreadsheet with zero hits rules via dashboard, but definitely does not appear it is possible.

[Hugo_vd_Kooij]

I expanded this to a sript that will get all policies and create a hitcounter CSV file.

#!/bin/bash

# Show Hitcounters for all policies

NOW=`/bin/date +%Y%m%d`
PACKAGES=`mgmt_cli -r true --port 443 show packages --format json | jq '.packages[] | .name' | sed 's/\"//g'`

for POLICY in $PACKAGES
do
        echo "Hitcounters for $POLICY"
        mgmt_cli -r true --port 443 show access-rulebase name "$POLICY Security" show-hits true --format json limit 50000 \
                | jq  '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output \
                | sed 's/\[//g'| \
                sed 's/\]//g' > HitCount-$POLICY-Security-$NOW.csv
done

The assumption is that you have "default" policy names with " Security" added to the name of the policy package.

[Bob_Zimmerman]

Have you tested this? Pretty sure limit only goes up to 500.

[Paul_Gademsky]

Hi Hugo, just found this in searching for a way to export hit counts on pre R81.10.

When I run this, I get the following:

[Expert@SNS-MGMT:0]# ./hitcounter.sh
++ /bin/date +%Y%m%d
+ NOW=20220603
++ mgmt_cli -r true --port 443 show packages --format json
++ jq '.packages[] | .name'
++ sed 's/\"//g'
+ PACKAGES='New-SDC-consolidated
sdc-test
Standard
Standard_multi-layered
test
Unger-SDC-consolidated'
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for New-SDC-consolidated'
Hitcounters for New-SDC-consolidated
+ mgmt_cli -r true --port 443 show access-rulebase name 'New-SDC-consolidated Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for sdc-test'
Hitcounters for sdc-test
+ mgmt_cli -r true --port 443 show access-rulebase name 'sdc-test Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for Standard'
Hitcounters for Standard
+ mgmt_cli -r true --port 443 show access-rulebase name 'Standard Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for Standard_multi-layered'
Hitcounters for Standard_multi-layered
+ mgmt_cli -r true --port 443 show access-rulebase name 'Standard_multi-layered Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for test'
Hitcounters for test
+ mgmt_cli -r true --port 443 show access-rulebase name 'test Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for Unger-SDC-consolidated'
Hitcounters for Unger-SDC-consolidated
+ mgmt_cli -r true --port 443 show access-rulebase name 'Unger-SDC-consolidated Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
[Expert@SNS-MGMT:0]# ls -hal *.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-New-SDC-consolidated-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-Standard-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-Standard_multi-layered-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-Unger-SDC-consolidated-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-sdc-test-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-test-Security-20220603.csv

I'm assuming it has to do with the naming of the policies, but thought I'd ask before I chase my tail.

Thank you for any input.

Paul G.  CCSM

[Tomer_Noy · Accepted solution]

Following feedback from the field (including this post 😀), we're adding hitcount information to the csv export of the rulebase. This will of course be accessible without API scripting or expert privileges on the machine.

This enhancement will be included in the upcoming R81.10 and we are also looking into porting it back to earlier versions via JHF.

Here is a snippet of how it will look:

[_Val_]

Thanks @Tomer_Noy , we appreciate your prompt action here.

[Tomer_Noy]

Glad to assist 😀

Credit for fast implementation goes to @Alon_Alapi and @Youssef_Obeidal.

[the_rock]

Good to hear! 

[Ganesan]

Any possibility to get in SmartDashboard the hitcount extracting via csv? Any other option without API ?

[PhoneBoy]

Answered in an earlier post in this thread: https://community.checkpoint.com/t5/Management/Export-of-rules-with-zero-hits-in-dashboard/m-p/12055... 

[the_rock]

Sorry to respond to this few months later, but just for my own reference, will this work say if management is R81.10 and gateways are, for example, on R80.xx versions, or everything needs to be on R81.10 code?

[Tomer_Noy]

Yes, it will work when just the Management is R81.10.

The feature is implemented on the Management side, so gateway upgrades are not needed to get it. It relies on the same data that you see in the SmartConsole UI.

[Tomer_Noy]

BTW, you can also get this new feature on R81 with the latest JHF on the Management server + latest SmartConsole build.

The team ported it back to that version as well.

Of course, upgrading to R81.10 is still a great option 😀

[the_rock]

So I tried it on R81 mgmt managing R81 gateways (all jumbo 36) and latest console build 553 and when I export the rules, I see in the csv file there is hits column, but no numbers there at all, though in policy I see bunch of hits on every rule. Any idea?

[Tomer_Noy · Accepted solution]

You need R81 JHF 42, which is still ongoing.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 

[the_rock]

K, fair enough, le me try that and I will update the forum : )

[the_rock]

I like that a lot Tomer. Tried it in 2 setups and when I export the rules, what I really love about it is that hits column gives EXACT number of hits. So say if dashboard shows 5k as number of hits, then csv file would show say 5785 number, which is great. One more question if you dont mind...cant recall now. Does number of hits get reset only if you do reboot of the firewalls or does it happen in any other instance?

Thanks again!

[nzmatto1]

I was just searching for this and found it thanks to this thread. Exporting the rulebase to csv worked perfectly for me. I then tried the save on the NATs page, however that does not export the hitcount, so perhaps that's something which could be added. 
Also it would be great if there was a functionality to select a date range for the hit counts, so we can tell if a rule hasn't been hit in say >6 months. 

[the_rock]

R81+ codes came a long way when it comes to rule export/hit count, our customers are very happy with it.

Andy

[Eric_Smith]

Afternoon Tomer,

Is there going to be any attempt to also include First Hit and Last Hit to this export table?  I have numerous customers asking for this. It seems the data is there, when you hover over the hit count it is displayed, but why is it not in the export?

[the_rock]

Definitely not there in R81.20, tested in my lab.

[Youssef_Obeidal]

Hi,

We will look into it for the next version and try to backport to JHF of recent versions.

[Eric_Smith · Accepted solution]

I have a lot of customers who will be very happy to see and hear that. Thank you Youssef.