Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
the_rock
Legend
Legend
Jump to solution

Export of rules with zero hits in dashboard

 Hi everyone,

 

I saw some posts about this before, but there was never a confirmation if this was ever available. I am trying to export rules in excel format for a customer that requested list of rules with zero hits, but does not seem its possible. I exported all the rules and can filter for example for any disabled rules, but I dont see column anywhere in excel file for hits, though hits column is enabled in dashboard.

 

Any idea if this is possible in R81 at all? It is cloud mgmt, but I dont think that makes any difference. 

Also, another thing I noticed, though this could be pure cosmetic is that all 100 some NAT rules show zero hits, which also makes no sense, since we know bunch of them are getting hit for sure.

 

Thanks in advance!

0 Kudos
3 Solutions

Accepted Solutions
Tomer_Noy
Employee
Employee

Following feedback from the field (including this post 😀), we're adding hitcount information to the csv export of the rulebase. This will of course be accessible without API scripting or expert privileges on the machine.

This enhancement will be included in the upcoming R81.10 and we are also looking into porting it back to earlier versions via JHF.

Here is a snippet of how it will look:

Tomer_Noy_0-1623105315090.png

 

View solution in original post

Eric_Smith
Participant

I have a lot of customers who will be very happy to see and hear that. Thank you Youssef.

View solution in original post

40 Replies
Bob_Zimmerman
Authority
Authority

The level of possibility depends on how much effort you're willing to expend. 😉

It's relatively easy to handle sectionless rules (that is, rules above any section header) and rules within sections. It's a lot harder to handle inline layers, as you have to run a separate API call to get their contents. Try this:

 

mgmt_cli -r true \
--format json \
show access-rulebase \
uid "<UUID>" \
show-hits true \
use-object-dictionary false \
| jq -c '.rulebase[]|if .rulebase then {section:.name,rule:.rulebase[]|{name:.name,hits:.hits.value}} else {name:.name,hits:.hits.value} end'

 

The 'if .rulebase then ... else ... end' structure gives you separate output for items which have a rulebase (read: rule sections) and objects which don't (sectionless rules). For one of the access layers on my development box, it returns this:

 

{"name":"Sectionless","hits":0}
{"name":"Bad browsing","hits":0}
{"section":"WebApp-1","rule":{"name":"Internet access in","hits":0}}
{"section":"WebApp-1","rule":{"name":"Web to App","hits":0}}
{"section":"WebApp-1","rule":{"name":"App to DB","hits":0}}
{"section":"WebApp-1","rule":{"name":"Admin access","hits":0}}
{"section":"Some Other Web App","rule":{"name":"Internet access in","hits":0}}
{"section":"Some Other Web App","rule":{"name":"Web to App","hits":0}}
{"section":"Some Other Web App","rule":{"name":"App to DB","hits":0}}
{"section":"Some Other Web App","rule":{"name":"Admin access","hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Access to Public Services","rule":{"name":null,"hits":0}}
{"section":"Cleanup","rule":{"name":"Cleanup rule","hits":0}}

 

You can then grep for "hits":0 and get the rules you're interested in. The jq filter should be relatively easy to expand to cover whatever fields you want.

the_rock
Legend
Legend

I hear ya, but I wanted to do this via dashboard, not cli, as its cloud server, so there is no ssh available to us, only from backend.

0 Kudos
Bob_Zimmerman
Authority
Authority

You should still be able to make API calls. mgmt_cli even has a way to log in to a remote management server (though the '-r true' won't work; you'll need to provide credentials).

0 Kudos
PhoneBoy
Admin
Admin

This was in SmartDashboard back in the day but had not been added to R8x SmartConsole.
I don’t know what the precise plan is to add it back.
Current workaround is to use the API and there are several examples in the community on how to do this.

0 Kudos
the_rock
Legend
Legend

I tried running below, but no luck, command did work, but I cant see anything that shows hits values at all...

 

mgmt_cli show access-rulebase offset 0 limit 20 name "Network" details-level "standard" use-object-dictionary true show-hits true hits-settings.from-date "2021-04-30" hits-settings.to-date "2021-05-30"

0 Kudos
Bob_Zimmerman
Authority
Authority

Without the 'hits-settings' part, I get this on R80.40 jumbo 114:

uid: "b406b732-2437-4848-9741-6eae1f5bf112"
name: "Network"
rulebase: 
- uid: "f8206d78-7c5f-4995-b519-865e2bf0730c"
  name: "Sectionless"
  type: "access-rule"
  domain: 
    uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
    name: "SMC User"
    domain-type: "domain"
  rule-number: 1
  track: 
    type: "29e53e3d-23bf-48fe-b6b1-d59bd88036f9"
    per-session: false
    per-connection: false
    accounting: false
    enable-firewall-session: false
    alert: "none"
  source: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  source-negate: false
  destination: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  destination-negate: false
  service: 
  - "97aeb405-9aea-11d5-bd16-0090272ccb30"
  service-negate: false
  vpn: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  action: "6c488338-8eec-4103-ad21-cd461ac2c472"
  action-settings: 
    enable-identity-captive-portal: false
  content: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  content-negate: false
  content-direction: "any"
  time: 
  - "97aeb369-9aea-11d5-bd16-0090272ccb30"
  hits: 
    percentage: "0%"
    level: "zero"
    value: 0
  custom-fields: 
    field-1: ""
    field-2: ""
    field-3: ""
  meta-info: 
...

Do you get the hits section if you leave the 'hits-settings' off?

0 Kudos
the_rock
Legend
Legend

Tried that, but no luck...TAC said they will run web visualization script on backend to see if they can parse it that way, but sounds like a lot of work though. I really wanted to give customer excel spreadsheet with zero hits rules via dashboard, but definitely does not appear it is possible.

0 Kudos
Hugo_vd_Kooij
Advisor

I expanded this to a sript that will get all policies and create a hitcounter CSV file.

 

#!/bin/bash

# Show Hitcounters for all policies

NOW=`/bin/date +%Y%m%d`
PACKAGES=`mgmt_cli -r true --port 443 show packages --format json | jq '.packages[] | .name' | sed 's/\"//g'`

for POLICY in $PACKAGES
do
        echo "Hitcounters for $POLICY"
        mgmt_cli -r true --port 443 show access-rulebase name "$POLICY Security" show-hits true --format json limit 50000 \
                | jq  '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output \
                | sed 's/\[//g'| \
                sed 's/\]//g' > HitCount-$POLICY-Security-$NOW.csv
done

The assumption is that you have "default" policy names with " Security" added to the name of the policy package.

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Bob_Zimmerman
Authority
Authority

Have you tested this? Pretty sure limit only goes up to 500.

0 Kudos
Paul_Gademsky
Employee Employee
Employee

Hi Hugo, just found this in searching for a way to export hit counts on pre R81.10.

When I run this, I get the following:

[Expert@SNS-MGMT:0]# ./hitcounter.sh
++ /bin/date +%Y%m%d
+ NOW=20220603
++ mgmt_cli -r true --port 443 show packages --format json
++ jq '.packages[] | .name'
++ sed 's/\"//g'
+ PACKAGES='New-SDC-consolidated
sdc-test
Standard
Standard_multi-layered
test
Unger-SDC-consolidated'
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for New-SDC-consolidated'
Hitcounters for New-SDC-consolidated
+ mgmt_cli -r true --port 443 show access-rulebase name 'New-SDC-consolidated Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for sdc-test'
Hitcounters for sdc-test
+ mgmt_cli -r true --port 443 show access-rulebase name 'sdc-test Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for Standard'
Hitcounters for Standard
+ mgmt_cli -r true --port 443 show access-rulebase name 'Standard Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for Standard_multi-layered'
Hitcounters for Standard_multi-layered
+ mgmt_cli -r true --port 443 show access-rulebase name 'Standard_multi-layered Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for test'
Hitcounters for test
+ mgmt_cli -r true --port 443 show access-rulebase name 'test Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
+ for POLICY in $PACKAGES
+ echo 'Hitcounters for Unger-SDC-consolidated'
Hitcounters for Unger-SDC-consolidated
+ mgmt_cli -r true --port 443 show access-rulebase name 'Unger-SDC-consolidated Security' show-hits true --format json limit 50000
+ jq '.rulebase[] | .rulebase[] | [."rule-number", .name, .hits.value]' --compact-output
+ sed 's/\[//g'
+ sed 's/\]//g'
jq: error: Cannot iterate over null
[Expert@SNS-MGMT:0]# ls -hal *.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-New-SDC-consolidated-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-Standard-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-Standard_multi-layered-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-Unger-SDC-consolidated-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-sdc-test-Security-20220603.csv
-rw-rw---- 1 admin users 0 Jun 3 10:07 HitCount-test-Security-20220603.csv

I'm assuming it has to do with the naming of the policies, but thought I'd ask before I chase my tail.

Thank you for any input.

Paul G.  CCSM

0 Kudos
Tomer_Noy
Employee
Employee

Following feedback from the field (including this post 😀), we're adding hitcount information to the csv export of the rulebase. This will of course be accessible without API scripting or expert privileges on the machine.

This enhancement will be included in the upcoming R81.10 and we are also looking into porting it back to earlier versions via JHF.

Here is a snippet of how it will look:

Tomer_Noy_0-1623105315090.png

 

_Val_
Admin
Admin

Thanks @Tomer_Noy , we appreciate your prompt action here.

0 Kudos
Tomer_Noy
Employee
Employee

Glad to assist 😀

Credit for fast implementation goes to @Alon_Alapi and @Youssef_Obeidal.

the_rock
Legend
Legend

Good to hear! 

0 Kudos
Ganesan
Explorer

Any possibility to get in SmartDashboard the hitcount extracting via csv? Any other option without API ?

 

 

0 Kudos
PhoneBoy
Admin
Admin
0 Kudos
the_rock
Legend
Legend

Sorry to respond to this few months later, but just for my own reference, will this work say if management is R81.10 and gateways are, for example, on R80.xx versions, or everything needs to be on R81.10 code?

0 Kudos
Tomer_Noy
Employee
Employee

Yes, it will work when just the Management is R81.10.

The feature is implemented on the Management side, so gateway upgrades are not needed to get it. It relies on the same data that you see in the SmartConsole UI.

0 Kudos
Tomer_Noy
Employee
Employee

BTW, you can also get this new feature on R81 with the latest JHF on the Management server + latest SmartConsole build.

The team ported it back to that version as well.

Of course, upgrading to R81.10 is still a great option 😀

0 Kudos
the_rock
Legend
Legend

So I tried it on R81 mgmt managing R81 gateways (all jumbo 36) and latest console build 553 and when I export the rules, I see in the csv file there is hits column, but no numbers there at all, though in policy I see bunch of hits on every rule. Any idea?

0 Kudos
Tomer_Noy
Employee
Employee
the_rock
Legend
Legend

K, fair enough, le me try that and I will update the forum : )

0 Kudos
the_rock
Legend
Legend

I like that a lot Tomer. Tried it in 2 setups and when I export the rules, what I really love about it is that hits column gives EXACT number of hits. So say if dashboard shows 5k as number of hits, then csv file would show say 5785 number, which is great. One more question if you dont mind...cant recall now. Does number of hits get reset only if you do reboot of the firewalls or does it happen in any other instance?

 

Thanks again!

0 Kudos
nzmatto1
Contributor

I was just searching for this and found it thanks to this thread. Exporting the rulebase to csv worked perfectly for me. I then tried the save on the NATs page, however that does not export the hitcount, so perhaps that's something which could be added. 
Also it would be great if there was a functionality to select a date range for the hit counts, so we can tell if a rule hasn't been hit in say >6 months. 

 

the_rock
Legend
Legend

R81+ codes came a long way when it comes to rule export/hit count, our customers are very happy with it.

Andy

0 Kudos
Eric_Smith
Participant

Afternoon Tomer,

Is there going to be any attempt to also include First Hit and Last Hit to this export table?  I have numerous customers asking for this. It seems the data is there, when you hover over the hit count it is displayed, but why is it not in the export?

(1)
the_rock
Legend
Legend

Definitely not there in R81.20, tested in my lab.

Youssef_Obeidal
Employee
Employee

Hi,

We will look into it for the next version and try to backport to JHF of recent versions.

Eric_Smith
Participant

I have a lot of customers who will be very happy to see and hear that. Thank you Youssef.

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events