Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
CarstenWeber
Participant

How to add/create rules in bulk/batch using a .csv file via mgmt_cli API?

Jump to solution

Hi there, 

does anyone know a way to add/create R77.30 style access rules (no inline 'n stuff) in R80.30 using the API (v1.5) and a csv file listing multiple rules in a multi domain environment?
The following is what I got so far, but that is pure guesswork...how would this look right?

So this is me trying to add a number of access rules containing all the relevant data  (I don't need "time", "install on", etc.).
I am guessing the following after deriving bits and pieces all over the forum and internet:

 

mgmt_cli add access-rule -b /var/log/tmp/rules.csv --version 1.5

 

How would the correct csv file first row (header) look like? Something like this?:

 

package,layer,position.above,source,destination,service,track,comments

 

package --- would be the according policy name used in that domain
postion.above --- will be a ruleUID as value

Does anyone have  a working example of the above szenario? I can't be that far away from a working set of command and csv syntax.

Thanks a lot in advance.

 

Consider me already logged in to the according domain (Multi Domain Management) before executing anything else.

 

 

 

mgmt_cli login -d <domain_objid> > sid.txt.$$

 

 

 

$$ --- will add the session ID for a more or less unique file name. I know, I know, not ideal...but it serves the purpose allright in this case.
objid --- can be gathered using teh following command...at least on a R80.30 MDM MGMT server. A single CMA does not need this parameter (-d).

 

 

 

psql_client cpm postgres -c"select objid,name from domainbase_data where dlesession=0 and not deleted;"

 

 

 

Publish every 200-300 changes!

 

 

 

mgmt_cli publish -s sid.txt.$$

 

 

 

Logout when done

 

 

 

mgmt_cli logout -s sid.txt.$$

 

 

 

I also used the API to create all the objects like groups, hosts (adding them to groups already created as well in one go) and services (ports).

 

 

 

# Groups
mgmt_cli add group -b groups.csv -s sid.txt.$$
CSV header:
name,comments

# Hosts
mgmt_cli add host -b host.csv -s sid.txt.$$
CSV header:
name,ipv4_address,comments,groups

# Ports
mgmt_cli add service-tcp -b TCPports.csv
mgmt_cli add service-udp -b UDPports.csv
mgmt_cli add service-icmp -b ICMPports.csv
mgmt_cli add service-dce-rpc -b DCEports.csv
CSV header:
name,port,protocol,match-for-any,comments

 

 

 

The reason to do the above is a vast number of connections (50+ million times hit count) I gathered from a few firewall logs matching too wide rules (something close to "src:<RFC1918_ranges> dst:<RFC1918_ranges> svc:any action:allow"). In order to create more specific set of rules for traffic really happening and bleeding out the too wide rules eventually. 
I used an Excelsheet with loads of formulas I have never ever used before to prepare tabs for each of the required csv-files (groups, hosts, ports, rules) based on the reference data collected in the first tab (src, dst, port, protocol). The tabs are filled automatically ready for export as seperate csv-files. And here I am stuck on the rule part and how it has to look like. Please help

 

PS: All the links in other articles about adding objects and nat rules did not really mention anything else I need. I don't have to mention that the API reference does not explain bulk/batch usage. This is odd, as the API should help automate processes.

previously known as (pka.) Carsten_Weber
0 Kudos
1 Solution

Accepted Solutions
Stuart_Green1
Employee
Employee

Apologies (my fault for trying to multi-task). You're quite right, the link only covers the items you mentioned but you can use the same approach for access-rules too. 

 

Eg - if you have a blank policy you can use something like this:

[Expert@cpmgmt:0]# cat testrules.csv


layer,position,action,source,destination,service
Network,bottom,Accept,srcHost,dstHost,https
Network,bottom,Accept,srcHost,dstHost,icmp-requests
Network,bottom,Accept,srcHost,dstHost,http
Network,bottom,Accept,srcHost,dstHost,telnet
Network,bottom,Accept,srcHost,dstHost,ssh

 

Make sure you have the right name for your layer and src / destination hosts. The format of the CSV isn't fixed, but make sure that you have 'columns' in your CSV that match up to, at least, the required properties for the command you're running.

 

Then run  mgmt_cli -r true add access-rule --batch testrules.csv

 

and you'll get something like this:

 

2022-01-25 10_09_50-20.123.158.215 - SmartConsole.png

 

 

View solution in original post

11 Replies
CarstenWeber
Participant

Is there anyone able to shed some light on this, please please? 😅

previously known as (pka.) Carsten_Weber
0 Kudos
Stuart_Green1
Employee
Employee

Hi Carsten,

 

You've got a couple of options. One is the SK here https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...n which goes through the rough process for batch importing from CSV. Your other option is to parse the CSV with Python (or your language of choice) and loop through each line to create your objects using the web / REST API. If you have the choice - try to use R81.10 (maybe R81 - would need to check the release notes) on the management side because we have new options for batch creation which are a lot faster than in previous versions.

0 Kudos
CarstenWeber
Participant

Thanks for your Reply Stuart,
The article has nothing new about this...I got the objects covered via CLI, as mentioned.
I will check the newer version for more features as soon as we have it installed...which will take time. So no choice here, yet.

You are saying there is no option for rules to be added via batch in CLI with CSV files, without using some external scripting creating oneliners?

Best regards
Carsten

previously known as (pka.) Carsten_Weber
0 Kudos
Stuart_Green1
Employee
Employee

Apologies (my fault for trying to multi-task). You're quite right, the link only covers the items you mentioned but you can use the same approach for access-rules too. 

 

Eg - if you have a blank policy you can use something like this:

[Expert@cpmgmt:0]# cat testrules.csv


layer,position,action,source,destination,service
Network,bottom,Accept,srcHost,dstHost,https
Network,bottom,Accept,srcHost,dstHost,icmp-requests
Network,bottom,Accept,srcHost,dstHost,http
Network,bottom,Accept,srcHost,dstHost,telnet
Network,bottom,Accept,srcHost,dstHost,ssh

 

Make sure you have the right name for your layer and src / destination hosts. The format of the CSV isn't fixed, but make sure that you have 'columns' in your CSV that match up to, at least, the required properties for the command you're running.

 

Then run  mgmt_cli -r true add access-rule --batch testrules.csv

 

and you'll get something like this:

 

2022-01-25 10_09_50-20.123.158.215 - SmartConsole.png

 

 

CarstenWeber
Participant

Hi Stuart,

Thanks so much for your more detailed reply. THis is what I was looking for!
I'll have to try this out later, when I have more time to spend. It looks as if this could make it work, though.

 

Best regards
Carsten

P.S.: How about adding all those batch examples and explanations to the documentation, eg. Management API Reference 

 

previously known as (pka.) Carsten_Weber
0 Kudos
CarstenWeber
Participant

Update: We just upgraded to R81.10 JHF 45.

In regards to the creation of objects with CSV-Files...
How does "add object-batch" make use of CSV-Files? I was not able to find any documentation and I am certainly do not use the CLI options to add objects in a mega long oneliner. That is impractical.

Best Regards

Carsten

previously known as (pka.) Carsten_Weber
0 Kudos
Tomer_Noy
Employee
Employee

The "add object-batch" is a versatile new API that can add many types of objects. Each of those object types may have different fields, which means that it's problematic to represent them in a single csv file with constant columns for all objects.

Obviously working with a very long command line is also problematic, so another alternative is to use the REST API flavor and pass everything in a JSON file. The JSON file format is hierarchical and can support multiple object types in a single file.

If you do want to remain with csv files, another alternative is to use an older API that was dedicated to creating hosts. That API has a similar --batch flag that accepts a csv and will create all the requested hosts, which you can then use in your rules.
The below SK has some explanations on how to do that:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

0 Kudos
CarstenWeber
Participant

Thanks for your reply Tomer 🙂

You are referencing to this post we are currently discussing the matter. You must have pasted the wrong link. Do you have the correct link?

Basically I am aware of how the CSV-files have to look like and I managed to do this on R80.30 (v1.5 of the API). Later (from v1.6 R80.40) the "add object-batch" has been added but nothing documented on how to use CSVs with that.


So you are saying, a real bulk of (more than 4) objects to be created cannot be done with this command option? To cuntinue using CSVs it has not been thought of to simply adding a column in CSVs that states what the current object is about?

Or alternatively an addition command option like: "host" e.g. add host object-batch


My suggestion for the first idea could look like this:

(Attention!!! - I am just fooling around/making a suggestion in the following box, this is not the actual syntnax, because there is none for CSVs with object-batch!!!)

 

# Groups
mgmt_cli add object-batch -b groups.csv
CSV header:
object-batch,name,comments
e.g.
group,group_of_hosts_xyz,"this is a comment, with a coma"

# Hosts
mgmt_cli add object-batch -b host.csv
CSV header:
object-batch,name,ipv4_address,comments,groups
e.g.
host,gip_10.0.0.1,10.0.0.1,17,"this is a comment, with a coma", group_of_hosts_xyz

# Ports
mgmt_cli add object-batch -b TCPports.csv
mgmt_cli add object-batch -b UDPports.csv
mgmt_cli add object-batch -b ICMPports.csv
mgmt_cli add object-batch -b DCEports.csv
CSV header:
object-batch,name,port,protocol,match-for-any,comments

e.g.
tcp-port,gtcp_321,321,6,none,false,"this is a comment, with a coma"
udp-port,gudp_123,123,17,none,false,"this is a comment, with a coma"
etc.

 

JSON is not a solution...it's a mess to look at. But hey. I get the logresults exported into JSON instead then and I am certain there is a way to process this and to get the result to look like I need it to in JSON, too. It's just I need to analyse/reduce the duplicates (based on: src, dst, port). And for that I use Excel. But I'm not smart enough to do that and Tufin is too limited for large log files.

 

The "old way" using "--batch" still works.

P.S.: For anyone interested...
I will post a threat on how I dealt with this (with external help from some clever heads NTT Data) when I have some time to spare. It's far from perfect, involves a few manual steps, scripting in PowerShell and shell (bash) and Excel (incl. a "teenie weenie" bit of macro, not too much, I promise!). Someone involved deeper into scripting than us might be able to do it all in one script, though. 😉
But we had some fun getting the "cow from the ice" as we say in German.

Keep an eye out on this threat, as I will update you from here, where to find it.

Regards
Carsten

previously known as (pka.) Carsten_Weber
0 Kudos
Tomer_Noy
Employee
Employee

Indeed I pasted the wrong link. I updated it now to point to a good SK that explains how to use the "old way" of --batch with csv file:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 

We did not remove the existing option of using csv files. For some customers it's easier to use and that's perfectly OK.
If you want to create many objects of a certain type, that's a good option.

The newer batch objects APIs are more versatile, support multiple types in the same command and work work JSON that better supports hierarchical fields. For customers that automate using REST APIs (not cli), this is a more convenient option.

0 Kudos
sudhir_mirajkar
Participant

Hi,

thanks for posting this, its really helpful.

 

i wanted to ask if we can add more then 1 source or destination in the CSV if yes can you please share an example.

thanks again 

0 Kudos
CarstenWeber
Participant

Hi Sudhir,

I was hoping for this to be of help to someone. 😊

Sure, you can add any number, BUT you better not exeed about 100 entries per file and throw in a "publish", just to make sure you are not causing issues on the management system!


This is an example for a host CSV (also using a coma "," inside the comments text):
(to be able to add hosts to a group like this, it has to exist! So create groups first if htey don't exist already.)
name,ipv4-address,comments,groups
Delta_Sevrer,10.3.4.5,Nice Server,MyGroup_1
Omega_Client,10.2.3.4,Great Client System,MyGroup_2
Alpha_Host,10.1.2.3,"Nice and tidy host, a bit small though",MyGroup_4


This is an example for a group CSV:
name,comments
MyGroup_1,This is my Group number one 1
MyGroup_2,This is my Group number one 2
MyGroup_3,This is my Group number one 3
MyGroup_4,This is my Group number one 4
MyGroup_5,This is my Group number one 5

 

Regards 
Carsten

previously known as (pka.) Carsten_Weber
0 Kudos