How to add/create rules in bulk/batch using a .csv file via mgmt_cli API?

[CarstenWeber]

Hi there, 

does anyone know a way to add/create R77.30 style access rules (no inline 'n stuff) in R80.30 using the API (v1.5) and a csv file listing multiple rules in a multi domain environment?
The following is what I got so far, but that is pure guesswork...how would this look right?

So this is me trying to add a number of access rules containing all the relevant data  (I don't need "time", "install on", etc.).
I am guessing the following after deriving bits and pieces all over the forum and internet:

mgmt_cli add access-rule -b /var/log/tmp/rules.csv --version 1.5

How would the correct csv file first row (header) look like? Something like this?:

package,layer,position.above,source,destination,service,track,comments

package --- would be the according policy name used in that domain
postion.above --- will be a ruleUID as value

Does anyone have  a working example of the above szenario? I can't be that far away from a working set of command and csv syntax.

Thanks a lot in advance.

Consider me already logged in to the according domain (Multi Domain Management) before executing anything else.

mgmt_cli login -d <domain_objid> > sid.txt.$$

$$ --- will add the session ID for a more or less unique file name. I know, I know, not ideal...but it serves the purpose allright in this case.
objid --- can be gathered using teh following command...at least on a R80.30 MDM MGMT server. A single CMA does not need this parameter (-d).

psql_client cpm postgres -c"select objid,name from domainbase_data where dlesession=0 and not deleted;"

Publish every 200-300 changes!

mgmt_cli publish -s sid.txt.$$

Logout when done

mgmt_cli logout -s sid.txt.$$

I also used the API to create all the objects like groups, hosts (adding them to groups already created as well in one go) and services (ports).

# Groups
mgmt_cli add group -b groups.csv -s sid.txt.$$
CSV header:
name,comments

# Hosts
mgmt_cli add host -b host.csv -s sid.txt.$$
CSV header:
name,ipv4_address,comments,groups

# Ports
mgmt_cli add service-tcp -b TCPports.csv
mgmt_cli add service-udp -b UDPports.csv
mgmt_cli add service-icmp -b ICMPports.csv
mgmt_cli add service-dce-rpc -b DCEports.csv
CSV header:
name,port,protocol,match-for-any,comments

The reason to do the above is a vast number of connections (50+ million times hit count) I gathered from a few firewall logs matching too wide rules (something close to "src:<RFC1918_ranges> dst:<RFC1918_ranges> svc:any action:allow"). In order to create more specific set of rules for traffic really happening and bleeding out the too wide rules eventually. 
I used an Excelsheet with loads of formulas I have never ever used before to prepare tabs for each of the required csv-files (groups, hosts, ports, rules) based on the reference data collected in the first tab (src, dst, port, protocol). The tabs are filled automatically ready for export as seperate csv-files. And here I am stuck on the rule part and how it has to look like. Please help

PS: All the links in other articles about adding objects and nat rules did not really mention anything else I need. I don't have to mention that the API reference does not explain bulk/batch usage. This is odd, as the API should help automate processes.

[CarstenWeber]

Is there anyone able to shed some light on this, please please? 😅

[StuartGreen]

Hi Carsten,

You've got a couple of options. One is the SK here https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...n which goes through the rough process for batch importing from CSV. Your other option is to parse the CSV with Python (or your language of choice) and loop through each line to create your objects using the web / REST API. If you have the choice - try to use R81.10 (maybe R81 - would need to check the release notes) on the management side because we have new options for batch creation which are a lot faster than in previous versions.

[CarstenWeber]

Thanks for your Reply Stuart,
The article has nothing new about this...I got the objects covered via CLI, as mentioned.
I will check the newer version for more features as soon as we have it installed...which will take time. So no choice here, yet.

You are saying there is no option for rules to be added via batch in CLI with CSV files, without using some external scripting creating oneliners?

Best regards
Carsten

[StuartGreen · Accepted solution]

Apologies (my fault for trying to multi-task). You're quite right, the link only covers the items you mentioned but you can use the same approach for access-rules too. 

Eg - if you have a blank policy you can use something like this:

[Expert@cpmgmt:0]# cat testrules.csv

layer,position,action,source,destination,service
Network,bottom,Accept,srcHost,dstHost,https
Network,bottom,Accept,srcHost,dstHost,icmp-requests
Network,bottom,Accept,srcHost,dstHost,http
Network,bottom,Accept,srcHost,dstHost,telnet
Network,bottom,Accept,srcHost,dstHost,ssh

Make sure you have the right name for your layer and src / destination hosts. The format of the CSV isn't fixed, but make sure that you have 'columns' in your CSV that match up to, at least, the required properties for the command you're running.

Then run  mgmt_cli -r true add access-rule --batch testrules.csv

and you'll get something like this:

[CarstenWeber]

Hi Stuart,

Thanks so much for your more detailed reply. THis is what I was looking for!
I'll have to try this out later, when I have more time to spend. It looks as if this could make it work, though.

Best regards
Carsten

P.S.: How about adding all those batch examples and explanations to the documentation, eg. Management API Reference 

[CarstenWeber]

Update: We just upgraded to R81.10 JHF 45.

In regards to the creation of objects with CSV-Files...
How does "add object-batch" make use of CSV-Files? I was not able to find any documentation and I am certainly do not use the CLI options to add objects in a mega long oneliner. That is impractical.

Best Regards

Carsten

[Tomer_Noy]

The "add object-batch" is a versatile new API that can add many types of objects. Each of those object types may have different fields, which means that it's problematic to represent them in a single csv file with constant columns for all objects.

Obviously working with a very long command line is also problematic, so another alternative is to use the REST API flavor and pass everything in a JSON file. The JSON file format is hierarchical and can support multiple object types in a single file.

If you do want to remain with csv files, another alternative is to use an older API that was dedicated to creating hosts. That API has a similar --batch flag that accepts a csv and will create all the requested hosts, which you can then use in your rules.
The below SK has some explanations on how to do that:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[CarstenWeber]

Thanks for your reply Tomer 🙂

You are referencing to this post we are currently discussing the matter. You must have pasted the wrong link. Do you have the correct link?

Basically I am aware of how the CSV-files have to look like and I managed to do this on R80.30 (v1.5 of the API). Later (from v1.6 R80.40) the "add object-batch" has been added but nothing documented on how to use CSVs with that.


So you are saying, a real bulk of (more than 4) objects to be created cannot be done with this command option? To cuntinue using CSVs it has not been thought of to simply adding a column in CSVs that states what the current object is about?

Or alternatively an addition command option like: "host" e.g. add host object-batch

My suggestion for the first idea could look like this:

(Attention!!! - I am just fooling around/making a suggestion in the following box, this is not the actual syntnax, because there is none for CSVs with object-batch!!!)

# Groups
mgmt_cli add object-batch -b groups.csv
CSV header:
object-batch,name,comments
e.g.
group,group_of_hosts_xyz,"this is a comment, with a coma"

# Hosts
mgmt_cli add object-batch -b host.csv
CSV header:
object-batch,name,ipv4_address,comments,groups
e.g.
host,gip_10.0.0.1,10.0.0.1,17,"this is a comment, with a coma", group_of_hosts_xyz

# Ports
mgmt_cli add object-batch -b TCPports.csv
mgmt_cli add object-batch -b UDPports.csv
mgmt_cli add object-batch -b ICMPports.csv
mgmt_cli add object-batch -b DCEports.csv
CSV header:
object-batch,name,port,protocol,match-for-any,comments

e.g.
tcp-port,gtcp_321,321,6,none,false,"this is a comment, with a coma"
udp-port,gudp_123,123,17,none,false,"this is a comment, with a coma"
etc.

JSON is not a solution...it's a mess to look at. But hey. I get the logresults exported into JSON instead then and I am certain there is a way to process this and to get the result to look like I need it to in JSON, too. It's just I need to analyse/reduce the duplicates (based on: src, dst, port). And for that I use Excel. But I'm not smart enough to do that and Tufin is too limited for large log files.

The "old way" using "--batch" still works.

P.S.: For anyone interested...
I will post a threat on how I dealt with this (with external help from some clever heads NTT Data) when I have some time to spare. It's far from perfect, involves a few manual steps, scripting in PowerShell and shell (bash) and Excel (incl. a "teenie weenie" bit of macro, not too much, I promise!). Someone involved deeper into scripting than us might be able to do it all in one script, though. 😉
But we had some fun getting the "cow from the ice" as we say in German.

Keep an eye out on this threat, as I will update you from here, where to find it.

Regards
Carsten

[Tomer_Noy]

Indeed I pasted the wrong link. I updated it now to point to a good SK that explains how to use the "old way" of --batch with csv file:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 

We did not remove the existing option of using csv files. For some customers it's easier to use and that's perfectly OK.
If you want to create many objects of a certain type, that's a good option.

The newer batch objects APIs are more versatile, support multiple types in the same command and work work JSON that better supports hierarchical fields. For customers that automate using REST APIs (not cli), this is a more convenient option.

[genisis__]

What I've always wondered is how to ensure you create a rule using a batch job, between other rules rather then at the bottom.  Its not something I've had the time to try,  I've created objects, groups, added to objects to groups, but never create a rule especially with a section heading between other rules, and to take it further a inline rule.

Any examples of this?

[sudhir_mirajkar]

Hi,

thanks for posting this, its really helpful.

i wanted to ask if we can add more then 1 source or destination in the CSV if yes can you please share an example.

thanks again 

[CarstenWeber]

Hi Sudhir,

I was hoping for this to be of help to someone. 😊

Sure, you can add any number, BUT you better not exeed about 100 entries per file and throw in a "publish", just to make sure you are not causing issues on the management system!

This is an example for a host CSV (also using a coma "," inside the comments text):
(to be able to add hosts to a group like this, it has to exist! So create groups first if htey don't exist already.)
name,ipv4-address,comments,groups
Delta_Sevrer,10.3.4.5,Nice Server,MyGroup_1
Omega_Client,10.2.3.4,Great Client System,MyGroup_2
Alpha_Host,10.1.2.3,"Nice and tidy host, a bit small though",MyGroup_4

This is an example for a group CSV:
name,comments
MyGroup_1,This is my Group number one 1
MyGroup_2,This is my Group number one 2
MyGroup_3,This is my Group number one 3
MyGroup_4,This is my Group number one 4
MyGroup_5,This is my Group number one 5

Regards 
Carsten

[DZ_KB]

Hi @StuartGreen , @Tomer_Noy 

How do you write the csv file in order to add a list of hosts on source/destination or a list of services in only on rule ?

Thanks in advance for your reply

[Phuong_Nguyen_L]

Hi Stuart_Green1,

I need 2 host are host1 and host2 into source column, How to put multi value into column !

Many thank! 

[Expert@cpmgmt:0]# cat testrules.csv

layer,position,action,source,destination,service
Network,bottom,Accept,srcHost,dstHost,https
Network,bottom,Accept,srcHost,dstHost,icmp-requests
Network,bottom,Accept,srcHost,dstHost,http
Network,bottom,Accept,srcHost,dstHost,telnet
Network,bottom,Accept,srcHost,dstHost,ssh

[PhoneBoy]

Using your example, the file will look like:

layer,position,action,source,destination,service.1,service.2,service.3,service.4,service.5
Network,bottom,Accept,srcHost,dstHost,https,icmp-requests,http,telnet,ssh

[Phuong_Nguyen_L]

Dear PhoneBoy,

Thank for your help, Your example work fine :)!

Many thank!