Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Nicolas_Boisse
Employee Alumnus
Employee Alumnus

RulebaseExporter/RulebaseImporter

Hello Checkmates!

 

I wrote those two tools for one of my projects. I wanted to have an easy way of rulebases migration from Managment to management IE: Lab to prod, prod to lab... To do so, I leverage some useful and quick API calls:

 

As an example, the call:“show access-rulebase name "RulebaseName" details-level full” will return the whole rulebase. The json output will contain all objects(objects-dictionary[]) and rules(rulebase[]). 

Once we have the JSON in hand, we can then execute query on it using "jq".  This means fewer API calls are sent to the management server. Less API calls mean more speed.

 

Here some "jq" queries examples:

This is the command to list all available layers name inside current management.

First we get the JSON file by sending this one API call:

mgmt_cli show access-rulebases --session-id "$session_id" –format json > access-layers.json

Here the JQ query to get all layers name:

cat access-layers.json | jq '."access-layers"[]|.name'

 

This is the command To get the rulebase JSON:

   mgmt_cli show access-rulebase name “$LayerName” --session-id "$session_id" –format json

 

JQ query examples to work on that rulebase:

This will show all information for the access-layer Network:

cat Network.json | jq ‘.”rulebase”[]’

 

This will show information on rule number 12:

cat Network.json | jq ‘.”rulebase”[12]’

 

This will show the Sources of rule number 12:

cat Network,json | jq ‘.”rulebase”[12]|.source’

 

This will show all objects by this rulebase:

cat network.json | jq ‘.”objects-dictionary”[]’

 

You can then search for object name, UID, type like this:

This show information of object name R80GW:

   cat network.json | jq ‘.”objects-dictionary”[]|select (.name== “’R80GW’”)’

 

This show information of object UID aedb0f7c-c5f4-4f07-bad0-9e29200041dc:

cat network.json | jq ‘.”objects-dictionary”[]|select (.uid== “’aedb0f7c-c5f4-4f07-bad0-9e29200041dc’”)’

 

This will show all host in dictionary:

cat network.json | jq ‘.”objects-dictionary”[]|select (.type== “’host”)’

 

The advantage of using JSON and JQ: the export is fast. Don’t need to query the management for every single object. Just one API call gets the full rulebase and objects. All other processing is done locally using “jq” and the received json file. Objects that are not used by the rulebase won’t be exported. 

 

Base on this principle, I created those two scripts. RulebaseExporter.sh and RulebaseImporter.sh.

 

Usage:

Run the exporter script in an empty folder on your Source management server. It will extract all layers from that management server and create a "tar" file.

Copy the TAR archive in an empty folder on the destination Management server. Untar and run RulebaseImporter. It will process all available layers and create the required objects on this management server. 

 

If you are in a domain, provide the CMA name you want to export on the source Managment. On the target management, create an empty CMA then run the Importer and login to that empty CMA.

 

Supported Objects Types:

My scripts currently support those objects types:

  • Access-Rulebase
  • Access-Layers
  • Inline layers
  • Section Titles
  • Network Objects:
    • Host
    • Network
    • Address-Range
    • Group
    • Service-Group
    • service tcp
    • service utp
    • Application Categories (system created)
    • Application object (system created)
  • Login MDM CMA
  • Login to standard Management server
  • Action
    • Accept
    • Drop
    • Inline-layer
  • Enable/Disable rule
  • Rule name

 

TODO: NAT policies.

 

Notepad ++ required 😉

 

Happy scripting!

 

For the full list of White Papers, go here

7 Replies
Joshua_Hatter
Employee
Employee

I am quickly reading over the exporter and I have some feedback!

Appears this only works for < 500 rules.

You get 500 objects at a time with full-details, in large environments these commands can fail due to mgmt_cli timeout of 3 minutes.

We would typically suggest getting less objects maybe 50 at a time, and using offset to get the 'next' 50 until all objects are read.

Nicolas_Boisse
Employee Alumnus
Employee Alumnus

Hi Joshua, thanks for your feedback. Its in my plan to process the export with less objects at a time using the from to variables. I found a way to overcome this limitation though by using section titles. if you have a rulebase of 500 rules, adding a section titles in the middle will give you two sets of 250 rules. In the main policy you will see 2 rules in the main index (.rulebase[]). To be tested... 

I did't hit the 3 minutes time out on my tests. My larger test though was 1 policy with 100 rules, 1 inline layer per rule  with 3 rules per layer. this give 300 rules.  It took 15 seconds  to export everything on a 2 core management VM running on my laptop:

More to come 🙂

0 Kudos
Joshua_Hatter
Employee
Employee

The developed way we provide to get more than 500 of anything is to use the offset limit key.

In the json object that is returned, there are three keys that allow you to programmaticly control whether to pull more data or not.

They are the from, to and total keys. Essentially until to == total, you would increase the offset key in your request by your limit and keep pulling data.

I have python only examples but there are other tools in Code Hub with examples reference points on how to work with these in shell! (https://community.checkpoint.com/docs/DOC-1911)

I've seen as low as 20 rules at a time surpass 20 minute extended Web API timeouts, so it's just something to be mindful of!

Best of luck on your continued work on this. 

0 Kudos
Nicolas_Boisse
Employee Alumnus
Employee Alumnus

Thanks Joshua, I agree with you, there is always a place for improvements and I will add this in a future release. Maybe on the version 2 🙂 For that first version, I wanted to have only 1 json file per policy package/layers.  Then I work on the JSON with jq to rebuild the policy. It's easier to process on my importer script.

20 rules more than 20 minutes seems a very long time to my hears. In the RulebaseExporters example, I only use this API:"show access-rulebase name "rulebasebame" --format json details-level full" to get the full rulebase. This API call didn't take very long to run on all my tests. So I don't understand why it took so long on your example. Can you share a screenshot to my Checkpoint mail  so I can better understand that particular case? Or can you run the ExporterTool on that management to see how long it take? 

Have you tried my exporter script? My last test took 13,36 sec for a 28 rules policy from my production rulebase VM(2 cores only vm). Around 15 sec for 300 rules.

My intend is to have something quick and easy to use. This is what my script does. 

Thanks again for your feed back 🙂

0 Kudos
Joshua_Hatter
Employee
Employee

It was a very specific case of course. I haven't run the tool yet.

Just pointing out some things to help where others have also run into this issue, including myself!

bryanastudillo
Participant

Hello,

Would this script work for me to export from a R80.10 server and import this information into a R80.40 server?

0 Kudos
bryanastudillo
Participant

Hello,

 

I have used the script, apparently the export finishes fine but when I import the rulebase I get no rules, when I check the rulebase file I get an error "Timeout was reached"

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events