topic Re: RulebaseExporter/RulebaseImporter in API / CLI Discussion

RulebaseExporter/RulebaseImporter

Nicolas_Boisse — Mon, 08 Jul 2019 12:32:14 GMT

Hello Checkmates!

I wrote those two tools for one of my projects. I wanted to have an easy way of rulebases migration from Managment to management IE: Lab to prod, prod to lab... To do so, I leverage some useful and quick API calls:

As an example, the call:“show access-rulebase name "RulebaseName" details-level full” will return the whole rulebase. The json output will contain all objects(objects-dictionary[]) and rules(rulebase[]).

Once we have the JSON in hand, we can then execute query on it using "jq". This means fewer API calls are sent to the management server. Less API calls mean more speed.

Here some "jq" queries examples:

This is the command to list all available layers name inside current management.

First we get the JSON file by sending this one API call:

mgmt_cli show access-rulebases --session-id "$session_id" –format json > access-layers.json

Here the JQ query to get all layers name:

cat access-layers.json | jq '."access-layers"[]|.name'

This is the command To get the rulebase JSON:

mgmt_cli show access-rulebase name “$LayerName” --session-id "$session_id" –format json

JQ query examples to work on that rulebase:

This will show all information for the access-layer Network:

cat Network.json | jq ‘.”rulebase”[]’

This will show information on rule number 12:

cat Network.json | jq ‘.”rulebase”[12]’

This will show the Sources of rule number 12:

cat Network,json | jq ‘.”rulebase”[12]|.source’

This will show all objects by this rulebase:

cat network.json | jq ‘.”objects-dictionary”[]’

You can then search for object name, UID, type like this:

This show information of object name R80GW:

cat network.json | jq ‘.”objects-dictionary”[]|select (.name== “’R80GW’”)’

This show information of object UID aedb0f7c-c5f4-4f07-bad0-9e29200041dc:

cat network.json | jq ‘.”objects-dictionary”[]|select (.uid== “’aedb0f7c-c5f4-4f07-bad0-9e29200041dc’”)’

This will show all host in dictionary:

cat network.json | jq ‘.”objects-dictionary”[]|select (.type== “’host”)’

The advantage of using JSON and JQ: the export is fast. Don’t need to query the management for every single object. Just one API call gets the full rulebase and objects. All other processing is done locally using “jq” and the received json file. Objects that are not used by the rulebase won’t be exported.

Base on this principle, I created those two scripts. RulebaseExporter.sh and RulebaseImporter.sh.

Usage:

Run the exporter script in an empty folder on your Source management server. It will extract all layers from that management server and create a "tar" file.

Copy the TAR archive in an empty folder on the destination Management server. Untar and run RulebaseImporter. It will process all available layers and create the required objects on this management server.

If you are in a domain, provide the CMA name you want to export on the source Managment. On the target management, create an empty CMA then run the Importer and login to that empty CMA.

Supported Objects Types:

My scripts currently support those objects types:

Access-Rulebase
Access-Layers
Inline layers
Section Titles
Network Objects:
- Host
- Network
- Address-Range
- Group
- Service-Group
- service tcp
- service utp
- Application Categories (system created)
- Application object (system created)
Login MDM CMA
Login to standard Management server
Action
- Accept
- Drop
- Inline-layer
Enable/Disable rule
Rule name

TODO: NAT policies.

Notepad ++ required 😉

Happy scripting!

For the full list of White Papers, go here.

Re: RulebaseExporter/RulebaseImporter

Joshua_Hatter — Tue, 23 Oct 2018 20:14:33 GMT

I am quickly reading over the exporter and I have some feedback!

Appears this only works for < 500 rules.

You get 500 objects at a time with full-details, in large environments these commands can fail due to mgmt_cli timeout of 3 minutes.

We would typically suggest getting less objects maybe 50 at a time, and using offset to get the 'next' 50 until all objects are read.

Re: RulebaseExporter/RulebaseImporter

Nicolas_Boisse — Tue, 23 Oct 2018 21:06:58 GMT

Hi Joshua, thanks for your feedback. Its in my plan to process the export with less objects at a time using the from to variables. I found a way to overcome this limitation though by using section titles. if you have a rulebase of 500 rules, adding a section titles in the middle will give you two sets of 250 rules. In the main policy you will see 2 rules in the main index (.rulebase[]). To be tested...

I did't hit the 3 minutes time out on my tests. My larger test though was 1 policy with 100 rules, 1 inline layer per rule with 3 rules per layer. this give 300 rules. It took 15 seconds to export everything on a 2 core management VM running on my laptop:

More to come 🙂

Re: RulebaseExporter/RulebaseImporter

Joshua_Hatter — Tue, 23 Oct 2018 22:10:29 GMT

The developed way we provide to get more than 500 of anything is to use the offset limit key.

In the json object that is returned, there are three keys that allow you to programmaticly control whether to pull more data or not.

They are the from, to and total keys. Essentially until to == total, you would increase the offset key in your request by your limit and keep pulling data.

I have python only examples but there are other tools in Code Hub with examples reference points on how to work with these in shell! (https://community.checkpoint.com/docs/DOC-1911)

I've seen as low as 20 rules at a time surpass 20 minute extended Web API timeouts, so it's just something to be mindful of!

Best of luck on your continued work on this.

Re: RulebaseExporter/RulebaseImporter

Nicolas_Boisse — Tue, 23 Oct 2018 23:11:18 GMT

Thanks Joshua, I agree with you, there is always a place for improvements and I will add this in a future release. Maybe on the version 2 🙂 For that first version, I wanted to have only 1 json file per policy package/layers. Then I work on the JSON with jq to rebuild the policy. It's easier to process on my importer script.

20 rules more than 20 minutes seems a very long time to my hears. In the RulebaseExporters example, I only use this API:"show access-rulebase name "rulebasebame" --format json details-level full" to get the full rulebase. This API call didn't take very long to run on all my tests. So I don't understand why it took so long on your example. Can you share a screenshot to my Checkpoint mail so I can better understand that particular case? Or can you run the ExporterTool on that management to see how long it take?

Have you tried my exporter script? My last test took 13,36 sec for a 28 rules policy from my production rulebase VM(2 cores only vm). Around 15 sec for 300 rules.

My intend is to have something quick and easy to use. This is what my script does.

Thanks again for your feed back 🙂

Re: RulebaseExporter/RulebaseImporter

Joshua_Hatter — Wed, 24 Oct 2018 12:48:13 GMT

It was a very specific case of course. I haven't run the tool yet.

Just pointing out some things to help where others have also run into this issue, including myself!

Re: RulebaseExporter/RulebaseImporter

bryanastudillo — Thu, 04 Mar 2021 21:38:09 GMT

Hello,

Would this script work for me to export from a R80.10 server and import this information into a R80.40 server?

Re: RulebaseExporter/RulebaseImporter

bryanastudillo — Thu, 13 May 2021 15:26:15 GMT

Hello,

I have used the script, apparently the export finishes fine but when I import the rulebase I get no rules, when I check the rulebase file I get an error "Timeout was reached"