Deleting bulk objects inside the group

[sandeepsutar]

Hi All,

We have two groups, each containing more than 16,000 malicious IP objects. Previously, the team was blocking malicious IPs by creating individual host objects. Now we are blocking them through CSV imports using the Indicators feature.

From sometime policy installation has started taking a long time. To streamline the process, I exported all IPs from the existing groups, added them to Indicators, and now want to delete the old objects and their parent groups.

Is there a way to delete a group along with all the objects inside it (note: the IP objects have different names)?

Please let me know if this is clear, or if it needs further clarification.

Mgmt. server Version : R81.20 Take 99

Distributed setup

Regards

[emmap · Accepted solution]

If you delete the group, the objects that were in it will be Unused Objects. In the Objects Explorer you can use the built in 'unused objects' filter to display them 500 at a time and remove them.

You can also do this via APIs if you want to, I believe there's an API call to show unused objects so you can then do the deleting via API. 

[Don_Paterson]

It does look like using the API to do this will be efficient. 

You can run show group or show host and then use jq to filter the output for just object names and then use those to delete via csv batch job. 

You may also be able to do the drops and list maintenance more easily with the Network Feed object. 

https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/delete-host~v1.9.1%20

https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/show-groups~v1.9.1%20

https://sc1.checkpoint.com/documents/latest/APIs/index.html#tips_best_practices~v1.9.1%20 

https://sc1.checkpoint.com/documents/Sales_tools/DemoPoint/Quantum_R81.20/Topics/Network_Feed_Object...

[Don_Paterson · Accepted solution]

Here is an example of something you could test in a lab and then modify for your needs if it is a good option for you.

You are dealing with large numbers of objects (16000 +) so it can impact the management server performance when you use the API to manage (show or delete) so many objects.

I would get an export of the management server and import it into a lab to test (and monitor performance impact).

Malicious_IP_group API export and transform with jq

Replace fwadmin with your SamrtConsole user name.

The commands progress, showing different information each time.

You don't need to consider all of them, just the interesting ones.

This is done in expert mode on the management command line.

mgmt_cli login user "fwadmin" > api-sid.txt

mgmt_cli -s api-sid.txt show groups

mgmt_cli -s api-sid.txt show group name Malicious_IP_group --format json

mgmt_cli -s api-sid.txt show group name Malicious_IP_group --format json | jq -r '.'

mgmt_cli -s api-sid.txt show group name Malicious_IP_group --format json | jq -r '.members[] | .name'

mgmt_cli -s api-sid.txt show group name Malicious_IP_group --format json | jq -r '.members[].name' > GROUP1.csv

This command filters for just names and then appends delete host name so that it actually builds the API commands needed to delete the objects. That way you could do  it in manual batches, of 100 per session (publish after each one).

mgmt_cli -s api-sid.txt show group name Malicious_IP_group --format json | jq -r '.members[] | "delete host name " + .name' > script.txt

This one builds a CSV with just the object names.

You need to add the word name at the top (above the names).

mgmt_cli -s api-sid.txt show group name Malicious_IP_group --format json | jq -r '.members[].name' > Malicious_IP_group.csv

This will fail if the hosts are in the groups. Delete the groups first. Remove the groups from the rules before that. Publish changes.

The concern here is about the large number of objects names and therefore objects to delete in one session.

Might be best to break it into CSV files with 100 objects in each.

mgmt_cli -s api-sid.txt delete host --batch Malicious_IP_group.csv

Delete the groups using API (see above):

mgmt_cli -s api-sid.txt delete group name Malicious_IP_group

mgmt_cli -s api-sid.txt publish

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_RN/Content/Topics-RN/Maximum...

Changes in one session

100

No

To ensure optimal Management Server responsiveness, we recommend making 100 or fewer changes in each session (although the Management Server can support more than 500 changes at a time).

[sandeepsutar]

Thanks Don,

I followed the steps provided, i was able to delete the all the objects and now my policy installation is way faster.

At 1st, I deleted the rule and then deleted group and installed database and pushed policy on firewall.

then followed as below,

mgmt_cli login user "fwadmin" > api-sid.txt
mgmt_cli -s api-sid.txt show group name Malicious_IP_group1 --format json | jq -r '.members[].name' | tr -d '\r' > GROUP1.csv
awk '{print "delete host name \"" $0 "\""}' GROUP1.csv > script_fixed.txt
split -l 300 script_fixed.txt chunk_ --additional-suffix=.txt
for f in chunk_*.txt; do
echo "Processing $f"
while IFS= read -r cmd; do
[[ -z "$cmd" ]] && continue
eval "mgmt_cli -s api-sid.txt $cmd"
done < "$f"
mgmt_cli -s api-sid.txt publish
done

Regards,

[Don_Paterson]

That is great to hear! Thanks for getting back to me.

I forgot to add the logout command, so that the session is closed on the server/API.

It times out after 10 minutes so it's not a problem in most cases.

Logout is best practice, but at least you did it all in one session, and broke it up into chunks of 300. 👍

mgmt_cli -s api-sid.txt logout

For future reference:

This is a way to keep the session ID in a variable and avoid the text file.

SID=$(mgmt_cli login user "fwadmin" --format json \ | jq -r '.sid')

This is a way to do the same as above (SID --> variable)  plus use an API Key to log in.

SID=$(mgmt_cli login --api-key "$(cat /home/admin/api-key.txt)" --format json | jq -r '.sid')

For this one you need to create a new admin account in the SmartConsole (for example: api-fwadmin) and use the Authentication Method > API Key when you create the account.

[the_rock · Accepted solution]

What both Emma and Don had said is by far the best method to use. I remember doing this via API for the customer once and went well, no issues. You could also use object explorwer, mind you, might not be able to do ctrl+a for that many objects at once and delete, but you can do 15-20 at the time, thats feasable.