topic Re: Basic script for importing IP Address objects from feed (here office365) in API / CLI Discussion

Basic script for importing IP Address objects from feed (here office365)

Nüüül — Mon, 09 Jul 2018 18:14:07 GMT

Can be adapted to other feeds with CIDR addresses.

Created Objects are then added to a group - defined at the top

compares current group members with new feed and adds new / removes unused ones.

Logging is kind of ugly, but in progress.

i planned to run this every day on my lab environment as cron. kind of works

cheers,

Daniel

any hints and improvements appreciated

Re: Basic script for importing IP Address objects from feed (here office365)

Robert_Decker — Tue, 10 Jul 2018 07:43:59 GMT

Hi Daniel,

Thanks for sharing the script with us.

I've reviewed the code and it is wrong regarding the Management API context:

1. Your "logout" command doesn't match the "login" command, because it is inside the nested "if" block.

2. After the login command you add a group object, which is published inside the nested "if" block. if that object is not used in other circumstances, you have to consider to "discard" that change, before logout. Or, alternatively add that object inside the second "if" block just before the show command.

Remember, you must match login and logout commands, and discard the changes if you are not using them.

Otherwise you will have locked objects and orphan sessions.

Robert.

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Tue, 10 Jul 2018 14:08:28 GMT

Robert,

thanks a lot for your input. so i changed the script slightly according your comments.

If Changes are not saved, i am sending a "mgmt_cli discard".

Moved the mgmt login and logout, so there should no orphaned sessions occur again.

to 2. this group object is used. The network objects created by the script are directly added into this group.

mgmt_cli add group name "$v_grp" color "$v_objcolor" comments "$v_objcomment" -s id.txt
.
.
.
awk -v awk_grp="$v_grp" -v awk_opfx="$v_objprefix" -v awk_color="$v_objcolor" -v awk_comment="$v_objcomment" 'FNR==NR { a[FNR""] = $0; next } { print "mgmt_cli -s id.txt add network name \""awk_opfx""a[FNR""]"\" ",$0" color \""awk_color"\" groups.1 \""awk_grp"\" comments \""awk_comment"\""}' $v_diff_add $v_diff_add_netmask >$v_diff_add_sh

If the group exists, i just got the error :

code: "err_validation_failed"
message: "Validation failed with 1 error"
errors:
- message: "More than one object named 'ge_o365-networks' exists."

This was one of the parts, i think the script is ugly

So, I added a condition to check if the group exists and only to send the "mgmt_cli add group name" if not. So the message above should not occur again.

again - thanks. You helped me a lot!

Cheers,

Daniel

Re: Basic script for importing IP Address objects from feed (here office365)

brian_patterso1 — Thu, 12 Jul 2018 23:02:53 GMT

Hi Daniel,

Thanks much for posting and sharing the script. I have been thinking about the same thing for O365.

I noticed that the script is pulling XML data from the MS so I thought I would mention that I read MS is planning to move to REST. XML and RSS will be phased out.

https://support.office.com/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728eb46bf9a#webservice

At the web page, MS provides URI samples for testing.

Also, if I am reading it correctly, a change query is available so the consumer will not have to diff to figure out the changes.

Regards,

Brian

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Thu, 12 Jul 2018 23:30:54 GMT

Update 2018-07-13

File itself is slightly changed. mainly one new line at the variables:

#(if needed) define mail subject for notification
echo "Subject: "Activity Report - Office365 Import Script""

With this, you can schedule a cronjob with command:

sh /scripts/o365-api | /usr/bin/tee -a /scripts/o365_logging 2>&1 | /usr/sbin/sendmail --domain=<mail domain> -f <sender address> -v <recipient address> --host=<mail relay> 2>&1

If settings are correct, checkpoint logs, what the script is doing and is sending a small email after the job is done

example:

Subject: "Activity Report - Office365 Import Script"
################## Script starts : 2018.07.13-00.41.01 ##################
group ge_o365-networks already exists
No Changes!
message: "OK"

DONE
################## Script ends : 2018.07.13-00.41.01 ##################

Subject: "Activity Report - Office365 Import Script"
################## Script starts : 2018.07.13-00.50.01 ##################
group ge_o365-networks already exists
Files o365_helper_ms-objsorted.tmp and o365_helper_inst-objsorted.tmp differ
Found objects to remove
found new objects!
tasks:
- task-id: "0000000-0000-00000-00000-000"
task-name: "Publish operation"
status: "succeeded"
progress-percentage: 100
suppressed: false
task-details:
- publishResponse:
numberOfPublishedChanges: 184
mode: "async"
revision: "0000000-0000-00000-00000-000"
Done! Installing Policy!
.
.
.
message: "OK"

DONE
################## Script ends : 2018.07.13-00.50.01 ##################

not that ugly anymore

Daniel

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Fri, 13 Jul 2018 08:55:17 GMT

Hi Brian,

thanks for your input. i did let the script run against the URL mentioned by MS:

https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a7

did work so far, as ip addresses are still in CIDR

Regarding the changes only "feed", that might be good, but, if you miss one update, you will sooner or later run into problems. therefore I´d stay at the full list and compare it against the actual installed policy.

The diff does not produce heavy load so does not hurt...

And the script still is able to be adapted, when someone needs something similar, he just adds a text file on a webserver, containing addresses...

it looks like, MS is still in developing mode on this, will check in a month or so and adjust the feed URL.

Cheers,

Daniel

Re: Basic script for importing IP Address objects from feed (here office365)

Theis_Andersen_ — Fri, 13 Jul 2018 13:15:57 GMT

Daniel, could you post it on GitHub?

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Sun, 15 Jul 2018 17:04:00 GMT

GitHub - leinadred/IPaddressFeed2CheckPointAPI: Adding a IP Address feed (CIDR) into Checkpoint Objects (here Office 365…

Re: Basic script for importing IP Address objects from feed (here office365)

Nick_Mandafouni — Wed, 18 Jul 2018 10:57:55 GMT

Thanks for this, much needed indeed. Would you be able to group these IP addresses into individual Groups? i.e. we need to have traffic to Skype for Business /Teams to bypass our proxies whereas the rest of the O365 can go through them. It would be good if this IP address feed could be split into different MS groups like SfB, Exchange, Sharepoint, etc..

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Wed, 18 Jul 2018 14:49:06 GMT

This will be possible with the newer feed which is an API output.

There are titles like

  "serviceArea": "Exchange",    "serviceAreaDisplayName": "Exchange Online",

With this given, you should be able to i.e. grep the "Exchange Online" and set it into the group name...

Daniel

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Fri, 21 Jun 2019 09:25:01 GMT

What I just learned at the Tech Talks to R80.20.M1, there will arrive a good possibility for such updated cloud targets (office365, amazon and so on).

https://community.checkpoint.com/message/22778-new-cloudguard-iaas-features-with-r8020m1-smartconsole <-- look at the screenshot, the network objects menu opened has an entry "online services"

Tomer clicked through a bit, there were different service types, like skype or Exchange Online...

Maybe will have a look later this or next week in my lab...

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Tue, 24 Jul 2018 08:44:35 GMT

Hi,

According to the Security Management Guide - ( Security Management R80.20.M1 Administration Guide 😞

the Online Services are supported on Gateways running R80.20 or higher. <-- so at the moment i cannot test it in my lab
the addresses are automatically updated, without needing a policy push ore something, which is good...
Activities like updated addresses are visible in the log. which is good.

So another good point for discussing gateway upgrades to R80.20 - when it is GA

Daniel

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Sun, 09 Dec 2018 19:54:55 GMT

Hi all,

just updated the script regarding the new feed from Microsoft and for CP Version R80.20

#This machine version R80.20, uncomment the next line

source /opt/CPshrd-R80.20/tmp/.CPprofile.sh

#If this machine version R80.10, uncomment the line below

#source /opt/CPshrd-R80/tmp/.CPprofile.sh

(In case you want to use the script, even though the updatable objects at R80.20)

should be ok, to change / uncomment while implementing the script following the word document, which can be found at the github (latest Script and Implementation Document. (GitHub - leinadred/IPaddressFeed2CheckPointAPI: Adding a IP Address feed (CIDR) into Checkpoint Objects (here Office 365… )

Re: Basic script for importing IP Address objects from feed (here office365)

Cody_Ray — Wed, 19 Dec 2018 15:17:40 GMT

Is it possible to install policy to multiple cluster objects? If so, how should the variable v_poltarget be assigned? API mentions List: string.

v_poltarget=primarytarget secondarytarget

Excellent work btw.

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Wed, 19 Dec 2018 17:09:01 GMT

Thanks Cody!

sure, you can add to the header:

v_poltarget1=primarytarget
v_poltarget2=secondarytarget
v_poltarget3=tertiarytarget

Then, row 137

mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget1" -s id.txt
mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget2" -s id.txt
mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget3" -s id.txt

not sure if

mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget1" targets.2 "$v_poltarget2" targets.3 "$v_poltarget3" -s id.txt

works. If you want to try it and tell me if it works

Re: Basic script for importing IP Address objects from feed (here office365)

Cody_Ray — Fri, 11 Jan 2019 22:14:37 GMT

Single line doesn't work. You must enter it in as:

mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget1" -s id.txt

mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget2" -s id.txt

mgmt_cli install-policy policy-package "$v_polpack" access true threat-prevention true targets.1 "$v_poltarget3" -s id.txt

Thanks again

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Sat, 12 Jan 2019 11:58:55 GMT

Thanks for checking and sharing!

Re: Basic script for importing IP Address objects from feed (here office365)

Chris_Atkinson — Tue, 15 Jan 2019 02:03:04 GMT

Hi All,

For future reference please be aware of the following related articles:

sk131852 - Updatable Objects in R80.20

sk135572 - Microsoft Office 365 objects as Network Objects in R80.20

Re: Basic script for importing IP Address objects from feed (here office365)

Nüüül — Tue, 15 Jan 2019 11:07:29 GMT

Hi,

thanks, in this thread it is mentioned multiple times, that there is an ability of using the updatable objects with Gateways and Management on R80.20.

Although the script can be used for other sources (i.e. API Export from IP Address Management), as long as the output there is CIDR formatted.

Daniel

Re: Basic script for importing IP Address objects from feed (here office365)

Paul_Hagyard — Fri, 22 Nov 2019 02:34:28 GMT

Re the environment variables, how about:

. /etc/profile.d/CP.sh

Note the space between the "." and the script name

Saves you having to update the script after upgrades as CP.sh always points to the correct file location.