One-liner to update IpToCountry data on Security Managements

[Danny]

👍 Automates all tasks from sk120261
ℹ️ Can be easily scheduled to auto-run at specific times.

One-liner (Bash) to assist updating IpToCountry.csv on Check Point security managements.
In expert mode run:

if [[ `$CPDIR/bin/cpprod_util FwIsFirewallMgmt 2>/dev/null` != *'1'* ]]; then echo; tput bold; echo ' Not a security management!'; tput sgr0; echo; else curl_cli $(if [[ `grep proxy:ip /config/active` ]]; then echo -n '--proxy '; grep proxy:ip /config/active|cut -f2 -d' '|tr -d '\n'; echo -n :; grep proxy:port /config/active|cut -f2 -d' '; fi) -k https://sc1.checkpoint.com/freud2/IpToCountry.csv.gz | zcat > IpToCountry.csv; if [[ $? == 0 ]]; then echo; tput bold; echo -n ' Updating IpToCountry requires '; tput sgr0; tput setab 1; tput setaf 7; tput bold; echo -n 'CPSTOP'; tput sgr0; tput bold; echo -n '! Executing? '; tput sgr0; read -sn1; case $REPLY in '') echo; dos2unix IpToCountry.csv; chown admin:root IpToCountry.csv; chmod 770 IpToCountry.csv; cp -p $RTDIR/conf/ip2country.csv $RTDIR/conf/ip2country.csv_$(date +%Y%m%d%H%M); cp -p $INDEXERDIR/conf/ip2country.csv $INDEXERDIR/conf/ip2country.csv_$(date +%Y%m%d%H%M); cpstop; sleep 20; cp -v IpToCountry.csv $RTDIR/conf/ip2country.csv; mv -v IpToCountry.csv $INDEXERDIR/conf/ip2country.csv; cpstart; echo; tput setaf 2; echo 'Update completed!'; tput sgr0;; *) tput setaf 1; echo ' Abort!'; tput sgr0; esac; else echo; tput setaf 1; echo ' Download failed!'; tput sgr0; echo; fi; fi

This one-liner interactively downloads the newest IpToCountry.csv from Check Point, checks if the download needs to be performed via web proxy and asks before cpstop is to be executed.

Integrated with our ccc script.

Also available as cronjob version:

#!/bin/bash
#
# Auto-Update IP2Country.csv on Check Point Security Managements - sk120261
#

source /etc/profile.d/CP.sh
logger 'Starting to download ip2country.csv'
curl_cli -k https://sc1.checkpoint.com/freud2/IpToCountry.csv.gz|zcat > IpToCountry.csv
if [[ $? == 0 ]]; then dos2unix IpToCountry.csv; chown admin:root IpToCountry.csv; chmod 770 IpToCountry.csv; cp -p $RTDIR/conf/ip2country.csv $RTDIR/conf/ip2country.csv_$(date +%Y%m%d%H%M); cp -p $INDEXERDIR/conf/ip2country.csv $INDEXERDIR/conf/ip2country.csv_$(date +%Y%m%d%H%M); cpstop; sleep 20; cp -v IpToCountry.csv $RTDIR/conf/ip2country.csv; mv -v IpToCountry.csv $INDEXERDIR/conf/ip2country.csv; cpstart; logger 'ip2country.csv update completed.'; else logger 'ip2country.csv download failed!'; fi

[Jarvis_Lin]

Hi Denny,

Can the update be performed automatically without asking to cpstop?

Thanks.

Regards,

Jarvis

[Danny]

Hi @Jarvis_Lin , this isn't possible as the firewall management processes are actively working with these files. So the processes need to be stopped before the files can be replaced. Otherwise Check Point would have probably implemented an automatic update. I think it should be possible to update these files before the management processes are started at system startup. Then you would receive GeoIP updates whenever the system gets restarted.

[PhoneBoy]

Looks like we are addressing the underlying issue in R81.20 and will backport this to the various versions via JHF.
Not sure how far back it will go, but I imagine to at least R80.40.
Ultimately, we'll auto-download this information to the management daily at midnight.
It should also solve an issue where the names between our Updatable Objects and the logs aren't exactly aligned.

[Nikodemus]

Thank you Danny for the script. Already using a slighty modified version of it for a while running as a cronjob on R81.10. It works perfect.

We noticed that, starting from R81.10 JHF61 (not GA atm), a new feature is introduced:

PRJ-36199 - UPDATE: Mapping of IPs to
country/flag is now automatically
updated every day. These are
visible in the Logs and Events
views.

Source (pg 9, bottom): https://dl3.checkpoint.com/paid/10/107be089b9abbc276faf5c7dd1daa51c/CP_JumboHotfixAccumulator_for_R8...

Is above improvement the replacement for your script?

[the_rock]

I get this when I run it on my lab R81.10 mgmt server:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10 100 10 0 0 8 0 0:00:01 0:00:01 --:--:-- 8

gzip: stdin: not in gzip format

[Danny]

Check Point mentioned here that they updated the path to IpToCountry.csv.
It's not this URL anymore as referenced to in sk120261, but this new URL instead. I updated my one-liner.

[Ave_Joe]

I am getting the same output when using the updated one-liner with the new URL. 

Thoughts?

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

gzip: stdin: unexpected end of file

Download failed!

[Danny · Accepted solution]

I found out the the new URL is actually just a forwarder to another URL.

Fixed it. Should be working again now.

[the_rock]

You are a TRUE CHAMPION!!

[Ave_Joe]

Confirmed to be working now. 

Thank you. 

[demist95]

Hello Danny,

Many thanks for creating this script!

However, I compared the IptoCountry.csv's from the URL in your script, to the URL used in sk120261, it seems that both have a different md5sum:

d3dd4e3df9ae369afb6e2a2dbb41b482 IpToCountry.csv

21172b2bf3126b1af58fd27d0e4033b6 IpToCountry.csv

Furthermore, this Russian IP was showing a German flag in the Smart Console logs: 193.201.9.238.

I did a calculation on the IP using sk94364, did a lookup in both .csv's:

238 + (9 * 256) + (201 * 256 * 256) + (193 * 256 * 256 * 256) = 3251177966

3251177472,"3251178239","iana","410227200","RU","RUS","Russia" --> IpToCountry.csv  from sk106216

3251177472,"3251179519","iana","410227200","DE","DEU","Germany" --> IpToCountry.csv from the one-liner script

The link used in the SK does not work in the one-liner script. Is it possible for you to update your one-liner script with the correct URL?

EDIT: I just saw that you used the URL from a post by a Check Point employee. So it seems that the latest IpToCountry.csv Check Point has provided is not correct. I will let them know in a TAC case we have in regards to wrong country flags showing in Smart Console.

Kind regards,

Demis Terborg

[Danny]

Hi Demis,

looks like Check Point switched back to their original URL again, so I adjusted my script accordingly.

Regards, Danny

[GHaider]

i am not sure but the release notes for JHF 61 for R81.10 mention the following:

PRJ-36199

Logging

UPDATE: Mapping of IPs to country/flag is now automatically updated every day. These are visible in the Logs and Events views.

...so perhaps your script is not needed anymore? (will try to give feedback after testing this)

[Nikodemus]

Installed JHF61 for the same reason, but SSH access to the MGMT was not working anymore afterwards. Luckily HTTPS was still working, Troubleshooting with the console in the GAIA portal is a pain, so rolled back to JHF 55. Did not test further. 

[GHaider]

you should have used JHF 66 as i did, see here:

PRJ-40378, PMTR-84711

Gaia OS

After an upgrade, there are SSH connectivity issues, when the "allowed-host" feature in Clish is enabled.

[Nikodemus]

Good to know it has been fixed, but at that time, JHF61 was the latest available. And as the country script stopped working, I was searching for an alternative. 🙂

[demist95]

We have a setup running on R81.10 JHF 66.

The ip2country.csv does not seem to be updated automatically.

[Expert@CP-SC1:0]# ls -lah /opt/CPrt-R81.10/conf/ip2country.csv
-rw-r----- 1 admin bin 15M Mar 9 13:59 /opt/CPrt-R81.10/conf/ip2country.csv
[Expert@CP-SC1:0]# ls -lah /opt/CPrt-R81.10/log_indexer/conf/ip2country.csv
-rwxrwx--- 1 admin root 15M Mar 9 13:58 /opt/CPrt-R81.10/log_indexer/conf/ip2country.csv

[DavidKL]

My R81.10 Take 66 Management server also doesn't auto update the file, it was upgraded from R80.30 to R81.10 - not sure if that contributes to the bug in anyway. 

[Jan_Kleinhans]

Have you found a solution?

[GHaider]

did a clean migration of my fw-managment server to R82, and now this script is no longer needed, it seems the file is updated automatically on 01:10h 🙂

ls -la $INDEXERDIR/conf/ip2country.csv $RTDIR/conf/ip2country.csv
-rw-r----- 1 admin bin 20258867 Dec 2 01:10 /opt/CPrt-R82/conf/ip2country.csv
-rwxrwx--- 1 admin root 20258867 Dec 2 01:10 /opt/CPrt-R82/log_indexer/conf/ip2country.csv

[the_rock]

Thats right. You can also see the same in R81.20

Andy

R81.20 lab:

[Expert@CP-MANAGEMENT:0]# ls -la $INDEXERDIR/conf/ip2country.csv $RTDIR/conf/ip2country.csv
-rw-r----- 1 admin bin 20258867 Dec 1 19:10 /opt/CPrt-R81.20/conf/ip2country.csv
-rwxrwx--- 1 admin root 20258867 Dec 1 19:10 /opt/CPrt-R81.20/log_indexer/conf/ip2country.csv
[Expert@CP-MANAGEMENT:0]#

R82 lab:

[Expert@R82:0]# ls -la $INDEXERDIR/conf/ip2country.csv $RTDIR/conf/ip2country.csv
-rw-r----- 1 admin bin 20258867 Dec 1 19:10 /opt/CPrt-R82/conf/ip2country.csv
-rwxrwx--- 1 admin root 20258867 Dec 1 19:10 /opt/CPrt-R82/log_indexer/conf/ip2country.csv
[Expert@R82:0]#