Dynamic Block Lists for Check Point firewalls

[Daniel_Husand]

I have cooked together some further improvements on Check Points 'block TOR' scripts and built a small service around it. This is not an official Check Point function/product and is provided by me in my spare time.

At this moment the following blocklists are implemented:

• OpenBL
• Emerging Threats: Known Compromised Hosts
• TOR exit nodes
• BruteforceBlocker
• Blocklist.de All
• Talos
• Dshield

The feeds are downloaded, sanity checked and then published on cpdbl.net for free. I am currently running all lists on two separate clusters without any noticeable performance hit. Of course ymmv so all feedback is appreciated. If you want to try it out go to: https://cpdbl.net

Screenshot of the interface:

Gateway details:

These scripts utilize the rate limiting policy in SecureXL. Therefore blocking is done in fastpath and should not impact performance noticably.

Connections from IPs listed in the activated blocklists are only blocked INBOUND. Outgoing communications are currently allowed. I have roadmapped a toggle for this.

VSX is not supported for now.

Workflow:

The server(cpdbl.net) downloads all the lists nightly and

• Validates that all entries are valid IPs.
• Baselines the lists, makes sure a list does not suddenly grow enormously.
• Publishes the lists for the clients to download.

The client:

• Downloads fresh lists every 12 hours
• Times out entries in the block-table after 12 hours, hence if cpdbl.net is unavailable all entries will be removed at this time.
• Validates that only entries containing numbers and "-" are read into the system. (to stop possible code injection)
• Installs validated entries into blocking tables and waits for 12 hours before starting over again.

To monitor the blocked IP addresses:

R77.30:

In SmartView Tracker, search for "SecureXL message: Quota violation".

R80:

In SmartLog, search for "blade:Firewall Alert".

[Christian_Hurta]

Is there a way to use this with a proxy or does it need to have direct access from the gateway? Talking about R77.30

CP Dynamic Block Lists  is a killer, and I will be adding a customized internal url feed as well in addition to the existing ones on the script, and will modify the script to call that additional feed. 

[Dilian_Chernev]

Scripts need to be touched a bit to work on R80.20.

/opt/CPshrd-R80/bin/ path in all files should be replaced with /opt/CPshrd-R80.20/bin/

[Fire_Verse]

This is now opendbl.net

[Kaspars_Zibarts]

Absolutely awesome work Daniel! We love it! As we love dynamic objects 🙂

I "stole" your code (sorry!) and made VSX version that can be executed via crontab or manually if desired and all required protections listed in config file as follows

[Expert@vsx1-ext:0]# cat blacklist.conf
blocklistde-all
bruteforce
etknown
malwaredomain
sslblock
talos
tor-exit
zeustracker

It has hard coded directory set in /home/admin/dynamic_objects as all our dynamic objects are handled there but you can change it yourself of course. Not as pretty from UI point of view.

You will need to supply VS number when running script, i.e ./blacklist.sh 3

#!/bin/bash
# VSX version of the opendbl tool https://opendbl.net/
# based on version R80-0.5

. /opt/CPshared/5.0/tmp/.CPprofile.sh
source /etc/profile.d/vsenv.sh
fwv=`fw ver | awk {'print $7'}`
opendblv="0.5"
vsid=$1


# Update log
cd /home/admin/dynamic_objects
echo "`date` *** Starting update ***" >> blacklist.log
echo "   VS-$vsid" >> blacklist.log

# Download all lists via VS0
vsenv 0
while read line; do

  url=`echo "https://opendbl.net/lists/${line}.list"`
  curl_cli -s --cacert opendbl.crt --user-agent "$fwv $opendblv" --retry 10 --retry-delay 60 $url | dos2unix > ${line}.blacklist

done < blacklist.conf

# Implement all lists on desired VS
vsenv $vsid
while read line; do

  # Create arrays with max size of 2000 of IP pairs
  y=0; z=0; todo=()
  while read ip; do
    if ! [[ "$ip" =~ [^0-9.-] ]]; then
      todo[$y]+=" $ip $ip"
        if [ $z -eq 2000 ]; then
          z=0
          let y=$y+1
        else
          let z=$z+1
        fi
    fi
  done < ${line}.blacklist

  # Purge fully existing dynamic objects
  listname=`echo $line | awk -F- '{print $1}'`
  dynamic_objects -do dynob_blacklist_${listname}
  dynamic_objects -n dynob_blacklist_${listname}

  # Update with new IP lists from each array
  for i in "${todo[@]}" ; do

    dynamic_objects -o dynob_blacklist_${listname} -r $i -a
  done

  # Update log
  let x=y*2000+z
  echo -e "      $x \t - $listname IPs set" >> blacklist.log

done < blacklist.conf

echo "*** Update finished ***" >> blacklist.log
echo >> blacklist.log
rm -f *.blacklist

[mickrose555]

Hello Daniel,

I don't have the skills to implement this safely but I would really want such dynamic IP blacklist on my cluster of R77.30 (with the management already upgraded to 80.X, the rest will follow ).

Can someone do it with me like a freelance or prof services? The alternative is my vendor but it's summer and everything is slower here in Switzerland.

thanks a lot,

mike

[Kurtis_Johnson]

Would anyone be able to comment if these Lists not already be monitored by Anti-Bot or Anti-Virus reputation feeds?

[mickrose555]

Here is the Forti URL to check such malicious IP:

https://fortiguard.com/learnmore#botnet

These 2 malicious IPs are not in there but in some of the block lists, like:

https://opendbl.net/lists/blocklistde-all.list

141.98.80.67

185.211.245.198

Someone can share other search engines maybe.

[Ted_Serreyn]

OK, so is this still the way to do things now that we have the ability to use external IOC feeds?

Open for discussion either way, just trying to think about improving things, and handling multiple feeds.

I  believe that I am going to either an IOC or a dynamic block list for stuff that I am seeing with my clients to share data among them anonymously.

Ted Serreyn

[Luis_Miguel_Mig]

The scripts at http://opendbl.net/opendbl-r80-0.52.tar seem not to be finished for dshield, right?
Has anybody  configured dshield with dynamic objects?
It seems that opendbl.sh doesn't contain the option for dshield and then ip-opendbl-include.sh doesn't parse ip ranges.

[Kaspars_Zibarts]

It's because dshield list is published as a range instead of single IP and update script cannot handle a range

194.147.140.0-194.147.140.255
92.63.196.0-92.63.196.255
45.155.205.0-45.155.205.255
167.248.133.0-167.248.133.255
92.63.197.0-92.63.197.255
78.128.113.0-78.128.113.255
46.161.27.0-46.161.27.255
94.102.51.0-94.102.51.255
193.27.229.0-193.27.229.255
192.241.224.0-192.241.224.255
89.248.168.0-89.248.168.255
138.99.216.0-138.99.216.255
92.118.160.0-92.118.160.255
92.118.161.0-92.118.161.255
192.241.223.0-192.241.223.255
83.97.20.0-83.97.20.255
185.153.197.0-185.153.197.255
45.146.165.0-45.146.165.255
80.82.77.0-80.82.77.255
89.248.165.0-89.248.165.255

this part needs re-writing to accept ipFirst-ipLast format in the list. Might look later today. Meetings right now 🙂

    while read ip; do
        if ! [[ "$ip" =~ [^0-9.-] ]];
        then
				todo[$y]+=" $ip $ip"
					if [ $z -eq 2000 ]
					then
						z=0
						let y=$y+1
					else
						let z=$z+1
					fi
        fi
     done

[Kaspars_Zibarts · Accepted solution]

This one did the trick for me:

  while read ip; do
    if ! [[ "$ip" =~ [^0-9.-] ]]; then
      if ! [[ "$ip" =~ "-" ]]; then
        # For single IP list
        todo[$y]+=" $ip $ip"
      else
        # For a range firstIP-lastIP format
        ip=`echo $ip | sed 's/-/ /'`
        todo[$y]+=" $ip"
      fi
      if [ $z -eq 2000 ]; then
        z=0
        let y=$y+1
      else
        let z=$z+1
      fi
    fi
    echo "${todo[@]}"
  done 

So dynamic object contents look ok now:

object name : dynob_blacklist_dshield
range 0 : 45.146.165.0           45.146.165.255
range 1 : 45.155.205.0           45.155.205.255
range 2 : 46.161.27.0            46.161.27.255
range 3 : 78.128.113.0           78.128.113.255
range 4 : 80.82.77.0             80.82.77.255
range 5 : 83.97.20.0             83.97.20.255
range 6 : 89.248.165.0           89.248.165.255
range 7 : 89.248.168.0           89.248.168.255
range 8 : 92.63.196.0            92.63.197.255
range 9 : 92.118.160.0           92.118.161.255
range 10 : 94.102.51.0            94.102.51.255
range 11 : 138.99.216.0           138.99.216.255
range 12 : 167.248.133.0          167.248.133.255
range 13 : 185.153.197.0          185.153.197.255
range 14 : 192.241.223.0          192.241.224.255
range 15 : 193.27.229.0           193.27.229.255
range 16 : 194.147.140.0          194.147.140.255

[Luis_Miguel_Mig]

I added a couple of sanity checks. You may also want to include your public ip addresses.

fwv=`fw ver | awk {'print $7'}`
opendblv="0.52"
y=0
x=0
z=0

function validateIP()
{
local ip=$1
local ip2=$1
local stat=0
local privip=1
local badip=1
if [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
OIFS=$IFS
IFS='.'
ip=($ip)
IFS=$OIFS
[[ ${ip[0]} -le 255 && ${ip[1]} -le 255 \
&& ${ip[2]} -le 255 && ${ip[3]} -le 255 ]]
badip=$?
fi

if ! [[ "$ip2" =~ ^(192\.168|10\.[0-9]|172\.1[6789]\.|172\.2[0-9]\.|172\.3[01]\.) ]];
then
privip=0
fi

if [ $privip -eq 0 ] && [ $badip -eq 0 ]; then
stat=1
fi
return $stat
}

function convert {
while read ip; do
if ! [[ "$ip" =~ [^0-9.-] ]];
then

if ! [[ "$ip" =~ "-" ]]; then
# For single IP list
validateIP $ip
vip=$?
line=" "$ip" "$ip
else
# For a range firstIP-lastIP format
ip1=`echo $ip|cut -d '-' -f 1`
ip2=`echo $ip|cut -d '-' -f 2`
validateIP $ip1
vip1=$?
validateIP $ip2
vip2=$?
vip=$((vip1*vip2))
line=" "$ip1" "$ip2
echo "ip $ip ip1 $ip1 ip2 $ip2 vip1 $vip1 vip2 $vip2 vip $vip line $line" > /var/log/secfeed.2.log
fi
if [ $vip -eq 1 ]; then
todo[$y]+="$line"
if [ $z -eq 2000 ]
then
z=0
let y=$y+1
else
let z=$z+1
fi
fi
fi
done

dynamic_objects -do openDBL_$type
dynamic_objects -n openDBL_$type

for i in "${todo[@]}" ;
do
dynamic_objects -o openDBL_$type -r $i -a
done
}

function update {
curl_cli -s --cacert $CPDIR/conf/opendbl.crt --user-agent "$fwv $opendblv" --retry 10 --retry-delay 60 $url | dos2unix | convert
echo "$(date): Security Feed $type update" > /var/log/secfeed.log
}

[Luis_Miguel_Mig]

by the way, does anybody know if a dynamic object is persistent / survives a reboot?

[Kaspars_Zibarts]

I know that I have rebooted standby cluster member and dynamic objects survived there 🙂 I think clusterxl syncs them via tables but not 100% sure if you rebooted both cluster members

[Luis_Miguel_Mig]

Thanks.
Interesting idea about the clusterxl. I guess they are not in sync lookin at this:

[Expert@fw1:0]# ls -l /opt/CPsuite-R80.40/fw1/database/dynamic_objects.db
-rw-rw---- 1 admin root 1084018 Jan 22 11:43 /opt/CPsuite-R80.40/fw1/database/dynamic_objects.db

[Expert@fw2:0]# ls -l /opt/CPsuite-R80.40/fw1/database/dynamic_objects.db
-rw-rw---- 1 admin root 1102273 Jan 22 04:23 /opt/CPsuite-R80.40/fw1/database/dynamic_objects.db

[Freco_Wong]

It really useful and I succeed to setup in non-production with self-defined dynamic blocklist, thanks!

However, I encounter other issue, could anyone can help?

How to ignore errors if incorrect IP address in the IP blocklist?

for example: 

-------

1.1.1.1

2.2.2.2

3.3.3

-------

I found the dynamic_objects will clear all data if one incorrect IP address inside instead of ignore the incorrect IP address.

Regards

Freco

[Kaspars_Zibarts]

Interesting - which of the public lists had incorrect IP included? Just curious.  I thought they would have made sure that IPs are correct before pushing out lists. 

[Freco_Wong]

It is possible if update by human, so I am asking how to minimize the impact instead of clear all data once problem triggered