URL Filtering FEED

[elicro]

Like Thread Feeds which can be downloaded using ioc_feeds I was wondering if there is an option to download a URL Filtering feed.

I have implemented everything that is needed for a thread feed to work in my 1530 device.

The only missing things are:
downloding (curl)
and validating the content.

It's pretty simple and I was wondering why it was not implemented in this 1530 NGFW?

* https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[G_W_Albrecht]

Afaik, URL Filtering is done using CP ThreatCloud information only. In GAiA OS, you can use feeds for Anti-Virus and Anti-Bot as explained in sk132193. In Embedded GAiA OS not even this is possible - see OS in sk132193 !

But why would you use feeds for URLF if you have an operating URLF service blade ?

[elicro]

@G_W_AlbrechtThanks for the sk.
I understand a bit better but if I have a 512MB RAM EdgeRouter Lite (https://www.dipole.co.il/product/ubiquiti-edgerouter-lite/) that can do that I do not see any reason in the world the any embedded GAIA cannot do that.
The decision to "lock" this feature from the device is not technical but Business/Enterpirse based.
ie: "If I can sell a device that costs a specific sum and get from the client a specific one, why should I provide him with a fully featured product in less money??"

That How I see it since I have a product that does everything I need and want...
I still need a real example and not docs, the docs are full of .. other things which I don't want to mentioned.

[G_W_Albrecht]

Even GAiA does not do it, so GAiA Embedded also can not do it. You have to choose yourself the product that fits you and your purposes best....

But EdgeRouter Lite has not much of a firewall nor ABot, AV, IPS - not to mention Theat Emulation or Zero Day MW Protection.

[elicro]

@G_W_Albrecht You are absolutely right!!!
This is why I wrote my own proxy, and it runs ontop of the Gaia embedded device... just not as CheckPoint made it.
The platform is nice, the webui is nice but I was thinking that CheckPoint is the best company in the world and specifically it's an Israelly company.
I assume that on an Israeli spaceship or plans or drones CheckPoint would be a star.(That is of course if you would or Dafna would agree)

I didn't wanted to write what I wrote^^^ but if the TAC(what is this acronym  exactly?) wasn't able to understand the situation and read it in 4 seconds then I was thinking about
starting to be a kinder garden teacher instead of being a tech expert.
It might even be better since the appliance has a lifespan of 3-5 years while teaching someone how to be smart can last until he will not need this world anymore.

[PhoneBoy]

TAC = Technical Assistance Center.

On the non-SMB appliances, in R81, we have a feature called Generic Data Center objects.
It doesn’t do URLs but it’s the same basic idea you’re talking about.
To do something similar for URLs would be an RFE.

With a centrally managed SMB, you could add URLs in bulk to a Custom Application/Site object via the API.
Locally managed SMB appliances don’t have a REST API.

[elicro]

@PhoneBoyWhat is RFE?

Currently with the latest R20.20.X GA it seems that the logs shows when a reload happens to a specific degree.
It's pretty simple to add a daemon that will add bulk Custom Application/Site object via an API. I can write one and publish it.
I wrote a DB named SquidBlocker which was meant replace to some degree SquidGuard, However only hospitals can get some benefit from this Damon.
What I still don't understand yet is how could a thread prevention system can apply rules and only 10 minutes later being used.
I maintain couple FW's and the delay I have seen in production is critical... it's not realistic in this era and the soon IPv6 era to apply rules in such a speed.

It's hard for me to see that such a REST API was not embedded into the system.
It requires:

- username+password / token / other solution

- couple very specific endpoints

Currently it's possible via clish so why no? From what I understood there is not money in this... did I understood right?

[PhoneBoy]

The REST API is part of our on-premise central management (a Smart-1 Appliance or a VM), which can manage policy on SMB appliances.
This will also address your "when will the policy be active" question since SmartConsole shows you when the policy installation (an explicit action) has completed.

I am not aware if there is a plan to include management API support on locally managed SMB appliances.

[elicro]

@PhoneBoyand what about cli?
This:

[Expert@Gateway-ID-7F9B74D9]# cpstat fw

Policy name: local
Install time: Tue Feb 9 11:26:18 2021

is this an indication that the rules need to apply?

Also what is the cause of the ttl for the rules to be applied?

When the fw_configload finish like this

# /opt/fw1/bin/fw_configload
Preparing Manifest (Modern Policy) files... Done.
FW:
Compiled OK.
Preparing Unified Policy... Done.

Does it mean that the rules should work?
I am not sure I understood exactly what these commands do.

In iptables/nftables and/or a simple proxy I can understand, but with your FW it's like you have created a whole new OS and Kernel which works..
The main issue now is that it took 10 minutes to apply.

Will continue on email.

[PhoneBoy]

Security enforcement is done in our own kernel modules that are very different from iptables and the like.
The policy (Access and Threat Prevention) is actually compiled from the configuration and loaded into our kernel modules.
For our non-SMB gateways, and when SMB gateways are managed by external management, there are explicit actions that take place:

• Make policy configuration changes (either SmartConsole or API)
• Publishing the configuration changes (required because of multiple administrator support)
• Install policy to relevant gateways, which actually compiles the configuration into the code that is run in the various kernel modules

On a self-managed SMB appliance, which use a similar code base, but different implementation, there is no concept of concurrent administrators, so no publish action is required.
However, the rest of the architecture is more or less the same.
Change to Access, Threat Prevention, VPN, or other policy requires compilation and installation before it is active.
While there's no actual "Install Policy" action you take in the WebUI or CLI, I can assure you it is happening.

It would be highly inefficient to recompile and reinstall the policy after each change in the WebUI or via the CLI.
I assume, and your experience seems to suggest, that we check every so often for configuration changes and only if there are changes, compile and install the policy.

Given that your current issue seems to be "I have to wait 10 minutes before the policy is active" I assume you want a way to force this.
I believe you're on the right track.

fw_configload appears to compile the basic access policy.
There are a couple of public references to it, one of which is: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 
Given that you execute it when you modify certain .def files, which are not in the UI but affect the security configuration, it's a pretty safe bet that's what the command does.

As there are other commands under $FWDIR/bin for other parts of the config (App Control, URL Filtering, QoS, NAT, etc), fw_configload may not be enough on its own to effect the necessary changes.
Given these commands aren't documented, I'm not precisely sure if/when they need to be executed. 
Given your use case, it could be one or more of: appi_configload, appi_cmi_configload, urlf_configload
There may be others I do not know about. 

cpstat fw simply tells you when the access policy was last loaded to the kernel module and some additional details (similar to the fw stat command I mentioned earlier).

All of that said, leveraging external management is probably the way to go here if you are managing more than a couple of these appliances.
This could be one or more of (depending on your precise requirements):

• Smart-1 Appliance (either physical or virtual)
• Smart-1 Cloud
• SMP (Also Cloud-Based, included in the purchase price of a 1500 series)

This would provide other benefits (ensuring the same policy is configured on every gateway, centralized logging/reporting, API support, etc).
It would also provide a much more deterministic policy installation process.

[elicro]

@PhoneBoyI have seen some of these binaries and I have seen the sources of some of these being used.
I didn't had enough time to research each and every one of them.
I assumed that I would receive some introduction to the device before I am buying it, maybe a demo of what's included in the price.

The fact that I need to learn the device by myself (while compared to others is pretty good..) is something I would like to avoid.
I have sent an email but didn't got any response to it.
A basic "this mail was read and we will contact you in 3 days" makes the difference between being a part of the process to being a "guest" in the process.

This SMP interesting but I don't know if it was included in the price I paid...
From what I remember there was a price for it when I spoke to someone in your company.

[PhoneBoy]

SMP is included as part of the most recent SMB appliances 1500/1600/1800.
See here for requesting your portal: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...
See also here for relevant API commands: https://sc1.checkpoint.com/documents/latest/SmpAPIs/#introduction~v1%20 

And, yes, it can also be purchased separately for older SMB appliances.

[elicro]

@PhoneBoyI will check the SMP in 3 days and will see how it is compared to what I have now.

[elicro]

@PhoneBoy OK lets leave API aside and use CLISH:

The next is a script with comparison to the current ruleset of the customized app-name

#!/usr/bin/env bash

DEBUG="1"
DRY_RUN="0"

APP_NAME="$1"

URL="http://f-gogs.ngtech.home/NgTech-Home/tls-bypass-lists/raw/master/010-GeneralTLSInspectionBypass"

if [ -z "${APP_NAME}" ];then
        echo "Missing App Name"
        exit 1
fi

if [ "$2" == "check" ];then
        DRY_RUN="1"
        echo "Running in dry run mode" >&2
fi

if [ ! -z "$3" ];then
        URL="$3"
fi

TMP_DOWNLOAD_FILE=$(mktemp)

wget "${URL}" -O ${TMP_DOWNLOAD_FILE}
RES=$?

if [ "${RES}" -gt "0" ];then
        echo "Error Downloading file from URL: \"${URL}\""
        rm -v "${TMP_DOWNLOAD_FILE}"
        exit ${RES}
fi

TMP_CLISH_UPDATE_FILE=$( mktemp )

TMP_CURRENT_CONFIG_FILE=$( mktemp )

TMP_CLISH_TRANSACTION_FILE=$( mktemp )

clish -c "show configuration"|egrep "^set application application-name \"${APP_NAME}\"" > ${TMP_CURRENT_CONFIG_FILE}

while read line; do
        if [ "${DEBUG}" -gt "0" ];then
                echo -n "DEBUG LEVEL 1: Working on regex: " >&2
                echo ${line} >&2
        fi
        echo "set application application-name \"${APP_NAME}\" regex-url true add url \"$line\"" >> ${TMP_CLISH_UPDATE_FILE}
done < ${TMP_DOWNLOAD_FILE}

DIFF=$(diff "${TMP_CURRENT_CONFIG_FILE}" "${TMP_CLISH_UPDATE_FILE}" )

DELETE_OBJECTS=$(echo "${DIFF}" |egrep "^-set " |awk '{print $7}')

for object in ${DELETE_OBJECTS}; do
        echo "set application application-name \"${APP_NAME}\" remove url ${object}" >> ${TMP_CLISH_TRANSACTION_FILE}
done

echo "${DIFF}" |egrep "^\+set " |sed -e "s@^\+set @set @g" >>  ${TMP_CLISH_TRANSACTION_FILE}

cat "${TMP_CLISH_TRANSACTION_FILE}"

if [ "${DRY_RUN}" -eq "0" ];then
        clish -f "${TMP_CLISH_TRANSACTION_FILE}"
fi

echo "Finished Transaction"
echo "Cleaning up files ..."

rm -v "${TMP_DOWNLOAD_FILE}"
rm -v "${TMP_CLISH_UPDATE_FILE}"
rm -v "${TMP_CURRENT_CONFIG_FILE}"
rm -v "${TMP_CLISH_TRANSACTION_FILE}"

Example of usage:

/storage/scripts/update-cron.sh "GeneralTLSInspectionBypass" check "http://f-gogs.ngtech.home/NgTech-Home/tls-bypass-lists/raw/master/010-GeneralTLSInspectionBypass"

Tell me if this seems reasonable enough for a case.. And I am expecting an answer... 

GITHUB LINK: cron-example-with-diff.sh 

[elicro]

@G_W_Albrecht 

An external threat feed is as simple as:

#!/usr/bin/env bash

wget http://f-gogs.ngtech.home/NgTech-Home/tls-bypass-lists/raw/master/010-GeneralTLSInspectionBypass -O /tmp/010-GeneralTLSInspectionBypass

CURRENT_CONTENT=$(clish -c "show application application-name \"GeneralTLSInspectionBypass\""|egrep -v "^(description:|$|application-urls:|Categories:|application-id:|application-name:)" |awk '{print $1}')

function add_regex() {
        echo "$2"|grep -x -F "$1" >/dev/null
        RES="$?"

        if [ "${RES}" -eq "1" ];then
                if [ -f "/storage/regex-ready-appliance" ];then
                        echo "This appliance is ready to be used with regex"

                        echo -n "Adding regex: "
                        echo $1

# Some fixes are required for clish to be able to add EOL
#                       clish -c "set application application-name GeneralTLSInspectionBypass regex-url true add url $1"
                else
                        echo "This appliance is not ready to be used with Regex in cli"
                fi
        fi
}


while read line; do
        echo -n "Working on regex: "
        echo ${line}
        add_regex "${line}" "${CURRENT_CONTENT}"
done </tmp/010-GeneralTLSInspectionBypass

The above example is missing the option to remove using some differential comparison and couple other things like rules naming but it's as simple as this.

And better response later then in a sec sometimes: The EdgeRouter Lite is a basic nDPI FW but has some limitation in the AV indeed.
About the IPS and ABOT I am not yet convinced it doesn't have the software capablities.
The way I see it is only missing the right proxy software.
I do have the software to do that but yet to publish it....
Zero Day MW Protection on my device, well I am not sure it's out of the proxy specification with some API.
I have yet to write every software CP wrote but with enough resources it's doable.
The funny thing is that Palo Alto wanted me to patch their systems and I refused and the reason is that one person cannot hold the entire system on himself.
I assume that above 100 Servers on-top of one person is normal in my industry but not alone in most cases.