URL Categorization - list of Artificial Intelligence sites

[Eric_Knopp]

We've just implemented a category block in our Application layer (R81.20). Before we implemented the block, we used the Application and Sites Log search to investigate which Artificial Intelligence (AI) sites were being categorized that our users were connecting to. We know there's the AppWiki, but that only shows up 148 Applications that are categorized Artificial Intelligence (AI). Now that the block is in place, we're playing catch-up with sites that are now blocked, but we did not know they were going to be.

I've searched through Checkmates and support to try to find a list of websites categorized as Artificial Intelligence (AI), but all I find is the reverse, being able to check a site's category. This is not helpful, we need to know what is being categorized. Where can we find that information? We cannot play whack-a-mole with these sites.

Thank you!

[the_rock]

Glad you asked, Im also looking for the same.

Andy

[the_rock]

Not sure if there is more official link/way, but I know you can check things as per below.

Andy

https://appwiki.checkpoint.com/appwikisdb/public.htm

 

[Eric_Knopp]

GitHub was blocked and not categorized as AI in that AppWiki! So that's why we really need something more robust of a list, instead of searching. Thanks for the recommendation. Maybe Checkpoint has an answer. I'll also reach out to our Diamond team.

[the_rock]

They may know, true. But, I will say this...I asked this exact same question before to PAN and Fortigate support, never got an answer. I even said to the guy once "Why is this a secret, arent customers entitled to this information?"...and there was about 10 seconds of silence and then he said "I will get back to you", which in a nice way literally means, buzz off buddy 🤣🤣

Anyway, its beyond me why vendors withhold this information, personally, makes no sense, but I suppose thats just how it is.

Andy

[PhoneBoy]

As an interesting, but relevant, aside, the URL filtering database in 2009 for one of the previous vendors we used was around 300mb and downloaded to the gateway.
If such a file were to be provided today, it'd likely be gigabytes...and be out of date before you even finished downloading it.

These days, I assume there is no longer "one database" and everything is looked up via APIs.
Most of our Threat Prevention blades operate on this premise (i.e. why ThreatCloud exists). 
Some of those APIs are likely to involve third party vendors (either directly or on the backend) that I presume are governed by technical and contractural restrictions.

[the_rock]

Okay, that explanation makes sense. I wish vendors could provide a link where those sites are listed, but guess that could be how knows how many sites...

Andy

[PhoneBoy]

There are two databases in play here:

• AppWiki, which are for applications we have specific App Control signatures for
• URL Categorization, which is for everything else web-related

Unfortunately, for a variety of reasons, it is not possible to provide a list of URLs in a given category.

[the_rock]

I cant, for the life of me, understand what those reasons are. And Im NOT talking just about Check Point, Im talking about any vendor out there. I really dont see any logical reasoning as to why companies cant disclose that information...

Andy