Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Bob_Zimmerman
Advisor

Duplicate app/site categories?

I'm running R80.40 jumbo 92.

In setting up my client to understand application/site objects and the categories around them, I noticed something weird. At first I thought it was evidence I wasn't deduplicating data properly, but it isn't.

 

[Expert@LabSC]# mgmt_cli -r true --format json show object details-level full uid 00FA9E44-415A-0F65-E053-08241DC22DA2
{
  "object" : {
    "uid" : "00fa9e44-415a-0f65-e053-08241dc22da2",
    "name" : "Anonymizer",
    "type" : "application-site-category",
    "domain" : {...},
    "description" : "An intermediary which prevents Web sites from seeing a user's Internet Protocol (IP) address.  Anonymizers are often used to circumvent company network acceptable use policies to potentially hide non-appropriate behavior, however in the home setting they may be used to provide additional layers of anonymity and protect privacy. \n \nExamples:    http://www.anonymizer.com, http://www.megaproxy.com, http://www.hidemyass.com",
    "user-defined" : false,
    "groups" : [ ],
    "comments" : "",
    "color" : "black",
    "icon" : "Objects/category",
    "tags" : [ ],
    "meta-info" : {...},
    "read-only" : false
  }
}
[Expert@LabSC]# mgmt_cli -r true --format json show object details-level full uid 00FA9E44-4046-0F65-E053-08241DC22DA2
{
  "object" : {
    "uid" : "00fa9e44-4046-0f65-e053-08241dc22da2",
    "name" : "Anonymizer",
    "type" : "application-site-category",
    "domain" : {...},
    "description" : "An intermediary which prevents Web sites from seeing a user's Internet Protocol (IP) address.  Anonymizers are often used to circumvent company network acceptable use policies to potentially hide non-appropriate behavior, however in the home setting they may be used to provide additional layers of anonymity and protect privacy.",
    "user-defined" : false,
    "groups" : [ ],
    "comments" : "",
    "color" : "black",
    "icon" : "Objects/category",
    "tags" : [ ],
    "meta-info" : {...},
    "read-only" : false
  }
}
[Expert@LabSC]# mgmt_cli -r true --format json show application-site-categories limit 500 | jq -c '.objects[]|.' | grep Anonymizer
{"uid":"00fa9e44-415a-0f65-e053-08241dc22da2","name":"Anonymizer","type":"application-site-category","domain":{...}}

 

`show application-site-categories` only gives me the one, but `show objects` gives me that one and a different one. The descriptions are slightly different, but both show as `"type" : "application-site-category"`.

So ... uh ... what gives? What's the difference between these? Why don't both show up in `show application-site-categories`?

0 Kudos
5 Replies
the_rock
Advisor

Hm, maybe its just the categorization is set up for it on CP side, not sure...

Andy

0 Kudos
Bob_Zimmerman
Advisor

I suspect the included app/site objects might use different category objects for the primary category and for additional categories.

Or maybe this is some kind of artifact of an upgrade? My lab management database is relatively small (around 9k objects, and 4k rules). Maybe the included categories changed UUIDs at some point, and the old categories are hidden but not deleted?

0 Kudos
the_rock
Advisor

That could be true...AFAIK, but I could be wrong, so maybe someone from CP can confirm, IM pretty sure their categorization works only based on primary category...so say youtube.com could be categorized as entertainment (just making that up, for argument's sake, could be another category), but I believe that youtube.com/videos would be the same category as well.

0 Kudos
Bob_Zimmerman
Advisor

Just confirmed both UUIDs are present in an R80.20 jumbo 188 management and an R80.10 jumbo 275 management which are completely unrelated to my SmartCenter above, and completely unrelated to each other. That strongly suggests they are present by default.

0 Kudos
Bob_Zimmerman
Advisor

Did some more digging. It looks like the ones which are duplicated are used as both primary categories and additional categories. Here's some command output showing this. Spoiler tags because the output is long.

First, a list of all "application-site-category" objects' names, sorted, then deduplicated with a count of how many times each occurred:

Click to Expand
[Expert@LabSC]# grep "application-site-category" objects.json | jq '.name' | sort | uniq -c
      1 "Adds other software"
      1 "Alcohol"
      1 "Allows remote connect"
      1 "Allows remote control"
      2 "Anonymizer"
      1 "Art / Culture"
      1 "BitTorrent protocol"
      2 "Blogs / Personal Pages"
      1 "Botnets"
      2 "Browser Plugin"
      1 "Browser Toolbar"
      2 "Business / Economy"
      2 "Business Applications"
      1 "Child Abuse"
      1 "Cloud Services"
      1 "Communication Standard"
      2 "Computers / Internet"
      1 "Content Provider and Sharing"
      1 "Critical Risk"
      1 "Cryptocurrency"
      1 "Custom_Application_Site"
      2 "Download Manager"
      2 "Education"
      2 "Email"
      1 "Encrypts communications"
      2 "Entertainment"
      1 "Facebook Business"
      1 "Facebook Education"
      1 "Facebook Entertainment"
      1 "Facebook File Sharing"
      1 "Facebook Friends & Family"
      1 "Facebook Games"
      1 "Facebook Lifestyle"
      1 "Facebook Popular"
      1 "Facebook Sports"
      1 "Facebook Utilities"
      2 "Facebook Widgets"
      1 "Fashion"
      2 "File Storage and Sharing"
      2 "Financial Services"
      2 "Friendster Widgets"
      2 "Gambling"
      2 "Games"
      1 "General"
      1 "Gnutella protocol"
      2 "Google Plus Widgets"
      2 "Government / Military"
      1 "Greeting Cards"
      1 "Hacking"
      1 "Hate / Racism"
      2 "Health"
      1 "High Bandwidth"
      1 "High Risk"
      2 "IPTV"
      1 "IRC protocol"
      1 "Illegal / Questionable"
      1 "Illegal Drugs"
      1 "Inactive Sites"
      1 "Instant Chat"
      2 "Instant Messaging"
      1 "Job Search / Careers"
      1 "Lifestyle"
      1 "Lingerie and Swimsuit / Suggestive"
      2 "LinkedIn Widgets"
      1 "Low Risk"
      1 "Marijuana"
      2 "Media Sharing"
      2 "Media Streams"
      1 "Medium Risk"
      1 "Micro blogging"
      1 "Microsoft & Office365 Services"
      1 "Mobile Software"
      1 "MySpace Entertainment"
      1 "MySpace Games"
      1 "MySpace Lifestyle"
      1 "MySpace Popular"
      1 "MySpace Sports"
      1 "MySpace Utilities"
      2 "MySpace Widgets"
      1 "Nature / Conservation"
      2 "Network Protocols"
      2 "Network Utilities"
      2 "News / Media"
      2 "Newsgroups / Forums"
      2 "Ning.com Widgets"
      1 "Non-profits & NGOs"
      1 "Nudity"
      1 "Opens ports"
      1 "Orkut Entertainment"
      1 "Orkut Games"
      1 "Orkut Lifestyle"
      1 "Orkut Popular"
      1 "Orkut Sports"
      1 "Orkut Utilities"
      2 "Orkut Widgets"
      2 "P2P File Sharing"
      2 "Personals / Dating"
      1 "Phishing"
      2 "Political / Legal"
      1 "Pornography"
      1 "Port agility"
      2 "Real Estate"
      1 "Recreation"
      1 "Religion"
      2 "Remote Administration"
      2 "Restaurants / Dining / Food"
      2 "SCADA Protocols"
      2 "SMS Tools"
      2 "Search Engines / Portals"
      1 "Sends mail"
      1 "Sex Education"
      1 "Sex"
      1 "Share Files"
      1 "Share Music"
      1 "Share links"
      1 "Share photos"
      1 "Share videos"
      2 "Shopping"
      2 "Social Networking"
      2 "Social Plugins"
      2 "Software Downloads"
      2 "Software Update"
      1 "Spam"
      2 "Sports"
      2 "Spyware / Malicious Sites"
      1 "Stealth Tactics"
      1 "Streaming Media Protocols"
      1 "Supports File Transfer"
      1 "Supports IM"
      1 "Supports Streaming"
      1 "Supports VoIP"
      1 "Supports video/webcam"
      1 "Suspicious Content"
      1 "Tasteless"
      1 "Torrent Trackers"
      2 "Translation"
      1 "Transmits Information"
      2 "Travel"
      1 "Tunnels"
      2 "Twitter Clients"
      1 "URL Filtering"
      1 "Uncategorized"
      2 "Unknown Traffic"
      1 "Used for Web-Based Support"
      2 "Vehicles"
      1 "Very Low Risk"
      1 "Video Conferencing"
      1 "Violence"
      2 "Virtual Worlds"
      2 "VoIP"
      1 "Voice Mail"
      1 "Weapons"
      2 "Web Advertisements"
      1 "Web Based Instant Messaging"
      1 "Web Browser Acceleration"
      2 "Web Browser"
      1 "Web Browsing"
      2 "Web Conferencing"
      2 "Web Content Aggregators"
      2 "Web Desktop"
      2 "Web Services Provider"
      2 "Web Spider"
      1 "Windows Messenger protocol"
      1 "Yahoo Messenger protocol"

Next, a list of names in the "primary-category" field and how many times each is used:

Click to Expand
[Expert@LabSC]# grep "primary-category" objects.json | jq -c '."primary-category"' | sort | uniq -c
    194 "Anonymizer"
      6 "Blogs / Personal Pages"
     56 "Browser Plugin"
     24 "Business / Economy"
    816 "Business Applications"
     48 "Computers / Internet"
    109 "Download Manager"
      9 "Education"
     87 "Email"
      7 "Entertainment"
      1 "Facebook Widgets"
    917 "File Storage and Sharing"
      8 "Financial Services"
      4 "Gambling"
    209 "Games"
      2 "Government / Military"
      4 "Health"
    167 "IPTV"
    361 "Instant Messaging"
    311 "Media Sharing"
     39 "Media Streams"
    130 "Network Protocols"
     57 "Network Utilities"
     10 "News / Media"
      4 "Newsgroups / Forums"
    343 "P2P File Sharing"
      6 "Personals / Dating"
      1 "Political / Legal"
      2 "Real Estate"
    142 "Remote Administration"
      4 "Restaurants / Dining / Food"
    817 "SCADA Protocols"
     15 "SMS Tools"
     55 "Search Engines / Portals"
      9 "Shopping"
   1133 "Social Networking"
      8 "Social Plugins"
      5 "Software Downloads"
     32 "Software Update"
      4 "Sports"
     23 "Spyware / Malicious Sites"
      5 "Translation"
      5 "Travel"
    241 "Twitter Clients"
      1 "Unknown Traffic"
      3 "Vehicles"
    117 "Virtual Worlds"
     77 "VoIP"
      3 "Web Advertisements"
     31 "Web Browser"
      1 "Web Browsing"
    122 "Web Conferencing"
    120 "Web Content Aggregators"
     16 "Web Desktop"
     49 "Web Services Provider"
    375 "Web Spider"

Finally, a list of the names in the "additional-category" field and how many times each is used:

Click to Expand
[Expert@LabSC]# grep "primary-category" objects.json | jq -c '."additional-categories"[]|.' | sort | uniq -c
jq: error: Cannot iterate over null
     50 "Adds other software"
    976 "Allows remote connect"
    847 "Allows remote control"
    193 "Anonymizer"
     82 "BitTorrent protocol"
     12 "Blogs / Personal Pages"
     70 "Browser Plugin"
     32 "Browser Toolbar"
     24 "Business / Economy"
    848 "Business Applications"
    479 "Cloud Services"
     34 "Communication Standard"
     81 "Computers / Internet"
    158 "Content Provider and Sharing"
    196 "Critical Risk"
     18 "Cryptocurrency"
    113 "Download Manager"
     10 "Education"
     87 "Email"
   1154 "Encrypts communications"
     20 "Entertainment"
      5 "Facebook Entertainment"
      4 "Facebook File Sharing"
      3 "Facebook Friends & Family"
      2 "Facebook Games"
      2 "Facebook Lifestyle"
      1 "Facebook Utilities"
      1 "Facebook Widgets"
    932 "File Storage and Sharing"
     13 "Financial Services"
      4 "Gambling"
    213 "Games"
     40 "Gnutella protocol"
      2 "Government / Military"
      5 "Hacking"
      4 "Health"
   1550 "High Bandwidth"
    712 "High Risk"
    171 "IPTV"
     34 "IRC protocol"
    743 "Instant Chat"
    358 "Instant Messaging"
   3422 "Low Risk"
    335 "Media Sharing"
     55 "Media Streams"
   1555 "Medium Risk"
    806 "Micro blogging"
     51 "Microsoft & Office365 Services"
    528 "Mobile Software"
    130 "Network Protocols"
     66 "Network Utilities"
     13 "News / Media"
      5 "Newsgroups / Forums"
   1100 "Opens ports"
    349 "P2P File Sharing"
      6 "Personals / Dating"
      1 "Political / Legal"
    441 "Port agility"
      2 "Real Estate"
    150 "Remote Administration"
      4 "Restaurants / Dining / Food"
    817 "SCADA Protocols"
     16 "SMS Tools"
     60 "Search Engines / Portals"
    195 "Sends mail"
   1418 "Share Files"
    417 "Share Music"
   1824 "Share links"
   1278 "Share photos"
   1340 "Share videos"
     13 "Shopping"
   1153 "Social Networking"
      9 "Social Plugins"
      9 "Software Downloads"
     32 "Software Update"
      9 "Sports"
     24 "Spyware / Malicious Sites"
     62 "Stealth Tactics"
     79 "Streaming Media Protocols"
   1466 "Supports File Transfer"
    252 "Supports IM"
    735 "Supports Streaming"
    247 "Supports VoIP"
    241 "Supports video/webcam"
    146 "Torrent Trackers"
      5 "Translation"
   1214 "Transmits Information"
      7 "Travel"
    229 "Tunnels"
    241 "Twitter Clients"
      1 "Unknown Traffic"
     54 "Used for Web-Based Support"
      3 "Vehicles"
   1419 "Very Low Risk"
    160 "Video Conferencing"
    118 "Virtual Worlds"
     76 "VoIP"
     15 "Voice Mail"
     23 "Web Advertisements"
     59 "Web Based Instant Messaging"
      6 "Web Browser Acceleration"
     31 "Web Browser"
    122 "Web Conferencing"
    120 "Web Content Aggregators"
     18 "Web Desktop"
     59 "Web Services Provider"
    375 "Web Spider"
     40 "Windows Messenger protocol"
     35 "Yahoo Messenger protocol"

"Friendster Widgets","Google Plus Widgets", "LinkedIn Widgets","MySpace Widgets", "Ning.com Widgets", "Orkut Widgets" all have two objects and are not the primary category of any stock application/site objects. Not sure why there are two instances of these category objects.

"Web Browsing" is only used as a primary category and does not appear in the "additional-categories" field of any objects. There is only one category object with that name.

0 Kudos