Finding a URL in all custom Application/Sites rather than just one

[Tom_Cripps]

Hi,

I have a script currently utilising the mgmt api to open a specific application/site and show all the URL's listed. I then can then grep a saved output to find a single URL. I look to add filtering built into the script but has anyone had any experience with looping over all application/sites to find a single URL?

Tom

[PhoneBoy]

I assume at a high level, it something like:
1. List out the custom applications using show application-sites
2. For each application listed, run the code you've already written.

[Tom_Cripps]

Makes sense in theory. Will wait to see if anyone else has any feedback.

[Tom_Cripps]

Hi Daemon,

Looking at this, show application-site appears to show too much or doesn't allow me to look at Custom app/sites we've already made.

Anyone in the community you know who could assist?

Thanks Tom

[Maik]

Not sure if I understand you correctly, but once you have created a custom application/site you can search for it via the "show application-site" command - you just need to have the name, uid or application id of the related object. So if you have the name and it is called "CustomTest" you an use the following command to receive your required Information; the contained URLs:

> show application-site name "CustomTest" --format json

{
  "uid" : "e43c6931-8810-482c-836a-ffc5a2a2750a",
  "name" : "CustomTest",
  "type" : "application-site",
  "domain" : {
    "uid" : "41e821a0-3720-11e3-aa6e-0800200c9fde",
    "name" : "SMC User",
    "domain-type" : "domain"
  },
  "application-id" : 0,
  "primary-category" : "Custom_Application_Site",
  "description" : "",
  "risk" : "Medium",
  "user-defined" : true,
  "url-list" : [ "www.test.org", "www.test.com", "test.biz" ],
  "urls-defined-as-regular-expression" : false,
  "groups" : [ ],
  "comments" : "",
  "color" : "black",
  "icon" : "Objects/application",
  "tags" : [ ],
  "meta-info" : {
    "lock" : "locked by current session",
    "validation-state" : "ok",
    "last-modify-time" : {
      "posix" : 1575322111151,
      "iso-8601" : "2019-12-02T21:28+0000"
    },
    "last-modifier" : "admin",
    "creation-time" : {
      "posix" : 1575322111151,
      "iso-8601" : "2019-12-02T21:28+0000"
    },
    "creator" : "admin"
  },
  "read-only" : false
}

If you want a list of all the custom application site objects that you have created you need to execute the command/query the api via "show application-sites" - this will return all related objects. Note; per default it returns 50 per call, this can be increased but should not exceed more than 200-300 per api call - in the best case you keep it at 50. In order to receive all objects you need to call this command over and over again and increase the related Offset parameters. More can be read in the API documentation. In parallel to that you also need to make sure to "take note" of all objects that have the primary-category of "Custom_Application_Site", aka all obejcts that were created manually. To achieve this you can simply write down their related uid in a list, that can later be iterated over with the show application-site command.

for identifier in customApplSiteUID_list

    do "show application-site uid identifier"

[Of course this is not real code, but it should give you an idea what needs to be done.]

Afterwards you have the output from above for all the required sites. Now you only need to take each of These Outputs and write the URLs into a list, which can be used for your searching purposes.

[Tom_Cripps]

Hi Maik,

Thanks for that. The second option is more inline with what i'm trying to achieve. My issue though, is identifying what is custom and what is not without having the ability to search against the primary category in the api call. I'll have to see if there is some other way of doing it without looking at the uid as this requires manually updating of the uid. 

Thanks for the insight.

Tom

[chymmmy]

Hello
Following with the thread. A question that I cannot see in the webs about a searching in the whole checkpoint.

I have a 90 custom application / application site.  So, in one o several applications exists an URL called "redhat_test".
How we can find in which application is placed ?

I cannot use wildcard to show all applications
smartserver>mgmt_cli show application-site name *
I cannot use a filter as grep in case set one by one application
smartserver>mgmt_cli show application-site name destyapp_wsus | grep redhat_test

Are there easy way to receive an screen output as
smartserver>mgmt_cli show application-site name * | grep redhat_test
destyapp_wsus "url-list" : [ " redhat_test
destyapp_lin "url-list" : [ " redhat_test
destyapp_shop "url-list" : [ " redhat_test

[PhoneBoy]

You cannot use a wildcard in an API call.
It's also not necessary as there is an endpoint that will list all the application-sites:
https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/show-application-sites~v1.9%20
Note that you may have to make multiple calls to this endpoint (using limit/offset for paging) to retrieve all results.

[Duane_Toler]

Following on from @PhoneBoy , you can use the API, but you'll need to be a little more creative with it.  If you have custom application sites, then you likely have them in a custom application group.  If not, put them in one.

(nb: i have a bash shell function named 'mcli' to be the curl syntax and things to handle all the HTTPS communication to the mgmt API server; it's not meant to be human-interactive; it just takes a JSON input and sends it via HTTP POST.)

First fetch your custom application site by name:  (here i have one named "Test_Site_Group")

echo '{ "name" : "Test_Site_Group" }' |mcli show-application-site-group
{
  "uid" : "9d36b14c-b7c9-4282-a5a4-19d83778c8f3",
  "name" : "Test_Site_Group",
  "type" : "application-site-group",
  "domain" : {
    "uid" : "41e821a0-3720-11e3-aa6e-0800200c9fde",
    "name" : "SMC User",
    "domain-type" : "domain"
  },
  "members" : [ {
    "uid" : "3d765aa4-2b3e-4d9e-a26b-26c20379da78",
    "name" : "Malicious_Sites",
    "type" : "application-site",
    "domain" : {
      "uid" : "41e821a0-3720-11e3-aa6e-0800200c9fde",
      "name" : "SMC User",
      "domain-type" : "domain"
    },
    "icon" : "Objects/application",
    "color" : "black"
  } ],
....
}

The ".members[]" key is a list of your custom sites.  This is where PhoneBoy said to iterate over that:

$ echo '{ "name" : "Test_Site_Group" }' |mcli show-application-site-group |jq -r '.members[].name' |while read app_site;
do
  echo $app_site
done

Malicious_Sites
...

The output is the list of custom application sites you would have.  Here, my customer happened to have one like yours, with a list of URLs.  I just stole theirs for this demo.  Thus, you'll then call your API again for each of the custom application sites in that site group.  Then search the "url-list" key with a JQ filter:

$ echo '{ "name" : "Test_Site_Group" }' |\
  mcli show-application-site-group |\
  jq -r '.members[].name' |\
  while read app_site; do
    echo '{ "name" : "'${app_site}'" }' |\
    mcli show-application-site |\
    jq -r --arg url_search "chromeupdates.online" 'select(."url-list"[]|contains($url_search)).name';
  done

Malicious_Sites

Again, the output is the name of the custom site that had an URL in the list that you wanted to find.  In this case, I passed a static string to JQ as an argument.  "chromeupdates.online" is the value of the variable in JQ as $url_search.  You can replace that string with whatever you want and however you want.

Ansible can do this, too, but with a little more work with  JMESQuery. 

If you intend to run this same thing locally on the mgmt server with mgmt_cli, you can, but just replace my forceful JSON with parameters to mgmt_cli.  Oh, and be sure to do "export MGMT_CLI_FORMAT=json"  before you run mgmt_cli.