Custom Application - regular expression

[abihsot__]

Hi there,

Gateway R80.40 JHF156 with full https inspection.

Not sure what I am doing wrong. I want to match specific domain (test.example.com) AND any subdomains, so according to sk106623 and sk165094, I create custom application with regular expression enabled:

\/test\.example\.com - to match exact domain

\.test\.example\.com - to match any subdomain

But it doesn't match test.example.com

If I go one level down and remove "test" portion - it works, but this time it is being matched by "subdomain" regular expression, so my suspicion is that specifically "\/example\.com" doesn't work as intended.

\/example\.com

\.example\.com

Registered a ticket for TAC, but at this point they are more guessing, so I was hoping to check here if anyone had similar issue and maybe used different regex.

[PhoneBoy]

Recommend you review the following sk, which should help: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[abihsot__]

Thanks for reply, however this is exactly the same SK I listed. I am defining custom application as instructed by the table:

[Bob_Zimmerman]

One seriously important note: \/example\.com would also match "superlegitsite.phish/example.com". Matching is performed against the entire URL, including scheme, so to be safe, you must anchor it like this:

^https?://example\.com/

To allow zero or more subdomains:

^https?://([^/]+\.)*example.com/

To require subdomains, change the asterisk to a plus.

[G_W_Albrecht]

Have a look at https://regex101.com/, a good place for learning and testing RegEx !

[abihsot__]

thanks for that, but I prefer to stick with recommendations by Checkpoint rather than using my own regex. Depending how Checkpoint implemented URL parsing some definitions can lead to performance degradation. 

[the_rock]

Just curious, what is exact domain you tried? Reason I ask is because me, I never follow that sk at all. What I do and it works 100% of the time is this. Say I want to whitelist ANYTHING youtube, I just add custom app site and do *youtube*

You follow exact same example for any website out there...say *news* or *google*...etc

Also, something to keep in mind is this as well...with https inspection enabled, you may need to do the same in https inspection policy too.

[abihsot__]

ouch... You should be very strict with custom URL regex definitions.

See sk106623, this is exactly the problem you might have. If you whitelist *example.com, xxxexample.com will match too. 

Also, don't you get warning about performance impact when adding * ?

[the_rock]

Yea, of course there is warning, but thats there simply for not following the sk itself, I mean, I even had 3 TAC people tell me to do it the way I described, since we could not make it work with any available method from the article, so really no point spending 10 hours on the phone for something that may not function : - )

What I did took literally 2 minutes and it worked!

[abihsot__]

Gotcha. The thing is, that some phishing sites are created to contain valid names, like newyoutube.com and it can slip through the filtering.

Forgot to mention, my case is actually not for whitelisting, but for bypassing ssl inspection instead. I want to bypass repo1.maven.org and any of its subdomains.

[the_rock]

K, not sure what time zone you are in, but we can do remote later and see whats going on. Im in EST (GMT - 5)

[abihsot__]

Thanks for the offer, but there is no urgency here and I applied workaround meanwhile. The sole purpose of this thread is to make sure checkpoint recommendations and regex syntax is up to date.

[Marcel_Gramalla]

The sk's are beeing updated shortly (at least according to TAC) as the parsers are also different between the different blades. They recommended using this website https://regex-generator.olafneumann.org/

It's especially useful when you want to do more than just the domain eg test.example.com/test/abc

However the example you mentioned should work for Access Control Rulebase.  

[Sorin_Gogean]

Hello, 

Keep in mind that there was an SK stating that RegExp are CPU intensive and there was a recommendation to use that with precaution.  (not sure if it was sk106623)

Secondly, why you need regexp, as if you would use *.mydomain.com it would match all subdomains - or am I wrong.

In your case, for test.mydomain.com you should only use test.mydomain.com and that's all . 

No RegExp no nothing....

Thank you, 

[abihsot__]

Hi there,

I am just trying to use what was suggested by Checkpoint in the form of multiple SKs. Indeed you have the option to define custom app without regex, however that was not my initial question.

If that's only documentation issue, that is an easy fix, however even TAC seems struggling to give definite answer. If "best practices" are confusing, I see why people just go with their own regex definitions, which could lead to questionable enforcement.

[the_rock]

Unfortunately, this is one of those things where different people take different approach. I agree with you though, wish there was one definite answer/method as far as defining those things.

[Bob_Zimmerman]

The non-regular-expression matching expression "*.mydomain.com" turns into ".*\.mydomain\.com" when the object is sent to the firewall. That's dangerous, as it matches "sub.mydomain.com.superlegitsite.phish" and "superlegitsite.phish/sub.mydomain.com".

[Sorin_Gogean]

Understood, but still the translated/"converted to" RegExp would be much more simpler than the complex ones that they address in SK's.

Like you confirmed RegExp would be still used for matches, as it's easiest way to match URL's .

In order to prevent some of those unwanted matches, we could writhe *.mydomain.com as *.mydomain.com/ ???? 

That would cut some of the unwanted matches.... 

We're not using too many Custom Applications as we preferred to start using the FQDN objects. Still not perfect according to others, but for us is quite OK 🙂 .

Maybe there could be written an script or smth that could emulate the load that the system will get when using an complex RegExp compared with simple one... or like @abihsot__ requires, a clearer document/guide. 

Simple: ^https?://([^/]+\.)*example.com/

Complex: ^(https?:\/\/)?([^\/.]+?\.)*?leading2lean\.com(:[0-9]+)?(\/|$)

Thank you,

PS: I remember reading in the past (and few minutes ago) about "Application Control Signature Tool", but I never went using it.

[Bob_Zimmerman]

Optimizing regular expression match performance is a fairly complicated topic. The short version is to minimize your use of . and character classes, and to make your repeated matches non-greedy.

Non-greedy matching is pretty simple. Regular expression matching engines try to match the whole string, then if that fails, they step back one character and try to match the whole string again, repeating until they get a match or fail. This is called greedy matching, because each repeated item will match all repetitions in a row. Adding a question mark after a repetition causes that repetition to be matched non-greedily. A +? will match one character, then try to match the rest of the expression. If it fails, it will match two characters, then try to match the rest of the expression. The match expands from the start rather than contracting from the end. For example, "[a-zA-Z0-9]+?\." is faster to match than "[a-zA-Z0-9]+\." is.

[PhoneBoy]

I’m pretty sure the expression matched starts with the host (i.e. what’s immediately after https://).
Which means if you want to match example.com (and ONLY example.com), you’d use something like: ^example\.com($|\/)

That should match example.com as well as any other URL on example.com.