The URL list inside the Custom Application/Site object is not being applied.

[dkzndkqh]

Here’s a natural English translation of what you wrote:

---

Currently, as seen from the nslookup, I have added the following domains to the URL list in a Custom Application/Site object:

gig-ai-g-prod-australiaeast-2-app-v4-tag.australiaeast.cloudapp.azure.com Address: 20.213.196.212

Aliases: dc.services.visualstudio.com

dc.applicationinsights.microsoft.com

dc.applicationinsights.azure.com

global.in.ai.monitor.azure.com

global.in.ai.privatelink.monitor.azure.com

dc.trafficmanager.net

australiaeast-global.in.applicationinsights.azure.com

gig-ai-prod-australiaeast-global.trafficmanager.net

For each domain, I have added three entries in the URL list, for example:
dc.services.visualstudio.com, *.dc.services.visualstudio.com, *dc.services.visualstudio.com.

Despite this, the policy containing this Custom Application/Site object is not being applied.

So I tested using regular expressions. To access gig-ai-g-prod-australiaeast-2-app-v4-tag.australiaeast.cloudapp.azure.com, I added the first domain in the chain, dc.services.visualstudio.com, to the custom object’s list as:

\/dc.services.visualstudio.com

\.dc.services.visualstudio.com

(Note: I did not remove the previous URL list entries that were not regular expressions.)

However, drop logs are still being generated starting from the first domain in the chain. If my understanding is correct, if the regular expressions for the first domain in the chain were being applied, the drop logs should appear for the second domain in the chain. Am I correct in thinking this?

---

current using : SG6200  R81.20SP JHT89 and management server : Smart1 5050 R81.20 JHT84

[PhoneBoy]

Can you provide a full log card via a screenshot? (Redact any sensitive details)

[the_rock]

So what exact domain fails? If you can give us a log example, like @PhoneBoy had asked, it would help.

Andy

[dkzndkqh]

It seems that, according to the logs, the URL Filtering blade detects first, and then the Firewall appears.

The URLs to be allowed are as follows. They seem to be the URLs required for using the Azure service.

• gig-ai-g-prod-australiaeast-0-app-v4-tag.australiaeast.cloudapp.azure.com
• gig-ai-g-prod-australiaeast-1-app-v4-tag.australiaeast.cloudapp.azure.com
• gig-ai-g-prod-australiaeast-2-app-v4-tag.australiaeast.cloudapp.azure.com
• dc.services.visualstudio.com

The screenshot shows how the regular expression was written, but could it be that I made a mistake in the regular expression? For now, since I don’t know what kind of subdomain might appear under the URL to be allowed, I specified it in the format like \/gig-ai-g-prod-australiaeast-0-app-v4-tag\.australiaeast\.cloudapp\.azure.com. And to allow only the main domain itself, I specified it like \.gig-ai-g-prod-australiaeast-0-app-v4-tag.australiaeast.cloudapp.azure.com. 

Would the format ^gig-ai-g-prod-australiaeast-0-app-v4-tag\.australiaeast\.cloudapp\.azure.com only match the exact domain gig-ai-g-prod-australiaeast-0-app-v4-tag.australiaeast.cloudapp.azure.com?

Going further, ultimately I’m wondering why it doesn’t take effect even when I add the URLs into the existing Custom Application/Site list, and also why it doesn’t work when I use regular expressions. Could it be because the URL list inside is too large? There are about 290 entries, and most of the domains configured inside are using *.

[the_rock]

Just add *cloudapp.azure* as custom app site and it will work.

Andy

[dkzndkqh]

When accessing the domain gig-ai-g-prod-australiaeast-0-app-v4-tag.australiaeast.cloudapp.azure.com, do I also need to register the actual SNI along with it?

[the_rock]

I dont have lab access atm, will check in the morning, but either way, if you use ordered layer with appc and urlf blades on or same as network layer, just create a rule with services as custom url object and add "cloudapp.azure" and see if it lets you check regular expression (used to be able to in R81.20), but may not in R82. Regardless if it does or not, that should work.

Andy

[G_W_Albrecht]

Have a look at https://regex101.com/, a good place for learning and testing RegEx !

[G_W_Albrecht]

Have a look at https://regex101.com/, a good place for learning and testing RegEx !

[PhoneBoy]

The hostname part of the URL is always matched on SNI, just FYI.
That applies even with full HTTPS Inspection enabled.
If you're having matching issues, that's where I'd start looking.

Yes, ^ anchors the expression at the beginning of the URL after https://.

[dkzndkqh]

In the screenshot log I attached, the SNI value appears as dc.services.visualstudio.com. However, even if I add this URL to the URL list, it does not apply. Without using a regular expression, is there a better way to register it than using .services.visualstudio.com or dc.services.visualstudio.com?

[PhoneBoy]

SNI is in a field called Resource, I believe (not the Destination). 

[Bob_Zimmerman]

In my research, the carat anchors before the "https", so you need to include the scheme in your expression.

[Wolfgang]

@dkzndkqh to use URL-Filtering on HTTPS websites you must use HTTPS inspection or the light version "Categorize HTTPS websites". With the light version the URLs have to be seen via the SNI.

Which application or service do you want to use ? Maybee a service from the "Updatable objects" can be used to allow instead of the the URL filter.

[dkzndkqh]

At present, I can only confirm that it is related to Azure services. I also do not know exactly which specific Azure service is being used.

[the_rock]

Have you tried using updatable objects to see if that makes a difference?

Andy

[Wolfgang]

Question again..... Is HTTPS inspection and/or "categorize HTTPS websites" enabled ? URLF does not work for encrypted websites witout this.

As mentioned by @the_rock  and me you can try with the "Azure Services" as destination. If you use these updatable objects you don't need HTTPS inspection.

[dkzndkqh]

Currently, I am using Categorize HTTPS, not Inspection. Is the method of using updatable objects similar to FQDN?