R80.30 URL filtering blocking allowed categories

[mark239]

Hi

I recently rolled out a pair of small appliances to two sites.

The web filtering policy for a particular user group is layered, and it has an allow list and the next rule is a drop all, with block message. HTTPS scanning is enabled with the cert rolled out. (I have also tried breaking the layers and having the standalone accept rule and then the standalone drop all rule after it)

On one site this works perfectly.

On another site, regularly (Every day or at least every other day) from early in the morning the firewall starts blocking all requests to anything categorised 'Computers/Internet' (Which is an allowed category) and a lot of things stop working. There are no failed category updates in the system log (Before the upgrade this same behaviour occurred, but we had updates failed and then database failed to reload so i suspected this initially). It's like the allow rule is being completely ignored. User auth is working, as the user name is logged in the log entry with the message the site was blocked as it belongs to the computers/internet category.

The only way to stop this is to remove the drop rule after the allow for this user group, Once you re-enable it and install the policy it will be fine again until the next time it happens out of the blue.

I previously upgraded the appliance from R80.20, as they were getting an HTTPS inspection error around certificate length (>1000) that the fix seemed to be upgrade to R80.30.

Any ideas?

[PhoneBoy]

There is no downloaded database for URL categorization, all lookups are done via the cloud.
What is the exact rule that is supposed to allow the traffic?
When the traffic is blocked, what rule is it hitting?
What are examples of the site(s) in question?
Also, maybe the issue isn't the URL categorization, but the gateway is failing to do LDAP lookups on users for some reason?

[mark239]

Ah, I assumed there was a DB due to the system events about installing application/url filtering database versions (And the old errors about failing to update and failing to reload DB)

The exact rule is basic. Its from a user group, to internet and allow certain categories.

The rule immediately below is block everything else. Block rule is currently disabled.

LDAP I assume to be ok - as a username is listed with every log.

Examples of sites being blocked are 

entrust.net (blocked as business/economy) - an allowed category

google.com - blocked as search engines (allowed category)

etc etc.

[abihsot__]

Hello,

what is the setting for "fail mode"  - blades -> appl/URLF > general > fail mode ?

When traffic is blocked, in the logs if you check "matched rule" tab, what is the number of rule?

[mark239]

The matched rule is the block rule, immediately below the allow rule.

I have already set fail-mode to open at the start of the problems, hoping that would fix it (It didnt)

Thanks

[abihsot__]

So there are two options, you don't hit allow rule because of source not matched access role or not matching category.

From one ticket with TAC, I was advised to put categories directly to the policy instead of creating custom application group and using it. I wasn't convinced at that time and for me it was just stupid, however it is working fine now. If your custom application group is not super lengthy, maybe you can try that.

[PhoneBoy]

The user is communicated via AD Query, Identity Collector, etc.
LDAP is done from the gateway specifically to look up groups for the given user.
If LDAP is failing for some reason, then you would not be matching the Access Role in your allow rule.
It would explain why you're seeing the behavior you're seeing.

[abihsot__]

In fact on a second look, I think LDAP is working fine, because exactly the same access role is used in rule 9, so there is something wrong with matching app/urlf category.

[PhoneBoy]

The original poster is experiencing issues on one gateway, but not other.
The categorization should be the same and we can confirm this by looking at the log card of the dropped connection.
That pretty much leaves LDAP (or something with the lookup process) as the only culprit.

[abihsot__]

Where do you see log card of dropped connection? I fail to find this screenshot... 

[PhoneBoy]

It wasn't provided, but I surmise, based on the original poster's description, that's what we'd find.

[abihsot__]

oh, I see... But OP mentioned that rule 9 was matching, which has exactly the same access role as rule 8...

[mark239]

I'll upload a few examples later - thanks so far guys for the time/consideration