Post R80.10 Mgmt Upgrade - APP/URL filtering silently dropping traffic

[Matt_Taber]

Post management R80.10 upgrade things were fine after the 1st few policy pushes.  It wasn't until we installed database, and pushed policy we started seeing: "dropped by fwpslglue_chain Reason: PSL Reject: internal - reject enabled;" in fw ctl zdebug drop on our R77.30 clusters.  This is mainly HTTPS traffic that is being permitted by the FW blade, but dropped anyhow.

I found sk33328 which clears out $FWDIR/state directory to resolve policy corruption issues and is the same SK CP support has advised.  This is a nuclear option, however as both MGMT and gateways need to be cpstop'd.

Have any of you run into this issue before and did you have a solution other than what was described in this SK?

[Matt_Taber · Accepted solution]

After working with multiple CP support resources and finally with a Tier 3 tech, we determined that it was APP/URL filtering silently dropping the traffic.  Glad to have found the issue and not having had to take down clusters.

[Eyal_Rashelbach]

Hi Matt,

Following your update i've changed the title to reflect the issue as needed.

Also appreciate if you mark the thread as answered 🙂

[PhoneBoy]

Did TAC happen to explain why it was dropping?

[Matt_Taber]

I *believe* it was because HTTPS wasn't explicitly allowed in the APP/URL policy.  Behavior change from R77.30 MGMT to R80.10 MGMT possibly.

[KennyManrique]

Hi Matt,

I am having a similar issue with HTTPS traffic in R80.10. Does any hotfix was provided to you? CP mention any plans to add the solution for future HFA's?

Regards.

[G_W_Albrecht]

If i can read the explanation correctly, issue has been resolved when HTTPS was explicitly allowed in the APP/URL policy.

[KennyManrique]

I read that as "believe", not as final solution...For me, that's a workaround. In fact, I had to do the same for a customer a month ago after upgrade to R80.10.

It seems as an architecture error, so in enviroments where exists a Drop Any rule at the bottom of application layer; you must allow HTTPS before final rule for application traffic that should be already allowed explicitely right? This is a huge gap open to certain traffic not recognized as application.

Regards.

[Matt_Taber]

Yes, after HTTPS was fixed, we found other HTTPS traffic on non-standard port 443 was having the issue as well.  Very troubling indeed.

[G_W_Albrecht]

Afaik from CP TAC, APP/URL filtering rules should have no "Drop Any" rule as the last rule at all. Also, CP does recommend to remove/disable as many Accept rules in URLF/Application rules as possible. URLF/Application control accept rules serve no enforcement purposes, since any traffic which is not explicitly blocked will just be allowed. Such rules, however, do cause traffic to be matched on them - which causes high CPU usage.

So URLF/Application rules should just restrict unwanted traffic and let the rest pass. But of course i know that there may be special requirements that can not be fullfilled using that concept...

[Timothy_Hall]

Thanks for the followup Matt, when researching my book I spent a lot of time trying to find a way to disable APCL/URL filtering (and even Limit actions) "on the fly" to help isolate conditions such as this, and my eventual conclusion was that it is not possible.   APCL/URLF must be a bit too tightly intertwined with the Firewall blade; the Application Control and URL Filtering boxes must be unchecked on the gateway object and policy reinstalled to achieve this effect.

On the fly disablement is possible for IPS/Threat Prevention as covered in my CPX presentation.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com