topic Re: Domain object failure in R80.10 (sk120558) in General Topics

Domain object failure in R80.10 (sk120558)

Kaspars_Zibarts — Mon, 29 Jan 2018 20:54:21 GMT

Just running pass if anyone else has come across this one

DNS resolves OK manually on CLI but the problem is that both SK and SR engineer wants kernel debug that potentially may overload the FW. Considering that this is business critical firewall potentially causing $100k loss every minute it is dead and it's remote, I'm very reluctant to run debugs. Asked support engineer to come up with something else but hit the stone wall (that's a topic i really want to start - when will CP will come up with better debugging )

Doing graceful cluster reboot (standby reboot > failover > new standby reboot) seems to have "fixed" it for now.

Anyone with better ideas regarding domain object "checks" / tricks in R80.10 before diving into kernel debug?

This is plain firewall cluster (5900) with only firewall and IA blades, nothing fancy.

Re: Domain object failure in R80.10 (sk120558)

PhoneBoy — Mon, 29 Jan 2018 22:26:56 GMT

Moving this to General Product Topics‌

It's possible the kernel DNS lookup timed out somehow before it got a response.

Which would explain why it worked when you checked on the appliance.

I'm guessing that would show on the debugs.

Re: Domain object failure in R80.10 (sk120558)

Kaspars_Zibarts — Tue, 30 Jan 2018 05:27:02 GMT

I guess my description was not good enough - it wasn't just a temporary issue but full stop on domain object based rules. They didn't work and logs were full with those alerts. Whilst manual lookup worked just fine.

Re: Domain object failure in R80.10 (sk120558)

PhoneBoy — Tue, 30 Jan 2018 14:53:29 GMT

That sounds like the DNS resolution process got hung/crashed somehow.

And yeah, we'd probably need some detailed debugs to see what's going on.

Re: Domain object failure in R80.10 (sk120558)

Kaspars_Zibarts — Tue, 30 Jan 2018 15:26:39 GMT

Turned out that we had it all over the place including VSX firewalls running R80.10 and regular ones. Since I saw it on standby cluster members too, I collected debug from one of them and it turned out to be the same bug as SK.

But I would like to bit more info on actual root cause as SK is very short on it. How come that most firewalls have got the same problem now - is it DNS specific? Is it actual object specific?

"Internal failure in DNS health check state of Domain Objects"

Re: Domain object failure in R80.10 (sk120558)

PhoneBoy — Tue, 30 Jan 2018 16:20:17 GMT

Looking at the various internal information I have access to, there's not much more than is in the SK.

The good news is that there is a hotfix for the issue.

Re: Domain object failure in R80.10 (sk120558)

Kaspars_Zibarts — Tue, 30 Jan 2018 16:33:54 GMT

Thanks for trying Dameon!

Re: Domain object failure in R80.10 (sk120558)

Kaspars_Zibarts — Wed, 31 Jan 2018 14:03:41 GMT

Couldn't just give up on it as I wanted the explanation. Eventually pulled together information from our DNS team that did some planned work on weekend which meant that DNS was not totally down but could have apparently had "slow" responses. And it looks like that's enough to kill the DNS cache for domain objects in majority of our firewalls.

I was able to replicate it in the lab (sort of) by changing DNS IP temporary to a dummy IP address. To accelerate the process I did cpstop/cpstart so FW started using new IPs which in turn would not respond. And it didn't take that long before I got the same alerts there. And it seems like it never recovers from it, until you cpstop/cpstart again the gateway.

In nutshell - if

you use domain objects and
have a DNS hiccup in the network and
start seeing DNS alerts in logs (like one above) and
don't have the hotfix

cpstop/cpstart on the gateway seems to restore the functionality as long as DNS is functioning again correctly.. In cluster case you may do graceful cpstop/cpstart on each member (standby/fail over/standby)

Re: Domain object failure in R80.10 (sk120558)

PhoneBoy — Wed, 31 Jan 2018 14:46:59 GMT

Appreciate your diligence in tracking this down.

Re: Domain object failure in R80.10 (sk120558)

phlrnnr — Thu, 15 Nov 2018 16:48:15 GMT

Does this bug exist in R80.20 as well? The SK only shows R80.10 as being affected, but I'm seeing similar symptoms in R80.20.

More specifically, I'm seeing something similar with R80.20 Mgmt and R80.10 + Jumbo 112 VSX GWs.

Re: Domain object failure in R80.10 (sk120558)

PhoneBoy — Thu, 15 Nov 2018 17:04:03 GMT

The gateway in your example is still R80.10, which is where the name resolution takes place.

Re: Domain object failure in R80.10 (sk120558)

phlrnnr — Thu, 15 Nov 2018 18:14:55 GMT

Ok. Well, the SK says it is fixed in R80.10, Take 42. Since this is running Take 112, I guess I'll reach out to TAC. thanks.

Re: Domain object failure in R80.10 (sk120558)

phlrnnr — Thu, 15 Nov 2018 18:32:14 GMT

Kaspars Zibarts, was TAC able to tell you what the hotfix actually did? Does it monitor the service and restart it if it crashes?

Re: Domain object failure in R80.10 (sk120558)

Kaspars_Zibarts — Thu, 15 Nov 2018 21:12:13 GMT

Unfortunately I couldn't find explanation in the SR even I asked for it. Just hotfixes

Re: Domain object failure in R80.10 (sk120558)

Ray_Xiao — Thu, 13 Jun 2019 12:51:05 GMT

Did anyone try to do dns flush?