DNS problems possibly due to firewall issue

[SG22]

Hello everyone,

We have 3 DNS servers, one in my office and 2 in another location. Firewall is doing DHCP, in Firewall configuration DNS in mu location is primary and DNSes in another location are secondary and tertiary. 

Recently we started to experience problems with DNS. It takes forever to resolve queries. It takes a minute to open webpage, it takes 2-5 minutes to log in to workstation. Nslookup doesn't work. 

I temporarily changed DNS on my PC to google DNS, and it started to work. So I thought it is DNS problem. First I changed primary DNS on my workstation to be one from another location, but nothing changed. Then I changed Forwarders in DNS settings, cleared cache, but still same problem.

So I rebooted firewall and after firewall reboot, everything started to work without any issues for a couple of hours. After couple of hours same problem. So I rebooted firewall again, still just a temporary solution.

In other office we have same firewall, and there everything works without issues. DNS resolves queries. But even when I put their DNS as a primary on my workstation, or any workstation in my office, we have a problem.

Do you have an idea what can be issue? Please have in mind that I'm just junior without any firewall experience, so try to explain it to me like I'm an idiot.

[_Val_]

How do you know the FW is in fault? Any logs to prove it?

[SG22]

Well, I don't know. It is just a guess because it started to work after reboot.

No, I don't. That is why I started the topic, I don't know where to look and what to do to exclude firewall as possible cause.

[the_rock]

@SG22 ...thats why we are here to help you. So, lets start with basics...if you search the logs in dashboard and filter for say IP of your dns server and action drop, do you see anything when issue is happening? You can also run from fw expert mode this command -> fw ctl zdebug + drop | grep 53 (thats for dns port) or fw ctl zdebug + drop | grep x.x.x.x (just replace with dns server IP) and observe the results. 

[SG22]

No, no drops. And for fw ctl zdebug i get obsolete command. Do you know alternative command?

[the_rock]

Can you send exact debug command you ran?

[_Val_]

Which version of FW are you running?

[SG22]

Checkpoint 5200

[the_rock]

Val meant software version, not model. Is it R80.30, R80,40, R81? Something else? What jumbo?

Just run cpinfo -y all and send the output.

[SG22]

Oh sorry. I also get "Deprecated command" as a resault. But I'm pretty sure we use R80.40

[the_rock]

Ok, message me directly. Lets do remote session, I have time today. I really want to see this behavior for myself.

[_Val_]

"fw ctl zdebug" is not a deprecated command on R80.40. A screenshot maybe?

[the_rock]

Hey @_Val_ ...I just did zoom with @SG22 and expert mode was not set, since firewall was reconfigured, so we did that and fw zdebug ran fine.

We actually tested it with one of DNS server's IP addresses and only saw drops on port 135 on clean up rule, which is expected. Other than that, nothing dropped for port 53. We then ran curl_cli command and it showed correct output. To me, that would logically indicate its not a firewall problem, as it appears its resolving correctly from firewall side. 

[SG22]

I wasn't in expert mode. That is why it didn't work. @the_rock helped me a lot, but to him, it doesn't seem like a firewall issue.

[the_rock · Accepted solution]

Yea...personally, it seems like firewall is behaving like it should. You know what I did @SG22 ...since we spoke about 25 mins ago, I logged into 2 customers' environments and did exact same commands with their internal DNS servers and output was literally the same, no drops related to port 53 and also curl_cli gave correct output as well.

Since you mentioned there was power outage about 2 weeks ago, I have a feeling that caused this problem, if it started happening right after. Now, tracing it to specific "culprit" might be a bit of a challenge. Personally, though I know this would be a drag to do in big environments, I would start by process of elimination...so whatever device is eliminated as a problem, there is less things left to be cause of the issue. 

[Timothy_Hall]

fw ctl zdebug drop needs to be run from expert mode.

[_Val_]

That's appliance model. What's the SW version? Run "fw ver" command on the console

[_Val_]

Also, do you have a VPN between your location and the other one?

[SG22]

Yes we have VPN

[_Val_]

If you do, please check that if DNS traffic is using VPN, to communicate between DNS servers on both sites. It might be, your VPN S2S tunnel fails after two hours, and this breaks DNS. Is any other site to site communication is affected, when DNS fails?

[SG22]

The problem is only from our side. They can reach us without issue, but we can't reach them.

[the_rock]

@_Val_ is correct...I mean, I get what you described here, but its hard to draw logical conclusion that this is a firewall problem. If you reboot the firewall, great, we know it works for a bit, but still does not prove fw problem. Can you post any logs, captures when issue is happening? Try do pings on the firewall at the same time, because the response in mili seconds will give us a good idea. 

Andy

[SG22]

So, here is what I can tell you.

No DNS errors in Event viewer on server.

I used DNS Query Sniffer on my workstation and on my DNS server. 

On workstation: First 5 queries fail always and 6th one passes. It is always a tertiary server that resolves queries. No matter which one I use as tertiary.

On the server: All queries are passing (around 150 ms for uncached ones). But also I have a problem opening web pages on DNS server.

Ping in internal network I around 1 ms (doesn't matter if I use hostname or IP address)
Ping to other location is around 30 ms (I tried to ping external IP of the Firewall and a couple of servers)

Nslookup doesn't work on workstations
Nslookup works on DNS server, but only if you ping site first (and it is really slow)
 

[the_rock]

Let me ask you this, by the way, thanks for that explanation, thats very helpful! What happens if someone connects say via CP vpn endpoint client and tries to run these tests, is result exactly the same? Because if yes, that might be something to do with optional parameters on the gateway properties for vpn. Are you using right dns servers/dns suffix in gateway settings? If you navigate to dns and hosts from web UI page, you can check it there.

Andy

[SG22]

Users on endpoint client don't have any issues. DNS servers are ok, those are the one we use and suffix is ok.