topic Re: UDP DNS utilise the most CPU and concurrent connection in General Topics

UDP DNS utilise the most CPU and concurrent connection

Abeja_huhuhu — Sun, 03 Dec 2023 02:21:35 GMT

Hi Guys,

i'm not sure if this is normal or not. but as of today we are troubleshooting one of our customer firewall and we notice that UDP DNS is taking up most of the CPU processing and from almost 200K+- concurrent connections 180K+ is being used by UDP. Looking for some comment and suggestion from all the masters. We are currently doing some checking on what is causing the memory to utilize almost 70%. the appliance is using 6700 with 32GB of RAM installed inside the appliance.

Free -h

total used free shared buff/cache available
Mem: 31G 10G 5.3G 10G 15G 8.4G
Swap: 31G 3.0M 31G

fw ctl pstat

Virtual System Capacity Summary:
Physical memory used: 73% (19813 MB out of 27113 MB) - below watermark
Kernel memory used: 9% (2501 MB out of 27113 MB) - below watermark
Virtual memory used: 63% (17217 MB out of 27113 MB) - below watermark
Used: 17217 MB by FW, 36414 MB by zeco
Concurrent Connections: 197791 (Unlimited)
Aggressive Aging is enabled, not active

Kernel memory (kmem) statistics:
Total memory bytes used: 4064425705 peak: 17248058149
Allocations: 2378340323 alloc, 0 failed alloc
2235864245 free, 0 failed free

Cookies:
3511454616 total, 89080 alloc, 89080 free,
2167360 dup, 2206143026 get, 1969436521 put,
1807612677 len, 95659764 cached len, 23567 chain alloc,
23567 chain free

Connections:
981379508 total, 11899190 TCP, 964579454 UDP, 4900744 ICMP,
120 other, 456 anticipated, 24810 recovered, 197797 concurrent,
2034871 peak concurrent

Fragments:
560707 fragments, 276569 packets, 12 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
305076803/0 forw, 309191099/0 bckw, 1318730867 tcpudp,
4044131 icmp, 552888828-967582277 alloc

Sync: Run "cphaprob syncstat" for cluster sync statistics.

currently only below blades are enabled.

Firewall, IPSec VPN, Mobile Access.

Firewall version is R81.20 with HFA 26

Any suggestion

Re: UDP DNS utilise the most CPU and concurrent connection

PhoneBoy — Mon, 04 Dec 2023 21:12:55 GMT

If Mobile Access is enabled and serving clients, that could be the reason.
If your clients have to traverse the gateway to reach the DNS server, that could also cause this problem.

What are the exact symptoms you're concerned with?
That amount of memory utilization isn't unusual and is likely due to caching.

Re: UDP DNS utilise the most CPU and concurrent connection

Chris_Atkinson — Tue, 05 Dec 2023 00:31:54 GMT

Do you allow DNS traffic to source / destination any in your security policy?

Which version is this and do you have DNS tunneling protections enabled (Anti-bot, IPS etc)?

Re: UDP DNS utilise the most CPU and concurrent connection

Abeja_huhuhu — Tue, 05 Dec 2023 00:55:33 GMT

Hi Admin,

the usage for mobile access is not so heavy as it is only being use by us to access our customer network. roughly less then 5 users are active.

Basically, we just upgrade the firewall from R81.10 to R81.20. Previously when using R81.10. the firewall memory are only utilizing around 60% and went down when the traffic is low. However, after upgrade to R81.20 we notice that the firewall memory being utilize up to 90%+ and most of the time cause traffic drop. and i notice that UDP traffic is taking most of the utilizations. that is why i'm asking if it is normal to see UDP traffic that high.

Re: UDP DNS utilise the most CPU and concurrent connection

Abeja_huhuhu — Tue, 05 Dec 2023 00:57:26 GMT

Yes, we do have that. currently we enable firewall and mobile access. we plan to enable IPS but since the memory is not stable we disable it first.

i will try and see if your suggestion works for my environment.

thanks @Chris_Atkinson

Re: UDP DNS utilise the most CPU and concurrent connection

Chris_Atkinson — Tue, 05 Dec 2023 01:06:10 GMT

Recommend also restricting the rules allowing DNS traffic so it is less broad in that case.

I assume that there is an internal DNS server that users should be querying rather than external/public.

Re: UDP DNS utilise the most CPU and concurrent connection

Abeja_huhuhu — Tue, 05 Dec 2023 03:34:44 GMT

hi @Chris_Atkinson ,

i follow the tip 7 just now and can see concurrent connection drop from 180K to 80K. cpu utilization also coming down now. still monitoring the situation first. yes, our customer do have internal DNS server to query outside. but they also allow internal to query specific external DNS server. just not sure why during using R81.10 this issue does not arise. Only see this when upgrade to R81.20.

thanks for the recommendation.

Re: UDP DNS utilise the most CPU and concurrent connection

Timothy_Hall — Tue, 05 Dec 2023 12:59:07 GMT

If you have a cluster, you may also want to consider turning off state synchronization for whatever service you are using to match UDP/53 traffic (usually domain-udp), as this will save quite a bit of CPU resources (and a bit of memory) no longer trying to sync the rapid-fire, short-lived DNS recursive lookups.

Re: UDP DNS utilise the most CPU and concurrent connection

Abeja_huhuhu — Tue, 05 Dec 2023 14:40:52 GMT

Hi @Timothy_Hall ,

i try to do that just now. but it seems like the DNS server are not able to fully query domain name. there are certain domains that are not cache giving error. after re-enabling it back it seems like back to normal. currently i'm using Load Sharing unicast.

Re: UDP DNS utilise the most CPU and concurrent connection

Timothy_Hall — Tue, 05 Dec 2023 14:53:09 GMT

Ah for Load Sharing you won't want to turn off sync of DNS in the event of asymmetry through the cluster, I assumed you were using HA. My bad.

Re: UDP DNS utilise the most CPU and concurrent connection

Abeja_huhuhu — Tue, 05 Dec 2023 23:14:14 GMT

Hi @Timothy_Hall ,

yes, that is what i'm thinking. no problem. it is a good suggestion. i can use it if using HA after this. previously it is running on HA mode.