DNS Data Overflow causing high CPU and IPS Bypass

[iva]

We are running R81.10 JHF 132 on Quantum 6600 appliances. We are hosting DNS services for the public internet in a DMZ on TCP/UDP 53. Some time ago, our IPS started preventing DNS Data Overflow (Response packet too long, potential buffer overflow) attacks on TCP/53. In combination with these attacks, IPS bypass is activated and CPU Load increases to >80%. The appliance stops responding for some time, causing outages.

The traffic pattern usually includes a relatively low numer of connections from distributed source IPs. To me if looks like a OS vulnerability exploited by attackers.

To prevent this, we have contacted CheckPoint support, and activated DoS features such as rate limiting and penalty box. However, due to the traffic pattern mentioned above, these mitigations are not completely effective. 

I am contacting the community, hoping to some more input on alternative mitigation methods regarding this specific attack. Maybe somebody has experienced the same type of attack and managed to find a solution?

[Chris_Atkinson]

Is the traffic accepted by implied rules or specific rules you have configured?

As a side how is the memory utilization throughout and do your UDP DNS (domain-udp) service objects in the policy use the default timeout values?