There was a 2 hour outage, we found SK140612, where it is stated the proxy process sends requests to all configured DNS servers, plus that the first responding DNS servers answer is used and the other answers are discarded. So when the first responding server returns a bogus response, the rest is discarded and no proper responses will be received in time.
In the WSDNSD file we see a lot of these messages:
wsdns_monitor_init: registering active monitoring
wsdns_monitor_init: got process ID: 15958
wsdns_monitor_set_signal: called with signal=10
wsdns_monitor_init: registered monitor. RC: 0
wsdns_signals_init: daemon signals initiated successfully
wsdns_resolver_init: called
main: daemon entering loop...
Failed to open file 'fw_cmd.tmp': Too many open files
[wsdnsd 23719 4136200400]@GW1[17 Dec 14:14:57] wsdnsd: Mon Dec 17 14:14:57 2018
wsdns_monitor_init: registering active monitoring
wsdns_monitor_init: got process ID: 23719
wsdns_monitor_set_signal: called with signal=10
wsdns_monitor_init: registered monitor. RC: 0
wsdns_signals_init: daemon signals initiated successfully
wsdns_resolver_init: called
main: daemon entering loop...
Failed to open file 'fw_cmd.tmp': Too many open files
[wsdnsd 17889 4135504080]@GW1[17 Dec 17:04:46] wsdnsd: Mon Dec 17 17:04:46 2018
Weird thing is that at the exact same time of the last entry the internet access was restored.
Regards, Maarten