Hi guys, I would like to share our root cause analysis for the Geo-location update issue from July 10th:
On July 10th at 9:16 UTC our automatic geo-location update service issued an update package which included only small fraction of the geo-location information, resulting in Security Gateways around the world getting only partial information. This caused either allowing or disallowing more traffic than intended.
Throughout July 10th we started receiving reports from customers about this issue and on July 11th we reverted the update to the July 9th version while investigating the issue. We have since then found the bug and know its root cause. This was not caused by a software update or a faulty deployment. For transparency we would like to share the details of the incident:
In 2018, 9 years after creating the geo-location update service, we improved the service to become updated daily (instead of monthly updates as was until then). This was done to accommodate our customers’ wishes mostly in response to new US OFAC (Office of Foreign Assets Control) regulations. We created an algorithm to produce hybrid geo-location updates from two data sources – one that is more accurate but updates once a month, and another that is less detailed and possibly less accurate but updates once per day – taking the good parts from each data source and leaving the bad parts out. This algorithm proved itself – we saw a drastic decline in the number of complaints about geo-location misclassification and received positive feedback. Last week’s incident was caused by a daily update which assigned an IPv4 range to the country of Laos. Laos’s official name is “Lao People’s Democratic Republic”. Embarrassingly that name caused a sequence of events that caused our service to publish a valid but very partial geo-location package. Although this service is well monitored – the failure was in a blind spot and we did not get an alert prior to the service tickets.
We are very sorry this happened, we realize this service means a lot to our customers. We are doing a full review of the script monitoring system to make sure no other blind spots exist and make sure this doesn’t happen again.
