How do you rollback an old policy?

[Dave_Hoggan]

Hi,

Apologies at resurrecting an old thread but I am having a bit of difficulty in understanding why a pretty fundamental feature of the product seems to have been made less useful.

One of the nice features of the old database revision control was that it allowed a 'snapshot' of not only policies, but objects to be made prior to a change. For example the change of an ISP the would potentially require not only multiple security and NAT rules to be changed, but also several changes to the objects database (the gateway itself, manual NAT objects etc.) Roll  back was simple, clean and effective.

With the new scheme we can only revert the rules; all object changes must be made manually at the risk of increased time for roll back and errors being made by the person performing the roll back. Am I correct that where is no way to roll back the objects other than manually?

Right now the only way to guarantee a complete "one button" roll back is to make use of the GAiA backup feature and this seems hugely excessive.

Am I missing something obvious?

Thanks,

Dave

[Tomer_Sole]

So my question to you is how often do you find yourself completely reverting your Security Management database.

From our investigations with customers, the biggest use case was recovering from a disastrous policy being installed. You can still achieve that with the "Installation History" page (see this page above) - the GW immediately gets the previous policy installed, while the Management still contains the most recent version of the changes made by the admin.

We assume that people create and change objects for a reason. And if network traffic degrades, it's probably better to work on top of the changes rather than recreating them from scratch. Especially with full audit logs telling the story. Does this still leave you with an unsatisfying daily experience? Please elaborate.

Either way, we will bring back an option to completely revert revisions in the Management database in our next releases.

[Dave_Hoggan]

Hi Tomer.

Having the gateway receive the last installed policy is all well and good, but if that policy refers to objects that have been modified then there are still going to be issues with business communication. Depending upon the scale of the change then this can take a significant amount of time (significant being relative to the fact that the business will be suffering the outage of one or more services) to fix by individually reverting each affected object.

I'm assuming that your investigations were primarily with end users and as such the majority of such changes would be small-scale with relatively few dramatic changes within their environment? Working for a reseller I get involved with the larger changes where a simple change in ISP can require 50 or so objects to be changed; or the move or a number of VLANs off a core switch to the firewall to name two recent examples.

So, whilst I do understand your view in the context of the run-of-the-mill daily rule changes, when it really matters - the more significant changes to the environment - the answer is unfortunately "yes", the omission of a reversion mechanism that provides the same "snapshot" and roll back features that DRC gave us is unsatisfying. When faced with a maintenance window (which is usually not long enough) I would rather spend the majority of the time trying to fix the issue knowing it is easy (and quick) to roll back, than having to allocate a significant proportion of the maintenance window for potential "return to operations" tasks.

Thanks,

Dave

[Tomer_Sole]

Dave Hoggan wrote:

Having the gateway receive the last installed policy is all well and good, but if that policy refers to objects that have been modified then there are still going to be issues with business communication. Depending upon the scale of the change then this can take a significant amount of time (significant being relative to the fact that the business will be suffering the outage of one or more services) to fix by individually reverting each affected object.

When installing a previous revision, the Gateway gets the rules and objects as they were compiled on the date in the past. Rest assure none of the changes in the Management database since that revision will affect that policy.

I'm assuming that your investigations were primarily with end users and as such the majority of such changes would be small-scale with relatively few dramatic changes within their environment? Working for a reseller I get involved with the larger changes where a simple change in ISP can require 50 or so objects to be changed; or the move or a number of VLANs off a core switch to the firewall to name two recent examples.

So, whilst I do understand your view in the context of the run-of-the-mill daily rule changes, when it really matters - the more significant changes to the environment - the answer is unfortunately "yes", the omission of a reversion mechanism that provides the same "snapshot" and roll back features that DRC gave us is unsatisfying. When faced with a maintenance window (which is usually not long enough) I would rather spend the majority of the time trying to fix the issue knowing it is easy (and quick) to roll back, than having to allocate a significant proportion of the maintenance window for potential "return to operations" tasks.

Thanks,

Dave

We've done visits to customers, partners and service providers, but we're always interested with fresh opinions. So going into this scenario - in the case of 50+ changed objects and rules making a degradation on the network security or performance, assuming you rolled back the changes on the GW's completely, what would typically be the next step to still apply that desired change of the policy?

( I want to be clear - we know that sometimes just cancelling the changes on the 50 object and starting fresh is better than making additional changes on top of what's already been done, and full revert will be available in the next releases, I'm just trying to open this specific example)

[Dave_Hoggan]

Hi Tomer,

The "next step" usually depends upon the customer and their appetite for risk.

With some of our customers the end of the change window can only have one of two outcomes: a fully successful change or a complete reversion. The next stage in these cases is to have the appropriate analysis and re-attempt the entire change in a separate window. Excessive? Not if you have SLAs with a dozen clients and every minute that you are down outside of an agreed window carries punitive financial penalties. So, it is not a question of being "better", but one of meeting the expectations set out. Also, simply rolling back the changes on the gateway (and not the manager) would not only be seen as not leaving things as how they were before the start of the maintenance window, more critically it would mean that there is a difference in the policy and objects on the manager and those on the gateway.

With other customers, hitting the end of the maintenance window with the gateway running the reverted policy and objects whilst the manager is partially restored to its previous configuration would be less of an issue, simply as the business is operational. Outside of the maintenance window the objects could then be manually reverted which, assuming that the engineer has a properly completed project document, would be more an issue of time than anything else.

I guess it depends on the customer but in my experience once the maintenance window is closed there is no "next step" other than to get the change review board to agree to a new maintenance window.

As I said, I can fully understand how having the policy/objects on the gateway revert whilst the manager is reviewed or reverted manually is a nice feature - for small scale changes. Say, a new service being brought online. But for working services that are business critical or changes that have been risk assessed as having potential impact on business operation there needs to be a process by which changes can be utterly, completely and quickly undone.

Thanks,

Dave

[Tomer_Sole]

Thanks for your feedback.

[Ashley_Black]

Hi

Working from the support desk, I have to concur with Dave. DRC has saved hours on numerous occasions instead of attempting to unpick what a customer has done to their latest policy that has gone disastrously wrong.

[Norbert_Bohusch]

I can agree with you that such a rollback option would be great in such specific situations and therefore I hope it will be integrated soon, but you have to see the other side of the feature!

If you have now not only 1 administrator carrying out changes during the day (because of policy-lock), but much more, you cannot easily revert to a database revision every time, because you would revert changes of others (you might not be aware of).

On the other hand you can save a lot of time now with R80.x with being able to prepare big changes in advance, as you can work on the changes without publishing them and you or others doing daily work in other sessions.

Therefore the change window could be used better and time for troubleshooting could be saved!

[Luis_Miguel_Mig]

From a change management perspective, I think that a full rollback   (policies and objects on both gateways and firewall managers)  to a revision version is simple, predictable and consistent.

I guess that it can be useful if the firewall manager can keep new objects that are still not applied to the gateway but this is not as important as been consistent with a full rollback I think.

When will the full rollback be released? Is it in the roadmap? I am planning to upgrade to R80.10 but this to me is a blocker.

[Daniel_Morin]

I concur, every major system should have means of rollback, whether that be a backup of system and/or configuration with means of restore, or a rollback mechanism outside of backups.  Rolling back the gateway is fine for an emergency requiring a quick reconnect, it cannot be the final answer, since as Tomer points out, the GW would be out of sync with the manager.  On the next policy install from the manager, you are back to the broken configuration.

Besides that, there needs to be a means of backing up policy packages and other components of the manager that can be restored, similar to the options available within GAIA.  Fortunately we run CP Management on a VM so our enterprise backup solution can back up CPM as a whole, but if it was physical, that could be a problem if the backup system didn't have an adequate agent to install to/interact with GAIA to perform a file or disk based backup.