R80+ Change Control: A Visual Guide

[Timothy_Hall]

As mentioned in the article Revisions Management in R80.x, I do have an informal writeup I use when teaching CCSA R80.10 that helps summarize how R80+ Change Control and Revisions are handled for the inevitable questions that arise in class.  This document is a sprucing up of those notes complete with some new screenshots.

Absolutely no way this document would have been possible without the incredible contributions of Tomer Sole and in particular these articles with content contributed by him:

How to revert a Policy or discard changes? 

Revisions Management in R80.x 

How do you rollback an old policy? 

What follows is merely a roll-up of Tomer's content from an operational perspective, with some new screenshots I have put together.  I hope you find it useful.

Preface: Install the "Change Report" SmartConsole Extension (R80.30+)

Back in R77.30, there was an SMS feature called "SmartWorkflow".  One of the more useful elements of this feature was a "Change Report" which could clearly and succinctly show the object and policy differences between different sessions.  Based on feedback from the CheckMates User Community, the Change Report functionality is back in R80.30+,  but in the form of a SmartConsole Extension.  In other words this feature is NOT available by default in R80.30 and later, and you must manually add it to your R80.30+ SMS in order to use it. 

Trust me: You want to install this Change Report functionality if you have R80.30 or later (but it is not supported on a standalone SMS/gateway combination); the instructions to do so are here:

https://community.checkpoint.com/t5/API-CLI-Discussion-and-Samples/Show-changes-from-session-gt-from...

sk166435: How to view changes performed between revisions

Once this extension has been successfully installed a new button called "Changes" shows up on a variety of different SmartConsole screens.  Clicking it presents a popup window that looks like the following; the next three screenshots are scrolled through a single Change Report and its results:

The new "Changes" button will also appear in many more places other than the Security Policy layers screen as shown above, particularly on screens where Audit Logs can be potentially viewed.  A sampling:

Note: The "Changes" button will not be shown in any of the subsequent screenshots in this article or mentioned again.  It is up to YOU to keep an eye open for the "Changes" button that will appear on various screens and take advantage of it, assuming you have installed this very useful SmartConsole Extension.

Part 1: What are you about to do?

You are in the process of making changes in the SmartConsole and are unsure (or have lost track of) what you have done due to one of your coworkers constantly interrupting you.  If still in an unpublished session, you can see what changes are pending by enabling the Session Pane (only available in R80.10+ management) like this:

This will create a new pane on the far right of the SmartConsole where you can see pending unpublished changes:

This information can be very helpful when deciding to publish or discard a session.  Another way to find pending unpublished rule changes in your current session is to look for "Edited" Access Control rules, indicated by default using a purple line in the Smart Scrollbar:

Note that the colored lines in the Smart Scrollbar will also by default show rule locks currently held by other administrators (dark grey) and search results (yellow) by default.  You can even make section titles (light grey) and a selected/highlighted rule (blue) show up as well:

For more information about the Smart Scrollbar and some other great tips for efficiently navigating a large rulebase in the SmartConsole, see this post:

What are some of the tips and tricks for jumping between rules in the rulebase?

So you have now published your session and think you are ready to install policy to the gateway.  A very good habit to get into prior to installation is looking at how many changes you are about to make:

Along the top of the screen, you should ALWAYS look at how many sessions and by how many administrators will be part of what you are about to deploy on the gateway.  Note that this is the total number of changes made in the SMS config since policy was last installed to this particular gateway, and not every change counted here is necessarily part of this gateway's security configuration or will impact how it operates.  If you see sessions and changes that are unfamiliar or unexpected though, it is a very good idea to hit the View Changes button to see exactly what will be included:

Along the top of the screen is a summary of all the different published sessions whose changes will be included if you install policy to the gateway.  As shown above you can highlight one of those sessions, and then select the Audit Logs tab to see a very detailed list of exactly what changes were made in that particular highlighted session. 

Let's assume that everything looks OK and you proceed to install policy to the gateway.

Part 2: The Panic Button

Your phone is ringing nonstop, people are pounding on your door, and it has all gone horribly wrong!  Some very bad change got implemented when you pushed policy to the gateway at the end of the last section and it is impacting production traffic.  You need to fix what you did RIGHT NOW.  Thankfully the Installation History screen will be your savior in this case:

The first Installation Date shown above (1/15/2018) represents the most recent policy push (which is probably what messed everything up), just highlight one of the older installations below it, then click Install Specific Version like this:

When you hit Install, a previously installed known-good copy of the firewall policy will be installed and hopefully undo whatever bad change was installed to the gateway.  Note that doing this does not change any configurations shown to you in the SmartConsole, it ONLY changes what is installed on the gateway back to a good config that was previously installed.  If you hit the Install Specific Version "panic button", install the older policy to the gateway, then reinstall the current security policy again, you will be right back in the "panic" situation again!

So hopefully you have been able to halt the endless door pounding and phone ringing by hitting the "panic" button as shown.  You have bought yourself some time to now figure what boneheaded change was made by one of your coworkers (or you!) that caused this unfortunate situation to occur.

Part 3: The Investigation

While the Installation History screen is typically associated with "panic" reverts of gateway policies as shown in the last section, the View Installed Changes button on that same screen can be very handy for examining the specific changes in a suspect revision that came after the one you reverted to in the prior section:

To see even more information about a certain session, hit the View button which will bring up a read-only copy of the SmartConsole showing the exact state of the configuration after that particular highlighted session was published:

By this time you may have some suspicions that a certain policy layer and its rules may have been changed in a way that caused the "panic" situation to occur.  If so select the policy layer in question, then select Actions...History:

A nice concise list of all changes made in that policy layer in the various sessions is presented.  If you want to see the history of only a specific rule that you suspect is the culprit, simply highlight the rule and click its History tab:

You can also view all changes on the screen below if you aren't sure exactly where to look:

Suppose you have now identified a specific policy layer that was messed with and caused the panic situation to occur.  If there were a multitude of changes made and you don't want to manually back all of them out, you can Revert the policy layer configuration back to a specific point in time, thus discarding the changes made in one or more revisions like this:

In our example above we will be removing (or undo'ing) a total of 5 changes made in the two published sessions just above the one we selected.

Part 4: Your Final Log Analysis Option

If you still can't determine what changes caused the problem, your last ditch effort is to look at the raw system-wide Audit logs like this:

This technique was also possible in R77.30 with the Audit/Management tab of the SmartView Tracker.  The SmartWorkflow product also has some nice change reports in that version.

Part 5: Your Last Resort..."Revert to this Revision" (R80.40+ Only)

PLEASE READ THIS SECTION IN ITS ENTIRETY BEFORE INVOKING THIS FEATURE.

The "Revert" option covered in Part 3 has a glaring limitation: While it can certainly revert changes made to a particular policy layer as shown, it DOES NOT revert any changes to object properties that also might have been made in association with those policy layer changes, nor does it revert any other properties changes of any type either including Global Properties, Policy Layer properties or Threat Prevention properties, to name a few.  Therefore once you have reverted the policy layer as shown part 3, if the underlying problem is that the properties of some kind of object were tampered with, the Revert operation will not change the object properties back and the problem will still be present.

Enter the new "Revert to this Revision" feature introduced in R80.40, added by Check Point in response to various concerns expressed by the CheckMates User Community.  Note that all this feature requires for use is a R80.40+ SMS or MDS, it DOES NOT require R80.40 on the security gateways themselves for use.  On an R77.30 and earlier SMS it was possible to "Restore a Database Revision"  and essentially revert all elements of the Check Point configuration (objects, policies, settings) back to a known good point in time.  Any changes made since the restored revision was originally taken were GONE. 

That is exactly what the "Revert to this Revision" feature does in R80.40+.  You can essentially choose a published session that had known-good changes in it, and revert to it, which will completely remove all changes since that selected revision was published which includes all object, policy, and properties changes.  All changes made after the reverted session are GONE, just like they were after restoring a revision in R77.30.  Note that all the changes in the selected published revision are preserved after the "Revert to this Revision", so normally you'll want to select that known-good revision when invoking this operation.

A few cautions before use:

1) Once completed, a "Revert to this Revision" is permanent and cannot be undone.  If considering use of the "Revert to this Revision" feature, it is strongly recommended to take a backup of the SMS and verify the backup is good before proceeding.  This can be easily accomplished from the Gaia web interface of the SMS, and the resulting backup file can be downloaded directly to your desktop for safekeeping.  All SmartConsole administrators will need to exit the SmartConsole to ensure the backup is successfully run. Once downloaded, make sure the backup *tgz file can be successfully opened by tools such as 7-Zip before proceeding.

2) This feature should only be used as a last resort; if you have skipped to this section without reading all of this document I'd strongly recommend you STOP and read this whole document in its entirety first.  It is highly likely that the techniques documented earlier will be able to meet your needs without having to resort to the "big gun" of "Revert to this Revision".

OK so let's suppose that we have made 11 changes in a session that are causing major problems once installed to a gateway; you can see them summarized on the Revisions screen which we have seen earlier:

But what were all those 11 changes exactly?  Let's take a look:

Se we can see a smattering of object, policy, and global properties changes representing every part of our configuration.  These "bad" changes comprise the 11 changes shown in the first screenshot.  Now we want to essentially undo everything (and we mean EVERYTHING) in that session.  So we choose the next-oldest known-good published session (17 changes in our example) and pick "Revert to this Revision" like this (note that you must be using R80.40+ on your SMS to see this option):

Screen 1 of the Revert operation appears.  READ IT ALL BEFORE PROCEEDING by clicking Next:

Screen 2, which performs a verification of your current configuration's eligibility to be reverted, if this fails DO NOT PROCEED and contact TAC for assistance if you still need to Revert.

Screen 3, final confirmation:

Screen 4, Revert operation complete, you will next be prompted to restart the SmartConsole:

After logging back into the SmartConsole, we can see that all changes made in that 11-change published session are gone:

And that concludes our use of the "Revert to this Revision" feature, introduced in R80.40 but should only be used as a LAST RESORT.

Hopefully you found this writeup useful, please let me know if you have any other change control techniques that were missed and I'll be happy to add them.  Thanks again to Tomer Sole‌!

[Timothy_Hall]

> So with the "revert to revision" feature in r80.40 you can revert to a point back in history and by doing that you would lose all the newer revisions, objects, etc that are more recent, right?

Yes.

> But you could keep reverting to older revisions, correct?

Yes.  You can only go backwards with "Revert to this Revision" in R80.40+.  This is why it is strongly recommended to take a backup first.

> How was it in r77.30? Could you move back and forward between revisions?

You could move both backwards and forwards with R77.30 database revisions.  This is because the "revisions" in that release were simply a self-contained tgz file containing all of the flat text files comprising your configuration (objects_5_0.C, rulebases_5_0.fws, fwauth.NDB, etc.).  The tgz file contained everything you need (kind of like a migrate export/upgrade_export) so you could go both forwards and backwards.  However if you went backwards, made some intermediate changes, then jumped back forward to a newer revision, the intermediate changes would be lost as there was no ability to do any kind of merging when restoring a revision.  The one exception was ICA transactions which were always merged when restoring a revision, to ensure certificates that were revoked don't suddenly get un-revoked by restoring a revision which would be very bad.  This merging also helped keep SIC trust with gateways from getting broken by restoring a revision.

In R80+ the configuration is now implemented in a postgres database, and there really aren't self-contained "revisions" like we saw in R77.30 and earlier, it is all just a series of database transactions.  So I assume once you go backwards with Revert to This Revision, it simply rolls back all transactions in the database to that point, and any record of the rolled back transactions are gone.  Even if they weren't gone somehow, there is no way you could try to roll back forward without potentially conflicting with other intermediate changes you have made in the meantime.

> I guess it would be good if policy installations and revision were linked and then in case of panic we could install policies back and forward until we get to a solid old policity intallation.

You can sort of do this with "panic button" gateway policy reverts on the Installation History screen, but all this does is reinstall a known good policy back to the gateway.

[Luis_Miguel_Mig]

Thanks Timothy,
regarding the last point
>> I guess it would be good if policy installations and revision were linked and then in case of panic we could install policies back and forward until we get to a solid old policity intallation.

>You can sort of do this with "panic button" gateway policy reverts on the Installation History screen, but all this does is reinstall a known good policy back to the gateway.


But the panic button doesn't get the database or  in other words the Console view sorted, right? You need to go and do all the investigation to get it tidy. What I am suggesting is that if R80.40 can revert database revision, perhaps it could be easy enough to implement a second button to be used after the hitting the panic button to get tidy the database revision at that point in time.

[Timothy_Hall]

I don't think that is possible based on what is just available in the Installation History, when you hit the panic button all you are doing is installing a previously installed known-good copy of the policy to the gateway to buy time for troubleshooting.  All that compiled policy contains is the relevant configs for that particular gateway, not the entire configuration stored on the SMS.  I suppose you could manually revert to the same point in time that previous known-good policy was installed using "Revert to This Revision", is the second button you are asking for the ability to do that without having to manually hunt through all the published sessions and revisions yourself finding the right point in time?

[Luis_Miguel_Mig]

> Is the second button you are asking for the ability to do that without having to manually hunt through all the published sessions and revisions yourself finding the right point in time?

I was suggesting something different but that is a good idea and probably easier to implement.

What I was suggesting is pretty much resolving what you call "investigation"  in a deterministic and automatic way.

[Kevin_Canaday]

Excellent information. I'm curious how much of this is possible via the API.

Is it possible to push previous policy versions to gateways via the API? 

I do see that the API now has a call 'revert-to-version'. This appears to require a reference to a previous session. Are there any usage examples of this in action? Would one need to document their prior session ID's to know what to revert back to? 

[Timothy_Hall]

The revert-to-version and its sibling verify-revert appear to the the API version of the "Revert to this Revision" feature documented in Part 5 which was introduced in R80.40.  

If I'm reading the API documentation for install-policy correctly, it looks like you can pass an argument called revision which indicates the UID of the revision of the policy to install; the default is always the latest version if you don't specify this argument.  This is probably how one would do an "Installation History" revert via API of a gateway to a known-good previously installed policy as mentioned in Part 2.