topic Re: R80+ Change Control: A Visual Guide in Firewall and Security Management

R80+ Change Control: A Visual Guide

Timothy_Hall — Mon, 08 Jun 2020 23:50:02 GMT

As mentioned in the article Revisions Management in R80.x, I do have an informal writeup I use when teaching CCSA R80.10 that helps summarize how R80+ Change Control and Revisions are handled for the inevitable questions that arise in class. This document is a sprucing up of those notes complete with some new screenshots.

Absolutely no way this document would have been possible without the incredible contributions of Tomer Sole and in particular these articles with content contributed by him:

How to revert a Policy or discard changes?

Revisions Management in R80.x

How do you rollback an old policy?

What follows is merely a roll-up of Tomer's content from an operational perspective, with some new screenshots I have put together. I hope you find it useful.

Preface: Install the "Change Report" SmartConsole Extension (R80.30+)

Back in R77.30, there was an SMS feature called "SmartWorkflow". One of the more useful elements of this feature was a "Change Report" which could clearly and succinctly show the object and policy differences between different sessions. Based on feedback from the CheckMates User Community, the Change Report functionality is back in R80.30+, but in the form of a SmartConsole Extension. In other words this feature is NOT available by default in R80.30 and later, and you must manually add it to your R80.30+ SMS in order to use it.

Trust me: You want to install this Change Report functionality if you have R80.30 or later (but it is not supported on a standalone SMS/gateway combination); the instructions to do so are here:

https://community.checkpoint.com/t5/API-CLI-Discussion-and-Samples/Show-changes-from-session-gt-from-a-single-session/m-p/80570#M4670

sk166435: How to view changes performed between revisions

Once this extension has been successfully installed a new button called "Changes" shows up on a variety of different SmartConsole screens. Clicking it presents a popup window that looks like the following; the next three screenshots are scrolled through a single Change Report and its results:

The new "Changes" button will also appear in many more places other than the Security Policy layers screen as shown above, particularly on screens where Audit Logs can be potentially viewed. A sampling:

Note: The "Changes" button will not be shown in any of the subsequent screenshots in this article or mentioned again. It is up to YOU to keep an eye open for the "Changes" button that will appear on various screens and take advantage of it, assuming you have installed this very useful SmartConsole Extension.

Part 1: What are you about to do?

You are in the process of making changes in the SmartConsole and are unsure (or have lost track of) what you have done due to one of your coworkers constantly interrupting you. If still in an unpublished session, you can see what changes are pending by enabling the Session Pane (only available in R80.10+ management) like this:

This will create a new pane on the far right of the SmartConsole where you can see pending unpublished changes:

This information can be very helpful when deciding to publish or discard a session. Another way to find pending unpublished rule changes in your current session is to look for "Edited" Access Control rules, indicated by default using a purple line in the Smart Scrollbar:

Note that the colored lines in the Smart Scrollbar will also by default show rule locks currently held by other administrators (dark grey) and search results (yellow) by default. You can even make section titles (light grey) and a selected/highlighted rule (blue) show up as well:

For more information about the Smart Scrollbar and some other great tips for efficiently navigating a large rulebase in the SmartConsole, see this post:

What are some of the tips and tricks for jumping between rules in the rulebase?

So you have now published your session and think you are ready to install policy to the gateway. A very good habit to get into prior to installation is looking at how many changes you are about to make:

Along the top of the screen, you should ALWAYS look at how many sessions and by how many administrators will be part of what you are about to deploy on the gateway. Note that this is the total number of changes made in the SMS config since policy was last installed to this particular gateway, and not every change counted here is necessarily part of this gateway's security configuration or will impact how it operates. If you see sessions and changes that are unfamiliar or unexpected though, it is a very good idea to hit the View Changes button to see exactly what will be included:

Along the top of the screen is a summary of all the different published sessions whose changes will be included if you install policy to the gateway. As shown above you can highlight one of those sessions, and then select the Audit Logs tab to see a very detailed list of exactly what changes were made in that particular highlighted session.

Let's assume that everything looks OK and you proceed to install policy to the gateway.

Part 2: The Panic Button

Your phone is ringing nonstop, people are pounding on your door, and it has all gone horribly wrong! Some very bad change got implemented when you pushed policy to the gateway at the end of the last section and it is impacting production traffic. You need to fix what you did RIGHT NOW. Thankfully the Installation History screen will be your savior in this case:

The first Installation Date shown above (1/15/2018) represents the most recent policy push (which is probably what messed everything up), just highlight one of the older installations below it, then click Install Specific Version like this:

When you hit Install, a previously installed known-good copy of the firewall policy will be installed and hopefully undo whatever bad change was installed to the gateway. Note that doing this does not change any configurations shown to you in the SmartConsole, it ONLY changes what is installed on the gateway back to a good config that was previously installed. If you hit the Install Specific Version "panic button", install the older policy to the gateway, then reinstall the current security policy again, you will be right back in the "panic" situation again!

So hopefully you have been able to halt the endless door pounding and phone ringing by hitting the "panic" button as shown. You have bought yourself some time to now figure what boneheaded change was made by one of your coworkers (or you!) that caused this unfortunate situation to occur.

Part 3: The Investigation

While the Installation History screen is typically associated with "panic" reverts of gateway policies as shown in the last section, the View Installed Changes button on that same screen can be very handy for examining the specific changes in a suspect revision that came after the one you reverted to in the prior section:

To see even more information about a certain session, hit the View button which will bring up a read-only copy of the SmartConsole showing the exact state of the configuration after that particular highlighted session was published:

By this time you may have some suspicions that a certain policy layer and its rules may have been changed in a way that caused the "panic" situation to occur. If so select the policy layer in question, then select Actions...History:

A nice concise list of all changes made in that policy layer in the various sessions is presented. If you want to see the history of only a specific rule that you suspect is the culprit, simply highlight the rule and click its History tab:

You can also view all changes on the screen below if you aren't sure exactly where to look:

Suppose you have now identified a specific policy layer that was messed with and caused the panic situation to occur. If there were a multitude of changes made and you don't want to manually back all of them out, you can Revert the policy layer configuration back to a specific point in time, thus discarding the changes made in one or more revisions like this:

In our example above we will be removing (or undo'ing) a total of 5 changes made in the two published sessions just above the one we selected.

Part 4: Your Final Log Analysis Option

If you still can't determine what changes caused the problem, your last ditch effort is to look at the raw system-wide Audit logs like this:

This technique was also possible in R77.30 with the Audit/Management tab of the SmartView Tracker. The SmartWorkflow product also has some nice change reports in that version.

Part 5: Your Last Resort..."Revert to this Revision" (R80.40+ Only)

PLEASE READ THIS SECTION IN ITS ENTIRETY BEFORE INVOKING THIS FEATURE.

The "Revert" option covered in Part 3 has a glaring limitation: While it can certainly revert changes made to a particular policy layer as shown, it DOES NOT revert any changes to object properties that also might have been made in association with those policy layer changes, nor does it revert any other properties changes of any type either including Global Properties, Policy Layer properties or Threat Prevention properties, to name a few. Therefore once you have reverted the policy layer as shown part 3, if the underlying problem is that the properties of some kind of object were tampered with, the Revert operation will not change the object properties back and the problem will still be present.

Enter the new "Revert to this Revision" feature introduced in R80.40, added by Check Point in response to various concerns expressed by the CheckMates User Community. Note that all this feature requires for use is a R80.40+ SMS or MDS, it DOES NOT require R80.40 on the security gateways themselves for use. On an R77.30 and earlier SMS it was possible to "Restore a Database Revision" and essentially revert all elements of the Check Point configuration (objects, policies, settings) back to a known good point in time. Any changes made since the restored revision was originally taken were GONE.

That is exactly what the "Revert to this Revision" feature does in R80.40+. You can essentially choose a published session that had known-good changes in it, and revert to it, which will completely remove all changes since that selected revision was published which includes all object, policy, and properties changes. All changes made after the reverted session are GONE, just like they were after restoring a revision in R77.30. Note that all the changes in the selected published revision are preserved after the "Revert to this Revision", so normally you'll want to select that known-good revision when invoking this operation.

A few cautions before use:

1) Once completed, a "Revert to this Revision" is permanent and cannot be undone. If considering use of the "Revert to this Revision" feature, it is strongly recommended to take a backup of the SMS and verify the backup is good before proceeding. This can be easily accomplished from the Gaia web interface of the SMS, and the resulting backup file can be downloaded directly to your desktop for safekeeping. All SmartConsole administrators will need to exit the SmartConsole to ensure the backup is successfully run. Once downloaded, make sure the backup *tgz file can be successfully opened by tools such as 7-Zip before proceeding.

2) This feature should only be used as a last resort; if you have skipped to this section without reading all of this document I'd strongly recommend you STOP and read this whole document in its entirety first. It is highly likely that the techniques documented earlier will be able to meet your needs without having to resort to the "big gun" of "Revert to this Revision".

OK so let's suppose that we have made 11 changes in a session that are causing major problems once installed to a gateway; you can see them summarized on the Revisions screen which we have seen earlier:

But what were all those 11 changes exactly? Let's take a look:

Se we can see a smattering of object, policy, and global properties changes representing every part of our configuration. These "bad" changes comprise the 11 changes shown in the first screenshot. Now we want to essentially undo everything (and we mean EVERYTHING) in that session. So we choose the next-oldest known-good published session (17 changes in our example) and pick "Revert to this Revision" like this (note that you must be using R80.40+ on your SMS to see this option):

Screen 1 of the Revert operation appears. READ IT ALL BEFORE PROCEEDING by clicking Next:

Screen 2, which performs a verification of your current configuration's eligibility to be reverted, if this fails DO NOT PROCEED and contact TAC for assistance if you still need to Revert.

Screen 3, final confirmation:

Screen 4, Revert operation complete, you will next be prompted to restart the SmartConsole:

After logging back into the SmartConsole, we can see that all changes made in that 11-change published session are gone:

And that concludes our use of the "Revert to this Revision" feature, introduced in R80.40 but should only be used as a LAST RESORT.

Hopefully you found this writeup useful, please let me know if you have any other change control techniques that were missed and I'll be happy to add them. Thanks again to Tomer Sole‌!

Re: R80+ Change Control: A Visual Guide

Kaspars_Zibarts — Mon, 15 Jan 2018 21:47:04 GMT

Thanks Tim! I would be very interested to hear about VSX and revision control. Is it really all "under control" now with R80.x when you have VSes stretching across multiple CMAs? All routing and spoofing updates and object changes considered? Ability to roll back to revisions as far as 2000+?

Re: R80+ Change Control: A Visual Guide

Timothy_Hall — Mon, 15 Jan 2018 21:59:19 GMT

VSX is not really in my area of expertise, Tomer Sole should be able to weigh in though.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: R80+ Change Control: A Visual Guide

Danny — Tue, 16 Jan 2018 07:14:51 GMT

I'm very interested as well about the status of MDSM with VSX regarding R80.x revision control.

This is where it is really important to function properly.

Re: R80+ Change Control: A Visual Guide

Kaspars_Zibarts — Tue, 16 Jan 2018 07:35:15 GMT

I'm really concerned as our MDS had 3000 odd revisions available over 9 months period - would it really work on VSX to roll back to version 1 (the earliest) considering that there has been number of interface and routing changes on top of regular rules. If not, what's the point having 3000 revisions and making code more complex and chewing resources that could be utilised for better purpose? I really think that one should be able to turn it off completely. There are other tools available to achieve the same result (i.e. Tufin)

Since management is becoming noticeably resource hungry all these little things start to add up and decrease user experience - soon we will need management server performance optimisation book Tim - I only take 10% for the idea

Re: R80+ Change Control: A Visual Guide

Gaurav_Pandya — Tue, 16 Jan 2018 10:42:09 GMT

Thanks Tim,

This is really nice & Informative document.

I just want to know that when we click on History option, How many entry it will show. Means what is the limit of that we can get back to earlier config.

Re: R80+ Change Control: A Visual Guide

Kaspars_Zibarts — Tue, 16 Jan 2018 11:20:22 GMT

we had 1500+ revisions available on a busy CMA. That's since the first upgrade to R80 back in March 2017. If you discard VSX concerns then I guess you should be able to go back all the way. Theoretically

Re: R80+ Change Control: A Visual Guide

Tomer_Sole — Tue, 16 Jan 2018 12:44:56 GMT

Hi everyone,

Does the "panic button" as described by Tim (install a previous revision on the gateway) satisfy the case that you described - overcoming VSX misconfiguration?

Also I would like to thank Tim Hall‌ for centralizing all these features in one visual guide

Re: R80+ Change Control: A Visual Guide

Timothy_Hall — Tue, 16 Jan 2018 13:01:31 GMT

Hi Guarav,

It will show history back to when the SMS was first loaded/upgraded, unless you have purged prior sessions from the Manage & Settings...Revisions screen. Note that purging revisions doesn't appear to actually free up any disk space on the SMS as shown by the df command, but presumably makes more storage space within the configuration database available. Also just to be clear doing a purge does not roll back or undo changes in the purged sessions.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: R80+ Change Control: A Visual Guide

Gaurav_Pandya — Tue, 16 Jan 2018 14:35:00 GMT

Ok Great.

If this is the case then at some point of time we need to free up the space. I which directory it stores History/ Revision database.

Re: R80+ Change Control: A Visual Guide

Tomer_Sole — Tue, 16 Jan 2018 14:52:03 GMT

Hi Gaurav, revisions are not stored in "directories" anymore. The new R80 backend uses native lightweight revisions based on the diff. Purge is available from the GUI. See How can I control the size of my R80.10 Security Management Server?

Re: R80+ Change Control: A Visual Guide

AlekseiShelepov — Tue, 16 Jan 2018 14:57:41 GMT

They are stored now everywhere and nowhere Revisions Management in R80.x

Revisions are now built-in in the database, describing different baselines of the database state.

Please take a look at other threads here:

Revisions Management in R80.x

How can I control the size of my R80.10 Security Management Server?

The revisions themselves are very light and only contain the delta diff (this is unlike pre-R80 Management servers where a revision was a zipped copy of the entire configuration). Either way, you can always open the Revisions view and purge older revisions.

In order to avoid an ever-growing database size, R80.10 Jumbo Hotfix take 42 and above introduces automatic IPS purge which deletes revisions older than 30 days. In R80.10 Jumbo Hotfix take 42 and above this purge happens automatically every 7 days.

Re: R80+ Change Control: A Visual Guide

PhoneBoy — Tue, 16 Jan 2018 23:23:42 GMT

Bravo, Tim!

Re: R80+ Change Control: A Visual Guide

Kaspars_Zibarts — Thu, 18 Jan 2018 11:05:25 GMT

Hi Dirk! Not really applicable to this topic about revisions but I would use API script (or dbedit but that will be much slower) - we recently were faced with a similar issue so I just wrote an API script that created a group containing all IPs/subnets on the list. Depending on your environment (R80? SmartCentre Server or MDS/which CMAs, format of the list) could provide more advise.

Don't know but maybe https://community.checkpoint.com/people/dwelccfe6e688-522c-305c-adaa-194bd7a7becc can move this question to a new thread?

Re: R80+ Change Control: A Visual Guide

Timothy_Hall — Mon, 09 Apr 2018 13:52:07 GMT

Had a student point out that the Smart Scrollbar in the R80+ SmartConsole can also be used to locate pending unpublished changes, I added some text and screenshots documenting this technique.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: R80+ Change Control: A Visual Guide

Tomer_Sole — Mon, 09 Apr 2018 16:42:01 GMT

Yeah this is kind of a hidden feature, we really should publish this more What are some of the tips and tricks for jumping between rules in the rulebase?

See more features that maybe should get their shine

Re: R80+ Change Control: A Visual Guide

Kaspars_Zibarts — Tue, 10 Apr 2018 06:10:11 GMT

Haha didn't know it was "hidden" - had to check my presentation i did to my troops back in March 2017.. it was already there!

Re: R80+ Change Control: A Visual Guide

Ron_Izraeli — Tue, 29 Jan 2019 05:56:16 GMT

Come and check out our alpha version of 'Change Report' supported by SmartConsole Extensions in CPX 2019

Re: R80+ Change Control: A Visual Guide

Fernando_Hagels — Sat, 23 Feb 2019 19:40:11 GMT

Hi all :

is there a way( like SmartDashboard) to revert all object changes in one step ??? similar to revert an old database revision. ?

if so, I'll thanks to all for your help .

Re: R80+ Change Control: A Visual Guide

PhoneBoy — Sun, 24 Feb 2019 00:50:27 GMT

You can revert what was installed on a gateway in one step (the compiled policy) — Step 2 of the root post in this thread.

Reverting the whole database in one step like a R77.30 Database Revision? Not currently possible.