topic Re: Policy Push Time. Accelerated Policy Install. Global Objects in Firewall and Security Management

Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Wed, 18 Mar 2026 14:21:55 GMT

What are others seeing for policy install times of large policies, lets say 5,000 lines to a single firewall?

Our policy installs are again stretching out beyond 20 minutes. Over the last 6+ years I have worked with TAC on a ticket about once a year to work on policy push speed. Generally, after several months of work, we can get it down to 10 to 15 minutes. This usually involves several custom hotfixes that are eventually rolled into a Jumbo. Then over time, newer Jumbos, policy growth / changes, major version upgrades, feature additions, it works its way back towards the 20 minute mark. At which point, the cycle starts over again and I spend months working to speed things back up again.

There are no CPU constraints on the local MDS, nor the firewall itself. Pushing policy across a slow link to the other side of the planet is slower, but not significantly slower than to a firewall in the same datacenter on 10 gig links and sub 1ms ping times. I can push to 50+ firewalls in 25 to 30 minutes but 1 IP add to 1 group on 1 rule to 1 firewall is back up to 20 minutes since our recent upgrade of MDS to R82.

Accelerated Policy push is amazing, the 1% of the time we are able to use it. We share groups extensively across our various domains so the majority of our changes are to network groups in the global domain. Any assignment of the global domain to the domain you are working in causes a full policy to every firewall in the domain. So its very rare that it is available on any given policy push. I do have an open RFE (I think the 2nd or 3rd attempt) to fix the fact that global assignment breaks accelerated push.

TAC and our sales contacts have been a bit vague on how others are dealing with long policy push times and if we are the only ones complaining about this ongoing issue. So I'm asking to see what others are seeing pushing "large" policies out to firewalls.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

TurgutKaplanogl — Wed, 18 Mar 2026 14:44:12 GMT

Hello,

Are the 50+ firewalls you’re pushing the policy to SBM models, meaning their OS is Embedded Gaia? Additionally, are you using a single policy package for all 50+ firewalls? Also are you managing the 50+ firewalls with a single CMA? I think providing more detailed and comprehensive information would be helpful for this type of question.

Thank you

Re: Policy Push Time. Accelerated Policy Install. Global Objects

the_rock — Wed, 18 Mar 2026 14:46:18 GMT

Hey David,

Lets see what others have to say, but to me, logically, if someone has 5000 rules, 10-15 mins policy push time sounds sort of normal/expected.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Wed, 18 Mar 2026 14:58:53 GMT

I have 4 primary CMA's. Each CMA has between 15 and 400 firewalls, with a range of policy sizes. From 100 lines to 5000+ with 100+ inline layers. The 50 firewalls I was referring to above was the same policy pushed out to the 50 firewalls that run that specific policy vs pushing to only one firewall that runs that policy.

The vast majority of the policy push time is on the MDS, compiling policy, before it ever attempts to push the policy out to the firewalls. So I was not specific on the firewalls the policy gets pushed to as it seems to make little difference. We have a broad mix of firewall hardware, VSX, maestro, CloudGuard cores spread across on prem and AWS, Azure, standard active passive clusters from 3800's to 16,000's. The firewall hardware size and location on the planet makes some logical sense, in that pushing a large policy to a small or busy piece of hardware will take a minute or two longer but when dealing with a average time of 20 minutes, has little over all impact.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

TurgutKaplanogl — Wed, 18 Mar 2026 15:34:02 GMT

Hello,

In my opinion, the number of rules you mentioned is not very high if proper sizing has been done. In this case, the slowness issue can be addressed by checking the sizing information and limitations (including considerations to keep in mind during configuration and deployment) provided in the two SKs I shared below for your MDS, CMA, and MLM setups.

https://support.checkpoint.com/results/sk/sk183689

https://support.checkpoint.com/results/sk/sk178325

Additionally, some optimizations will need to be applied for a management architecture of this scale. I also recommend enabling MDPS on some gateways for testing purposes. I have shared details about MDPS(Management Data Plane Separation) in the SK below.

https://support.checkpoint.com/results/sk/sk138672

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Wed, 18 Mar 2026 17:02:42 GMT

I am running 6000XL's and other than total number of firewalls in one CMA, I'm not over any limits and I think with R82 and the current jumbo I'm again now back under that limit. I can be the only admin on the 6000XL in the middle of the night, when no other policies are pushing and we are still currently out in the 20 minute range.
I'd be willing to do some testing with MDSP, but as I stated, pushing the same policy to wildly varying hardware will make only a 5 or 10% change in the overall time.

What I'm asking is, is this constant fight that I've had over the last 6+ years to attempt to keep my policy push times around the 10 minute range what other customers are seeing?

10 ish minutes to load a policy in the middle of a outage is generally acceptable by everyone involved. But as we get out to 20 minutes and beyond it is very frustrating and really doesn't look good for Checkpoint.

Are other customers just accepting this? Working around it in other ways? Given up after a similar experiences with TAC cases that drag on attempting to address this issue for months and years?

Here is a common scenario:
Sitting on an outage bridge with a broad set of infrastructure and application support teams represented and after a few minutes of troubleshooting it is found that for whatever reason the fix we have decided to implement is to add a single IP to a single rule. I say OK, I've done that, now it will be 20 minutes to push that out to the one firewall that needs it. 20 minutes is a very long time to wait on a call with production down and no progress being made. Even more so when at 2 minutes after you start the policy push, the application team asks you to add a second IP. Now we are at 40 minutes to fix the issue. Sometimes, if I can turn off all our automation that might trigger a global policy reassignment, and I break some of our other procedures, I can add the second IP in such a way that I MIGHT get an accelerated push for the second IP.

This seems like I can't be the only customer of Checkpoint that feels this is a excessively long time to update policy on a firewall when there are not CPU or bandwidth constraints anywhere in the system.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

TurgutKaplanogl — Thu, 19 Mar 2026 06:58:14 GMT

Hi,

The 20 minutes duration and example you mentioned are of course not an acceptable timeframe. I believe there can be multiple reasons for this. In environments with the rule sets you specified, such durations are not observed. When such situations occur, we address them by applying different solutions to bring the timing to a more acceptable level. This can sometimes involve optimization of the rule set, performing an action on the gateway where the policy is being installed, adjusting buffer limits of services such as CPM, FWM etc. on the management server or making adjustments to the management architecture. However a 20 minutes policy installation on a single gateway is not considered a normal scenario.

If you wish, you can work with Check Point PS or TAC team.

Note: Please check you total GW objects in MDS for limitation;

https://www.checkpoint.com/downloads/products/smart-1-6000-security-management-datasheet.pdf

Thank you

Re: Policy Push Time. Accelerated Policy Install. Global Objects

the_rock — Wed, 18 Mar 2026 18:03:01 GMT

I also believe engaging PS team would definitely be a great idea, for sure.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

Timothy_Hall — Thu, 19 Mar 2026 15:38:44 GMT

The delays in installing policy can be summed up by one daemon: fwm.

In the modern management architecture, one of the few remaining responsibilities of the fwm daemon is INSPECT code generation and compilation (cpm has taken most of the rest). Unfortunately, because use of the postgres database is not known to the legacy fwm daemon, all relevant objects and data must first be dumped out of the postgres database back into the legacy files like objects_5_0.C and rulebases_5_0.fws so fwm can work on them. For a full policy installation, this takes a while and is known as the "Legacy Dump". One of the reasons accelerated policy installs are so fast is that they use the "Modern Dump", where many of fwm's compilation duties are performed ahead of time in a multi-threaded daemon like cpm. This information about the different dumps is in the current CCSE R82 courseware, but doesn't appear anywhere else in SecureKnowledge or the Documentation. Can't remember if it was ever there or got removed.

fwm is single-threaded, so all the cores in the world will not matter; in an MDS environment, there are multiple instances of fwm to service the Domains, but each one is still single-threaded. I don't believe Smart-1s have Intel's Turbo Boost enabled, which would allow a single core to overclock and help these types of single-threaded operations complete faster. Gateways don't generally support Turbo Boost either, though the 9300/9400 models are starting to use it. Hopefully SMT/Hyperthreading is not enabled on the Smart-1's in question here (cat /sys/devices/system/cpu/smt/active), as single-threaded processes like fwm do not benefit from SMT and will incur at least a 11% penalty. Remember reading about it in an SK article and can't find that now either. The consensus at CheckMates seems to be that the lower Smart-1 models (600/700) have SMT on, while the higher models (6000/7000) don't.

Ever notice how after changing something on a gateway or cluster object and hitting OK it takes a long time, to the point that the SmartConsole GUI client blurs out and maybe even stops responding? That's because fwm has to handle that specific operation involving gateway objects. Why is the installation of Threat Prevention policies usually so much faster than Access Control? Because fwm is generally not involved with TP policies.

So the bottom line is that until Check Point can finally replace the legacy fwm daemon's functions with something that is not single-threaded, these kinds of issues will continue. There is very little Check Point administrators can do about it. At some point along the way in R8X the Access Control policy verification process was taken away from fwm and given to cpm, which resulted in a big performance improvement for those types of operations. The other legacy daemon fwd was "scaled out" in R82 with multiple threads possible (sk182215: "You have reached the maximum capacity this worker's configuration can handle" message in the $FWDIR/log/fwd.elg file), hopefully something like this is in the works for fwm. But it would seem to me that policy compilation is, by its very nature, a linear process that may not lend itself well to parallelization.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Wed, 18 Mar 2026 19:54:21 GMT

Your explanation matches what I see. A single FWM process that sits at 100% for 15+ minutes, on a single core, then MDS starts talking to the actual firewall and pushing policy out for the next 5+ mins. Policy rule count, and object count have far more affect on the policy push time than how busy the MDS is with other tasks, anything about the destination firewall, or the network between them.
Small simple policies push fast, big complex policies push slow. Everything else is a rounding error.

I'm more frustrated this time as, after the upgrade from R81.20 to R82, I'm starting the cycle over again. That change on my MDS only took me from a borderline annoying ~15 minutes running 2 custom hotfixes to address policy push time to 20+ minutes after upgrading to R82 and adding back in the same custom hotfixes.

One of the reasons I waited this long was, I was hoping that some other customers burned their time to do the tuning on R82 to make it at least as good as R81.20 when pushing out large policies. I guess I need wait longer before I move to R82.10.

I'm really looking for other customers that are having the same experience or to see if I am still a early adopter, upgrading MDS 6 months after its the recommend version.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Thu, 19 Mar 2026 14:05:28 GMT

This is not exactly related, but the spec sheet for the 6000 series is still the version from 2022. R82 and R82.10 have much higher listed max number for many of the limits. 2022 when the 6000 spec sheet was listed we were in R80.30? So how do you figure out the real supported max for this hardware today with the current OS versions?
Until the 7000 series came out I always assumed that a 6000XL plus would run the max listed in the release notes for the OS version. When I've pressed Sales, Diamond or TAC for what the new max's are I get some very vague answers.
It seems like a hardware to OS version grid is really what is needed for the recommended limits if each new OS is really increasing these numbers.
The other item I've asked about is:
OK, I'm slightly over the recommended count on this one limit, but WAY under on these other 3. So am I OK? Other than the one single core that is maxed during policy compile, I have significant amounts of free CPU, RAM, and disk IO by every measure I can find. This would seem to say that capacity of the MDS hardware is not the issue.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

Timothy_Hall — Thu, 19 Mar 2026 15:59:49 GMT

I just poked around in R82.10, and it doesn't look like anything has changed with the fwm process, at least as far as I can see. fwm is also why some GUI functions are still stuck in the legacy SmartDashboard and the old SmartEvent client GUIs. I'm also not seeing any mention of improvements for fwm in the R82.20 EA release notes, although to be fair it is still EA, and I don't physically have the R82.20 code yet to look at.

One optimization technique would be to remove any unused objects displayed by the Objects Explorer, which would reduce the size of the legacy dump and give fwm less data to process during compilation. Can't really think of much else to do.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

PhoneBoy — Thu, 19 Mar 2026 16:02:44 GMT

Datasheet numbers are generated with whatever version is the most recent at the time.
They generally don't get updated with new software versions.
A lot of things can impact the maximum number of connections a given appliance can actually track, including actual traffic patterns, usage of NAT, and software blades enabled.

As far as the the length of time it's taking to install a large policy on your MDS, it's likely the single-threaded nature (i.e. single core) and the 32-bit nature the fwm process (limits a process to 4GB of addressable memory) which is not able to utilize the full resources of the MDS.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

Timothy_Hall — Thu, 19 Mar 2026 16:08:06 GMT

Great point Dameon, forgot that fwm is still 32-bit even though we've been using a 64-bit OS pretty much since Gaia was first introduced. The newer cpm process is definitely 64-bit.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Thu, 19 Mar 2026 16:45:28 GMT

My previous RFE's for fixing accelerated policy push after global assignment to the domain have ended in a hard NO. Or at least a, roadmap item so far in the future that it doesn't actually have a version number assigned to the OS yet, which in my experience is the same thing as a hard NO.
I've just had my weekly diamond meeting. Today I was told it was coming in a future jumbo. It was hinted it was already in progress as a feature add.
Now when I pressed further for details, like:

Which Jumbo?

Which year / quarter?

Will it be just MDS at that version or will it require a Jumbo / min Version on the firewall target?

Then things got a little more cloudy but I'm hopeful that is a result of just not having the details yet, and not the result of no work actually having started on the development of the fix. As the previous explanations of why a global assignment breaks the accelerated policy push have been that it was a significant issue with "global" and the way the accelerated dump is processed. And that interaction wasn't going away until basically all the above issues with FWM went away.

So I'm hopeful that this is really coming "soon"™️.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

PhoneBoy — Thu, 19 Mar 2026 18:21:31 GMT

Above and beyond the issue you raise here, there is one product we are pushing driving reduced need for fwm: Web SmartConsole.
Web SmartConsole needs REST API to do what it does.
While we've come a long way since R80 initially added REST API support 10 years ago, there are still several functions that don't have such support...all of which live in fwm.

As some of these functions were implemented before REST APIs were standard, they weren't always implemented in a REST API-friendly way.
This means these functions must be reimplemented to be integrated into something outside of fwm.
All of that takes time, of course.
Note that some features (Legacy VSX) may never have REST API support, though it has a replacement (VSnext) that is.

Bottom line: this is definitely in the roadmap.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

Tomer_Noy — Sat, 21 Mar 2026 16:50:48 GMT

I have some "inside information" on this 😉

It will only require an MDS update, no need to update the gateways for this enhancement.

If you'd like to get a private HF for your lab, even before it reaches JHF let me know and we can try to promote it. That can give you an early chance to see that it meets your needs.

It's a bit hard to say exactly which JHF, but we are working on it, so hopefully not very far in the future. Definitely this year.

BTW, we didn't get rid of fwm to do this, but it was a non-trivial development.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Mon, 23 Mar 2026 15:11:05 GMT

I received the same basic information from my Diamond representative on Friday. This makes me very hopeful that it really is on it way and will address a large portion of our policy install slowness.

We may look at a private HF to start some testing.

Now putting back on my disgruntled customer hat, 20 minutes is still a very long time for the remaining full policy installations. Especially as we also sometimes see the message: "There are to many unpublished changes since the last installation for accelerated push." Depending on how many is to many, that may be my next ask. It would be very disappointing to remove the Global issue and replace it with a "to many changes" issue. Also, "to many changes" isn't listed in sk169096 as a limitation.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Tue, 24 Mar 2026 15:24:33 GMT

@Timothy_Hall wrote:
I just poked around in R82.10, and it doesn't look like anything has changed with the fwm process, at least as far as I can see. fwm is also why some GUI functions are still stuck in the legacy SmartDashboard and the old SmartEvent client GUIs. I'm also not seeing any mention of improvements for fwm in the R82.20 EA release notes, although to be fair it is still EA, and I don't physically have the R82.20 code yet to look at.
One optimization technique would be to remove any unused objects displayed by the Objects Explorer, which would reduce the size of the legacy dump and give fwm less data to process during compilation. Can't really think of much else to do.

We do have some object cleanup we can do. We have some of this already scripted, but the API calls we were using in R81.20 started not seeing some global network groups as "in use" if they were only used in NAT rules or Encryption Domains so we paused some of the automation around this. We are starting to test again in R82 to see if the same issue exists. But we are looking at maybe a 5% reduction in total objects, so I'm not thinking that is going to be a huge help.

I thought the issue with dumping all the objects was a thing of the distant past. Like starting in 77.x or 80.x checkpoint no longer sent the full database of objects down to all the firewalls. At that point it only sent objects that are actually used in the specific policy as part of the policy. Are you saying that this legacy dump, dumps the full database, every time and not just the objects specifically used in the policy it is current compiling? That would seem to not track with how much faster the smaller policies compile.

Re: Policy Push Time. Accelerated Policy Install. Global Objects

David_Evans — Mon, 30 Mar 2026 18:42:33 GMT

@David_Evans wrote:
.......
Especially as we also sometimes see the message: "There are to many unpublished changes since the last installation for accelerated push." Depending on how many is to many, that may be my next ask. It would be very disappointing to remove the Global issue and replace it with a "to many changes" issue. Also, "to many changes" isn't listed in sk169096 as a limitation.

So we got our custom hotfix.... and so far, this is exactly what has happened. All of the errors about "Accelerated push is not compatible with the changes made since the last install"(global changes), have changed to "Not accelerated because there are too many unpublished changes".

Now hopefully this is some minor issue as there were zero locks / unpublished changes in the domain. So this error message must have some other meaning besides the common usage of "unpublished changes".