Check Point issues even they can’t resolve!

[cmpwat]

So, I have been advised by Phoneboy to post my issues here with the hope that someone might be able to provide some help on them so here goes..... oh it’s a long one!

We were forced to upgrade from R77.30 like a lot of people so arranged for a new gateway and management server to be built on new Dell PowerEdge servers (both of which were checked by CheckPoint and confirmed as ok; they were in the compatibility list.)

Both the management server and gateway were built on R80.30 and to begin with, all was fine. However before long we noticed a memory leak in that 32gb of ram was being consumed to the point the gateway was either crashing and rebooted or it required a reboot after 12 hours. It was decided that the issue was my licence hasn’t been cut correctly and was still running an old version so needed to be renewed, which is did, but then the issue with CPUs started.

the gateway was using more CPUs than it was licences for, cue some more investigations and was told that the only way to resolve this was to upgrade to R80.40.

this was planned and carried out and more issues occurred! Our Secureid VPNs no longer worked and the work around from CP didn’t work either. So to stop the whole business from not being able to connect, I rolled back, using the CP method and this failed. We had to reset the database on the gateway and the management server to get the 2 to talk again but in doing so the threat management blade screwed up and the only to resolve is to rebuild the gateway!

CP provided us with a 5900 to use whilst we had out open server rebuilt to R80.40.  The applicable is on R80.40 but we cannot use it as the Secureid VPNs still do not work. It’s been looked at a number of times and logs taken and the developers still haven’t got a fix.

So I have:

an open server with a potential memory leak if I use my licence. If I use an eval licence then we have too many CPUs being used.  The ‘fix’ for this is to update to R80.40 but this doesn’t work as it’s stops my users connecting to the VPN as the SDCONF.rec file keeps overwriting (yes the SK has been read and doesn’t work).

I have an 5900 that I cannot use as it’s in R80.40 and has the same issues as the open server with the SDCONF.rec file. Not sure about the memory or CPU as it’s not been used in anger!

so I am hoping that R81 will be the saviour to all of this as it’s been over 6 months now and I still do not have a fully working gateway and management server that doesn’t require some sort of babysitting on a weekly, if not daily, basis.

[Ryan_St__Germai]

Same. Last thing we want to do is break RSA right now. Its already a pain to troubleshoot CP+RSA issues because each points the finger at each other. 

[_Val_]

Check Point TAC and R&D both did make progress with this case. As soon as the case is resolved, we will disclose the root cause. Thanks for your patience. We are waiting form the customer's action to confirm

[_Val_]

The original reported issue is now resolved. It is related to misconfiguration introduced on the customer side when migrating from an appliance to open server.

[cmpwat]

It was not the customers fault at all, but the CP engineer that built the appliance in the first place.  Do not blame me when it was me that configured it.  Also how comes for week TAC didn't know what the issue was?  It was only the fact that one of your engineers were prepared to test it that he found the problem.

Re-word your comment.

[_Val_]

Well, sure, changed to "customer side".

In essence, one should renew Secret Node on SecureID server, when moving a security GW to a new hardware. Instead, old settings were kept on the SecureID server, causing communication to break.

Concerning "CP engineer", what fo you mean? Did you hire Check Point PS to assist you?

[cmpwat]

No Check Point provided a PS engineer FoC and he didnt know this was required

[_Val_]

I still believe @Ilya_Yusupov and his team deserve some credits for helping you out. It is up to you. 

[cmpwat]

Yes they do, If it were not for @Ilya_Yusupov being prepared to test in the Lab we would not be where we are at.  All credit goes to him and the team, as well as those that were prepared help out on Sunday.

[Bob_Zimmerman]

I'm curious about this memory leak. I'm running R80.40 jumbo 78 standalone on a small box for development purposes. After a month up (and lots of heavy API use), it's only using 2.9 GB of RAM and 890 MB of swap (system has 4 GB of RAM, 8 GB of swap allocated).

What is telling you your memory consumption is increasing? There are about 15 ways to classify "consumed" memory on Linux, and none of them line up perfectly with what people outside of kernel development think of as consumed memory.

Can you tell what process is consuming the memory?

[G_W_Albrecht]

Are you shure you reply to the post above ? Non sequitur, it looks to me...

[_Val_ · Accepted solution]

The issue is resolved.

After certain TAC and R&D efforts, we have identified the root cause and instructed the customer on how to fix it. 

When migrating from 5900 to an open server, Secret Node settings on the SecureID server were not refreshed. As the result, communication between GW and authentication server could not work. 

After setting up a new Secret Node, it is okay now.