topic Re: VSX cluster problem - are we alone? in Firewall and Security Management

VSX cluster problem - are we alone?

Raphael_Cote — Thu, 20 Dec 2018 14:33:26 GMT

Hey Check Point community, I need to know if we are alone in the world having so much difficulty implementing Check Point in a VSX cluster mode.

Here's our setup, two 15 600 in a VSX load Sharing mode. 6 vs and about 5000 users. We are using the FW, Anti-Bot, Ant-Virus, URL Filtering, SSL Inspection, and VPN blade. Pretty simple. Version 80.10 jhf 112.

The first time, we did the installation by ourselves, but as we had many problems, Check Point sent here their professionnal service to do the installation beacause the thought we were the problem. A week after he left, the exact same problem came back. They sent a second PS for antoher week without any results. It's been 15 months since we start the installation of Check Point and in the last 8 months I spoke almost daily with level 3 engineer to solve all the problem and after all this time, we still have many bugs. Here's a list :

- DNS problem (Firewall - Domain resolving error. Check DNS configuration on the gateway) - still in problem a year after opening a ticket
- Management console problem, the logs were not displaying. Had to reinstall the management from scratch.
- Update problem, corrution in the registry
- We see the VSX internal IP on our network, which we are not supposed as in the documentation. Problem is still there and no one has been able to explain it to me yet.
- Identity collector stop collecting data from the DC for 5 minutes interrmittently. Never completely resolve, found a parameter to drop the outage at 1 min instead of 5. As we have 3 collectors, it's okay for us not causing us incident, but...
- When we push our Internet security policy, we cause an outage to our TPV transaction. This was a crazy one. An allowed rule was actually dropping trafic but only when we push the policy. Had to add the block destination into the rule to solve the problem!
- Identity awarness problem. This is by far our worst one. Random user lost their Internet access because of the Pepd process that was choking, so missing important information about the user. It tooks 8 months and countless hours to find a solution, a hotfix.
- Unable to update the Ant-Bot, Anti-Virus or URL filtering. There was a problem with Epoch time
- Many problem with process crash. We had core dump for the Fw_full, dnsd, pepd, fw_vsnumber. Some hotfixes created to solve the problem.
- MUH agent on our server was disconnecting. Had to change a key in the registy
- Had to change many parameter in the fwkern.conf because the gateway were choking. This is not a bug as is, but the problem is that it's not documented anywhere how to fine tune the box for 5000 users, even the PS didn't know that.
- Usercheck page problem, it wasn't displaying. It wasn't configured for 5000 users as well, to many request had to change parameter in the httpd.conf file.
- SNMP trap we reveive were incomplete. Had to wait for 4 months to have a fix.
- RAD problem, the service stop respondig (URL Filtering - Rad Service not available). The problem is still there, Check Point is supposed to upgrade their cloud during the Holidays break..
- In the main page of the management, we see a red X saying Identity Awarness serious error for no reason
- In the main page of the management, we see a red X saying Anti-Bot db update fail
- As of now, our SSL inspection is not working well (Internal system error in HTTPS Inspection (Couldn't start inspection)). Our Internet access is slow as ....
- As of now, the NTP synchronozation as stop working on our gateway. The configuration is there, but there just nothing happening. Was working before but stop all of a sudden
- As of now, if I do a cpinfo -y all on my gateway, I can't see all the hotfix that are installed on it. Problem with the build.

I'd like to tell you that it's exaggerated, but in fact I probably forgot some bug that we had, this list is the strict minimum.

Is there someone who has pretty much that setup and it's working well?

Re: VSX cluster problem - are we alone?

cstueckrath — Thu, 20 Dec 2018 15:51:02 GMT

We are running a similar setup:

2x 15400 VSX VSLS, 80.10 JHF T154

- DNS problem (Firewall - Domain resolving error. Check DNS configuration on the gateway) - still in problem a year after opening a ticket

Seeing this sometimes

- Management console problem, the logs were not displaying. Had to reinstall the management from scratch.

- Update problem, corrution in the registry

Did not have those

- We see the VSX internal IP on our network, which we are not supposed as in the documentation. Problem is still there and no one has been able to explain it to me yet.

Same here. Annoying

- Identity collector stop collecting data from the DC for 5 minutes interrmittently. Never completely resolve, found a parameter to drop the outage at 1 min instead of 5. As we have 3 collectors, it's okay for us not causing us incident, but...

No problems so far

- When we push our Internet security policy, we cause an outage to our TPV transaction. This was a crazy one. An allowed rule was actually dropping trafic but only when we push the policy. Had to add the block destination into the rule to solve the problem!
- Identity awarness problem. This is by far our worst one. Random user lost their Internet access because of the Pepd process that was choking, so missing important information about the user. It tooks 8 months and countless hours to find a solution, a hotfix.
- Unable to update the Ant-Bot, Anti-Virus or URL filtering. There was a problem with Epoch time
- Many problem with process crash. We had core dump for the Fw_full, dnsd, pepd, fw_vsnumber. Some hotfixes created to solve the problem.
- MUH agent on our server was disconnecting. Had to change a key in the registy

Did not have those, either

- Had to change many parameter in the fwkern.conf because the gateway were choking. This is not a bug as is, but the problem is that it's not documented anywhere how to fine tune the box for 5000 users, even the PS didn't know that.

same here

- Usercheck page problem, it wasn't displaying. It wasn't configured for 5000 users as well, to many request had to change parameter in the httpd.conf file.

what did you change? We see similar things (200 Users...)

- SNMP trap we reveive were incomplete. Had to wait for 4 months to have a fix.
- RAD problem, the service stop respondig (URL Filtering - Rad Service not available). The problem is still there, Check Point is supposed to upgrade their cloud during the Holidays break..
- In the main page of the management, we see a red X saying Identity Awarness serious error for no reason
- In the main page of the management, we see a red X saying Anti-Bot db update fail
- As of now, our SSL inspection is not working well (Internal system error in HTTPS Inspection (Couldn't start inspection)). Our Internet access is slow as ....
- As of now, the NTP synchronozation as stop working on our gateway. The configuration is there, but there just nothing happening. Was working before but stop all of a sudden
- As of now, if I do a cpinfo -y all on my gateway, I can't see all the hotfix that are installed on it. Problem with the build.

Yeah, some of this sounds familiar...

Re: VSX cluster problem - are we alone?

PhoneBoy — Fri, 21 Dec 2018 02:21:54 GMT

Any TAC case(s) on the above issues?

Re: VSX cluster problem - are we alone?

Raphael_Cote — Fri, 21 Dec 2018 12:56:14 GMT

Thanks for the reply Christian. Check sk85040 for your usercheck page problem

Re: VSX cluster problem - are we alone?

Raphael_Cote — Fri, 21 Dec 2018 12:59:29 GMT

Yes most of them have a TAC case, you want more detail?

Re: VSX cluster problem - are we alone?

PhoneBoy — Fri, 21 Dec 2018 14:23:42 GMT

Please send the TAC cases in PM.

Re: VSX cluster problem - are we alone?

Louis_Poulin — Fri, 21 Dec 2018 16:47:01 GMT

Thanks for the help Dameon!

And it'd still be interesting to know if you are aware of other deployment of this kind in the world. Like Raphael said, we are under the impression that we are pretty much alone… and most of the errors encountered seems to require new hotfix to be resolved. So it's easy to feel like lonely guinea pigs.

Re: VSX cluster problem - are we alone?

PhoneBoy — Fri, 21 Dec 2018 17:24:19 GMT

Every setup is a little different and thus the issues may either be non-existent or different.

The issue with VSX "funny IPs" has come up a couple times on CheckMates threads, the other ones I'm personally less familiar with.

Re: VSX cluster problem - are we alone?

Louis_Poulin — Fri, 21 Dec 2018 17:33:35 GMT

I totally understand that every setup is a little different and I agree.

Maybe I can ask my question in a different way. In organizations having more than 5000 users browsing the web, do you have an idea of how common is a VSX Cluster running R80.10 with all the aformentioned blades active?

Are organizations still on R77.30? Or are they not using VSX? What is the most common setup these days?

Re: VSX cluster problem - are we alone?

JozkoMrkvicka — Sat, 22 Dec 2018 21:38:45 GMT

All of mentioned issues started after upgrading to R80.10, or were present also on R77.30 ?

What about R80.20 ?

Re: VSX cluster problem - are we alone?

Martin_Valenta — Thu, 27 Dec 2018 08:19:54 GMT

which fwkern.conf parameters were suggested to be modified and why?

Re: VSX cluster problem - are we alone?

Roy_Smith — Mon, 31 Dec 2018 12:12:10 GMT

Thanks for posting this. I have had similar issues with a 2 * 23500 VSX cluster with 11 * VS and 5000 users. We are also using all the blades you mention plus Application Control. We have had many of the issues that you mention. Many of the VS'es are small but we have 1 VS that is our Internet gateway and there are definite performance issues with it when there is an increase in the number of users. Case in point is the past week. During the holidays, we have less than half our users in offices and there have been no issues accessing Internet sites.

This was installed originally by a CP partner, who I do have complete faith in. However, even they were unaware of some fundamental "gotchas". For example, it took several weeks to discover that the VS were all in 32 bit mode by default (Why???) but this is not documented very well. Switching to 64-bit, obviously improved things considerably but we still have various issues.

I have had various TAC calls open and almost always get pointed to an SK article to make a CLI tweak, in particular fwkern.conf. I have also installed a newer JHF 3 times in the past 6 months at there recommendation. There seems to have been improvements but next week will be the real test

It is nice to know that I'm not alone with this.

Thanks

Roy

Re: VSX cluster problem - are we alone?

PhoneBoy — Mon, 31 Dec 2018 19:56:42 GMT

Just to be clear, not ALL of these issues are necessarily related to VSX.

The 32-bit VS issue is definitely VSX-specific and goes away in R80.20 since it is no longer possible to run VSes in 32-bit mode

Re: VSX cluster problem - are we alone?

Raphael_Cote — Thu, 03 Jan 2019 12:53:14 GMT

As it's a new deployment, we started at version 80.10

Re: VSX cluster problem - are we alone?

Raphael_Cote — Thu, 03 Jan 2019 12:59:57 GMT

Those parameter were change because of general performance problem, I can't be more specific I did it with the TAC and I don't have much more detail :

fwha_enable_state_machine_by_vs=1

fwha_freeze_state_machine_timeout=200

fwha_add_vsid_to_ccp_mac=1

fwha_forw_packet_to_not_active=1

fwmultik_input_queue_len=4096

Re: VSX cluster problem - are we alone?

Raphael_Cote — Thu, 03 Jan 2019 13:14:19 GMT

Let's keep in touch Roy in the next week to see how it goes. All I can say is that even with a bigger model of appliance than ours you have the same problem, so the problem is most likely due to software problem than a physical one.. BTW we are pretty much like you, one big Internet VS with all the problem and all the other small one doesn't have them. During the holidays everything is fine and when the load will be bigger next week the problem should reappear.

Re: VSX cluster problem - are we alone?

PhoneBoy — Thu, 03 Jan 2019 14:22:04 GMT

Is your vs bits also set to 32?

64 bit VS is definitely needed if you have a large VS (As is support for more than 10 cores in a VS).

Re: VSX cluster problem - are we alone?

Raphael_Cote — Thu, 03 Jan 2019 14:43:15 GMT

No we are at 64 bits

Re: VSX cluster problem - are we alone?

Roy_Smith — Tue, 08 Jan 2019 17:19:17 GMT

Hi Raphael

So far this week, everything is actually running great. I am now seeing the level of users, connections and traffic that I would expect with everyone back to work. Performance of the VS is fine and accessing internet sites is as snappy as I would expect. I'm hesitant to say the issues are resolved so will continue monitoring the situation.

One thing I did do, the week before the holidays, was to install JHF Take 154. It may be that there are some performance enhancements in the hotfix, which have helped things. I guess it's just a waiting game for the rest of the week

Roy

Re: VSX cluster problem - are we alone?

Raphael_Cote — Tue, 08 Jan 2019 18:49:08 GMT

Thanks for the follow up Roy! Unfortunately, on my side there is no improvement for my SSL problem, more than 100K errors today. I'll try to install a fix this evening, hopefully it will help. I can't install the latest JHF either as we have 5-6 personalized hotfix to solve other problem that we had, so we are stuck. I'd also like to migrate to 80.20, but everybody are scared of it, even the TAC.