MDS vsx_util upgrade fails

[Oryx]

Hello community,

We're trying to upgrade our VSX Cluster of Open Servers, which is currently running R81.10 JHF take 95, to R82.

When we run vsx_util upgrade from R81.10 to R82, we get the following error:

We have already tried the SK articles:
https://support.checkpoint.com/results/sk/sk179591
https://support.checkpoint.com/results/sk/sk183811
https://support.checkpoint.com/results/sk/sk108693

We had no luck with tose.

We also tried first to R81.20 using vsx_util upgrade, which completes properly. Then, we attemped again the vsx_util upgrade to R82, but the issue remains.

Finally, we reverted the object cluster back to R81.10 with vsx_util downgrade, which finishes as expected without any errors.

Any suggestion on how to overcome this issue?

Kind regards

[Chris_Atkinson]

Which version & JHF is the Management currently?

[Oryx]

Hi.

It's in R82, JHF take44.

[_Val_]

I would suggest opening a TAC case for this

[Oryx]

Hi,

I did that. I'm just trying to see if anyone experienced something like this to share knowledge. 

Thanks.

[Don_Paterson]

Did the elg file list any specific interfaces, so that you can check topologies on the VSs?

Search the log file  for the usual strings: missing, error, and failed.

Maybe it is an inconsistency in interface configuration on a VS.

You could try vsx_util check_interfaces and compare to the interfaces of each VS on each cluster member.

To do that you can use ifconfig in the relevant vsenv or vsx showncs <VSID>.

Also cphaprob -a if and of course hcp -r all

[the_rock]

Hey @Oryx 

I know it might be asking too much, but any way you could send us portion relevant to failure from .elg file at the bottom of that screenshot? 

[Oryx]

Hi,

I'm sharing the .elg file. I've just changed the Domains and VS names.

Kind regards

[the_rock]

Thanks! will check shortly

[the_rock]

I have a gut feeling below is why you had the problem...now why it happened, its another question. Did you send this to TAC?

		Starting Policy compilation
MyCommandCB started
***Reply is : (
	:note ("		firewall_application Policy installation/compilation for MARSFW01: '/opt/CPmds-R82/customers/SFCDMS01/CPsuite-R82/fw1/tmp/install_policy/ead9f43d-d7dc-49c4-b2d5-c2c22fcd5cfe/FW1/conf/MARSFW01.pf', line 9345: ERROR: table <zp_dummy_interface_ip> undefined( message from member VSX-GW02_MARSFW01 )")
	:format (line)
	:vsx_status_code (0)
	:vsx_operation_result (0)
	:message_type (1)
	:AdminInfo (
		:cpmi_cmd_status_code (0)
		:subject (operation-note)
		:operation (changever-vsx)
	)
)

		firewall_application Policy installation/compilation for MARSFW01: '/opt/CPmds-R82/customers/SFCDMS01/CPsuite-R82/fw1/tmp/install_policy/ead9f43d-d7dc-49c4-b2d5-c2c22fcd5cfe/FW1/conf/MARSFW01.pf', line 9345: ERROR: table <zp_dummy_interface_ip> undefined( message from member VSX-GW02_MARSFW01 )
MyCommandCB started
***Reply is : (
	:note ("		firewall_application Policy installation/compilation for MARSFW01: Error compiling IPv6 flavor.( message from member VSX-GW02_MARSFW01 )")
	:format (line)
	:vsx_status_code (0)
	:vsx_operation_result (0)
	:message_type (1)
	:AdminInfo (
		:cpmi_cmd_status_code (0)
		:subject (operation-note)
		:operation (changever-vsx)

[Oryx]

HI,

.Yup. I'm waiting for them to reply back. 

Thanks. 

[the_rock]

Here is what Im wondering though...did you have an issue at all pushing policy BEFORE upgrade attempt?

[Oryx]

Not as far as I can remember. 

But I'm far away of be the only one pushing policies in this deployment. 

Thanks. 

[the_rock]

I would ask TAC specifically about below.

firewall_application Policy installation/compilation for MARSFW01: '/opt/CPmds-R82/customers/SFCDMS01/CPsuite-R82/fw1/tmp/install_policy/ead9f43d-d7dc-49c4-b2d5-c2c22fcd5cfe/FW1/conf/MARSFW01.pf', line 9345: ERROR: table <zp_dummy_interface_ip> undefined( message from member VSX-GW02_MARSFW01 )
MyCommandCB started
***Reply is : (
	:note ("		firewall_application Policy installation/compilation for MARSFW01: Error compiling IPv6 flavor.( message from member VSX-GW02_MARSFW01 )")

[Don_Paterson]

I think zp is for zero physical (could only happen in VSX..)  

It may be a case where there was a VS deleted but it was not fully cleaned up on the cluster member. 

Maybe a vspurge would help.

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_CLI_ReferenceGuide/Topics-CLIG/VSX...

[the_rock]

That command rings a bell...might have seen someone do it back in R77.30 version.

[Alex-]

Maybe the Zero-Phishing blade is activated, which creates a dummy interface causing the issue here.

Due to constraints, I never activated it on VSX, though so can't check if this configuration bit is present on VSX clusters.

[the_rock]

Thats actually very good thinking, @Alex- 

[Don_Paterson]

It's an R81.10 gateway and Zph is R81.20 onwards only. 

It'll be a VSX specific thing where it has zero physical because of virtual interfaces or something else specific to VSX. 

The IPv6 part may be a red herring but it implies interface. 

[the_rock]

Thats what sort of threw me off, those ipv6 errors, but as you said Don, it could be red herring.

[genisis__]

It may be worth disabling Zero Phishing and trying again, I've had issues with this in the past, but realistically this should not cause a problem.

[the_rock]

For sure, worth trying.

[Alex-]

Correct, I overlooked the initial state was R81.10. To be followed with TAC then.

[Chris_Atkinson]

If you check VS0 via CLI are there any wrp interfaces that shouldn't be there or similar?

Short of involving TAC, I would suggest moving the VSX cluster to a higher JHF for R81.10 and trying the upgrade again (after an install policy / topology refresh).

[the_rock]

Excellent idea Chris, for sure.

[Chris_Atkinson]

Post upgrade of the Management to R82 did you move/edit/replace the table.def or crypt.def?

The IPv6 error came back to me as something you see if you've edited one of those files incorrectly to tweak a VPN etc.

[the_rock]

I recall seeing something like that in older versions too. Mind you not with VSX, but could be related, for sure.