MDS vsx_util upgrade fails

[Oryx]

Hello community,

We're trying to upgrade our VSX Cluster of Open Servers, which is currently running R81.10 JHF take 95, to R82.

When we run vsx_util upgrade from R81.10 to R82, we get the following error:

We have already tried the SK articles:
https://support.checkpoint.com/results/sk/sk179591
https://support.checkpoint.com/results/sk/sk183811
https://support.checkpoint.com/results/sk/sk108693

We had no luck with tose.

We also tried first to R81.20 using vsx_util upgrade, which completes properly. Then, we attemped again the vsx_util upgrade to R82, but the issue remains.

Finally, we reverted the object cluster back to R81.10 with vsx_util downgrade, which finishes as expected without any errors.

Any suggestion on how to overcome this issue?

Kind regards

[Chris_Atkinson]

Which version & JHF is the Management currently?

[Oryx]

Hi.

It's in R82, JHF take44.

[_Val_]

I would suggest opening a TAC case for this

[Oryx]

Hi,

I did that. I'm just trying to see if anyone experienced something like this to share knowledge. 

Thanks.

[Don_Paterson]

Did the elg file list any specific interfaces, so that you can check topologies on the VSs?

Search the log file  for the usual strings: missing, error, and failed.

Maybe it is an inconsistency in interface configuration on a VS.

You could try vsx_util check_interfaces and compare to the interfaces of each VS on each cluster member.

To do that you can use ifconfig in the relevant vsenv or vsx showncs <VSID>.

Also cphaprob -a if and of course hcp -r all

[the_rock]

Hey @Oryx 

I know it might be asking too much, but any way you could send us portion relevant to failure from .elg file at the bottom of that screenshot? 

[Oryx]

Hi,

I'm sharing the .elg file. I've just changed the Domains and VS names.

Kind regards

[the_rock]

Thanks! will check shortly

[the_rock]

I have a gut feeling below is why you had the problem...now why it happened, its another question. Did you send this to TAC?

		Starting Policy compilation
MyCommandCB started
***Reply is : (
	:note ("		firewall_application Policy installation/compilation for MARSFW01: '/opt/CPmds-R82/customers/SFCDMS01/CPsuite-R82/fw1/tmp/install_policy/ead9f43d-d7dc-49c4-b2d5-c2c22fcd5cfe/FW1/conf/MARSFW01.pf', line 9345: ERROR: table <zp_dummy_interface_ip> undefined( message from member VSX-GW02_MARSFW01 )")
	:format (line)
	:vsx_status_code (0)
	:vsx_operation_result (0)
	:message_type (1)
	:AdminInfo (
		:cpmi_cmd_status_code (0)
		:subject (operation-note)
		:operation (changever-vsx)
	)
)

		firewall_application Policy installation/compilation for MARSFW01: '/opt/CPmds-R82/customers/SFCDMS01/CPsuite-R82/fw1/tmp/install_policy/ead9f43d-d7dc-49c4-b2d5-c2c22fcd5cfe/FW1/conf/MARSFW01.pf', line 9345: ERROR: table <zp_dummy_interface_ip> undefined( message from member VSX-GW02_MARSFW01 )
MyCommandCB started
***Reply is : (
	:note ("		firewall_application Policy installation/compilation for MARSFW01: Error compiling IPv6 flavor.( message from member VSX-GW02_MARSFW01 )")
	:format (line)
	:vsx_status_code (0)
	:vsx_operation_result (0)
	:message_type (1)
	:AdminInfo (
		:cpmi_cmd_status_code (0)
		:subject (operation-note)
		:operation (changever-vsx)

[Oryx]

HI,

.Yup. I'm waiting for them to reply back. 

Thanks. 

[the_rock]

Here is what Im wondering though...did you have an issue at all pushing policy BEFORE upgrade attempt?

[Oryx]

Not as far as I can remember. 

But I'm far away of be the only one pushing policies in this deployment. 

Thanks. 

[the_rock]

I would ask TAC specifically about below.

firewall_application Policy installation/compilation for MARSFW01: '/opt/CPmds-R82/customers/SFCDMS01/CPsuite-R82/fw1/tmp/install_policy/ead9f43d-d7dc-49c4-b2d5-c2c22fcd5cfe/FW1/conf/MARSFW01.pf', line 9345: ERROR: table <zp_dummy_interface_ip> undefined( message from member VSX-GW02_MARSFW01 )
MyCommandCB started
***Reply is : (
	:note ("		firewall_application Policy installation/compilation for MARSFW01: Error compiling IPv6 flavor.( message from member VSX-GW02_MARSFW01 )")

[Don_Paterson]

I think zp is for zero physical (could only happen in VSX..)  

It may be a case where there was a VS deleted but it was not fully cleaned up on the cluster member. 

Maybe a vspurge would help.

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_CLI_ReferenceGuide/Topics-CLIG/VSX...

[the_rock]

That command rings a bell...might have seen someone do it back in R77.30 version.

[Alex-]

Maybe the Zero-Phishing blade is activated, which creates a dummy interface causing the issue here.

Due to constraints, I never activated it on VSX, though so can't check if this configuration bit is present on VSX clusters.

[the_rock]

Thats actually very good thinking, @Alex- 

[Don_Paterson]

It's an R81.10 gateway and Zph is R81.20 onwards only. 

It'll be a VSX specific thing where it has zero physical because of virtual interfaces or something else specific to VSX. 

The IPv6 part may be a red herring but it implies interface. 

[the_rock]

Thats what sort of threw me off, those ipv6 errors, but as you said Don, it could be red herring.

[genisis__]

It may be worth disabling Zero Phishing and trying again, I've had issues with this in the past, but realistically this should not cause a problem.

[the_rock]

For sure, worth trying.

[Alex-]

Correct, I overlooked the initial state was R81.10. To be followed with TAC then.

[Chris_Atkinson]

If you check VS0 via CLI are there any wrp interfaces that shouldn't be there or similar?

Short of involving TAC, I would suggest moving the VSX cluster to a higher JHF for R81.10 and trying the upgrade again (after an install policy / topology refresh).

[the_rock]

Excellent idea Chris, for sure.

[Chris_Atkinson · Accepted solution]

Post upgrade of the Management to R82 did you move/edit/replace the table.def or crypt.def?

The IPv6 error came back to me as something you see if you've edited one of those files incorrectly to tweak a VPN etc.

[the_rock]

I recall seeing something like that in older versions too. Mind you not with VSX, but could be related, for sure.

[Oryx]

Hi Chris,

Actually your suggestion is pretty relevant, since indeed on the move of the Management to R82 I needed to save the table.def of one of the Domains, which is the one giving me problems when I run the vsx_util. 

Out of curiosity, i've checked the original R82 table.def versus that on that I've manully saved before the upgrade. And this is the funny part, I'm seeing this:

This is the exact string error that I have when I runthe vsx_util upgrade. 

I removed the backup file table.def and put it back the original one. Then, I've run again the vsx_util upgrade to R82 and voilá, it concluded perfectly. 

My main concern now is if this is an issue before proceed with the upgrade or not. 

Any thoughs on why this is happening?

I'm still waiting for TAC analysis on this. 

Kind regards. 

[Chris_Atkinson]

Glad you have a good lead to follow now with TAC, suspect they will request details from R&D accordingly.

[Oryx]

Well, apparently I should not replace all the table.def of the Domain in the R82 MDS. At least, that was the answer from TAC. 

I should have use the original table.def and migrate the portions of the file manually from table.def backup to table.def original from R82. 

The TAC told me that I'm safe to proceed with the upgrade.