Issue with 12000, VSX, VSWITCH & R80.30

[Alex-]

I will open a case for this but I wonder if someone has seen this already.

I upgraded a 12600 VSX cluster from R80.20 Take 47 to R80.30. All went well but there was a strange issue afterwards.

Two VS talk to each other via a "front" VSWITCH used for inter-VS communication. These VS also have "back" VSWITCH for the networks which are located behind them. I'd rather use tagged interfaces but it's another story and there's a reason why they're present.

After upgrade to Take 19, some traffic never makes it through a backend LAN on VS-A to the backend of VS-B. 

In Smart Console, the traffic is seen as accepted. With fw monitor, the traffic is seen but stays in the "i" part on VS-B.

The weird thing is that only specific protocols didn't go through, for the other ones we could see the full "iI-oO" and they worked normally. Failing protocols were RDP & HTTPS, but maybe there were others (no HTTPS inspection blade runs on any of the VS, and this is internal traffic only).

Now the interesting bit: uninstalling Take 19 actually solves the issue. We tried with the second cluster member which exhibited the exact same behaviour: OK with R80.30.0, fails with R80.30.19.

We're now in full production on both systems with R80.30 and no Take. I guess I will need to replicate issue with TAC but it's challenging as we need to install the Take on a production system and take live traces which isn't always easy to arrange, so I thought I'd check if anyone here would have seen that kind of behavior and had an idea.

The chassis themselves are all OK in terms of CPU, RAM, I/O and so on so I think it's really a software issue.

[FedericoMeiners]

@Alex- 

Hope you are doing fine, I was planning an upgrade to R80.30 to a customer with a similar architecture as you described, when I was going to through possible issues I found this issue that was solved in Take 50 of 80.30

When looking at the described sk (sk160352) it seems that the issue is quite similar as the one that you are having since it involves drops when using virtual switches.

You may want to point the TAC in this direction to try to sort this out more quickly since the fix is to install JHF Take 50.

Please let me know how it goes, personally I have suspended the migation until that hotfix becomes EA.

Hope it helps 🙂

[Magnus-Holmberg]

We had major issues when upgrading to R80.30 with VSX.

also with bond interface connected to vswitch. After installing ongoing take 50 we got connections back up and running again.

instead all site to site vpn failed.

needed to get a privet Hotfix from support that was included in R80.20 but then removed in R80.30

so with vsx, bond interface with VR or VSwitch I would not upgrade to R80.30 before more HFA is released to fix basic functions.

[matangi]

Hi all,

We have 2 VSX issues:

1. With VS-VSW / VS-VR, when the external interface of the VSW/VR is bond interface. - for more info please refer sk160352

This fix was integrated to R80.30 JHF take 50.

2. We have another issue with secureXL on - PRJ-4956
CLONE - R80.30 jumbo hf | gogo_heat_188_main T296 | VS-VSW-VS | TP + SXL | HTTP packets are not passing

This issue solves particular case only when all of the following conditions are met 
a. VSX setup 
b. VS-VSW-VS Or VS-VR-VS topology 
c. One of the threat-prevention blades are enable on VSs (e.g. ips, threat extraction, anti virus, anti bot...)

This fix was integrated in R80.30 JHF take 51 (still not included in ongoing take)

Following Task 97283, There is a private fix on top of take 50

New hotfix has been provided:

\\galaxy\ckp\pkg\fw1_wrapper\R80_30_jhf_t50_168_main\latest_linux50\release.dynamic.ping

[matangi]

Alex_Gilis

Regarding the strange issue which reproduced on R80.30 JHF take 19 and not R80.30 GA

We had a bug in specific packet flow with VS-VSW-VS, probably in R80.30 The packet goes on different flow comparing R80.30 JHF 19.

Anyway this issue shouldn't reproduce on take 50 + the private fix.

[Alex-]

Thank you for the explanation, I hope that the JHF isn't too far away as this is quite an issue.