Possible SecureXL issue on inter-VS traffic

[Alex-]

Platform: 26000 VSX

OS: R80.30 3.10 Take 155

Please consider the following network design. Interface names are simplified for the sake of clarity.

Both VS have mutual static routes for SRC and DST pointing to each other's end of the VSWITCH and there's traffic flowing in both directions.

Now the issue: an application is hosted in VS2. End-users from VS1 launch a java client that connects on a custom port (this is an in-house program) to the server in VS2. For some reason, traffic is classified as SSLv3 at the application level, maybe because of the content of the connection, but is still accepted in the logs (both Security and Application policies allow that flow), but the application never connects. I also get the Alert in the logs stating that the domain can't be resolved and I should check my DNS configuration.

fw monitor shows that we don't go further than "i" on VS1 even though it shows as accepted in the tracker. 

Disabling SecureXL on VS1 immediately solves the issue and the application launches with a flurry of iI-Oo in FW monitor.

I will upgrade the VSX to Take 191 in the coming days and report if it solves the issue, but I don't know if others here would have seen a similar problem and have a suggestion. The setup itself isn't new, it has been migrated to a 26K-base a few months back and according to the customer, this issue appeared a few days ago.

[Kaspars_Zibarts]

Sounds like a TAC case to me in general.

As for DNS errors / logs - make sure that VS itself can reach DNS on both UDP and TCP port 53. At least it was the problem for us - TCP to DNS was was not explicitly allowed and implied rules was too far low in the rulebase (basically there was a drop rule before it):

[Alex-]

Hi Kaspars,

I thought of a TAC case but since there's a new GA Take, I suppose I'd get the advice to upgrade first.

Thanks for the DNS tip by the way, I will give it a look.

Alex

[Timothy_Hall]

> fw monitor shows that we don't go further than "i" on VS1 even though it shows as accepted in the tracker.

> Disabling SecureXL on VS1 immediately solves the issue and the application launches with a flurry of iI-Oo in FW monitor.

Hmm sounds like something is going wrong when the connection is being offloaded to SecureXL after i.  Try this:

1) Use the fw monitor -F srcIP,0,dstIP,dstPort,6 syntax which will allow you to continue to see the traffic even after offload into SecureXL.

2) Is fw ctl zdebug drop showing any reason for the traffic disappearing when SecureXL is enabled?  This command will show drops inside SecureXL/sim as well.

3) Finally there is always excluding this specific traffic from acceleration by SecureXL without completely disabling SecureXL completely which is never a good idea: sk104468: How to disable SecureXL for specific IP addresses.  But you should still investigate with TAC why this is happening as this is just a a workaround.

[Alex-]

Hi Tim,

No drops seen in zdebug. Interestingly, I got in the meantime a similar issue on another VS for a specific flow completely unrelated to the first issue.

Disabling SecureXL for that VS also did the trick.  While I understand it's preferable to have SecureXL on, I will leave it like this for now as I'm going to the latest GA Take ASAP and will check if it solves the issue or not, in which case I'll take another course of action.

I will also open a case anyway, now it's more than just one flow, to get a head start should Take 191 not provide a solution. I certainly don't want to have to turn off SecureXL on each VS.

Alex

[PhoneBoy]

If disabling SecureXL solves your problem, TAC needs to be involved.

[HeikoAnkenbrand]

Hi @Alex-,

I noticed the same problem with R80.30 on a 26000 appliance and VMWare open server.

At my LAB (VMWare) it works with JHF 191.

[Alex-]

Take 191 didn't help, so I will go on and open a TAC. I have in the meantime applied @Timothy_Hall's recommendation to use sk104468 as disabling fwaccel broke some non TCP traffic and of course to avoid all traffic going f2f.

Thanks all for the replies.

[Alex-]

After upgrading the VSX cluster to Take 196 and removing the f2f exception for the IP's in table.def, the flows work again and Smartlog now shows them in both VS.

The interesting thing is that I still see only "i" in the incoming VS after doing so, although the users have tried from multiple sources which now all work. I will follow-up with the SR I've open.