R81.10 Bug?

[biskit]

Today I was Billy Big Balls and upgraded a customer from R80.20 to R81.10.  All went well, except site-to-site VPN tunnels didn't come back up.  

"zdebug" showed outbound IKE dropping on the cleanup rule.

Previously this worked via the Global "Accept outgoing traffic from the gateway" tick box.

This box is still ticked, but doesn't seem to be working...  hence maybe this is a bug?

I've added a security rule to allow outbound IKE and the tunnels all came back.

So - one to be aware of if you go to R81.10 and your tunnels stay down.

[Danny]

Thanks for sharing your experience with us.

[biskit]

I've also informed TAC.

[the_rock]

Just out of curiosity, can you send us a screenshot of the rule you had to change/add?

[biskit]

Hmm, that would give the customer away 😉

The rule was literally 

SRC:  My cluster object
DST:  Azure object
Service:  IKE (UDP/500)
Action:  Accept

The tunnel came straight back up then.  For 3 years prior, that rule hasn't been needed.  So I'm assuming at this point that the "accept outgoing traffic from gateway" tick box is no longer doing what it says on the tin in R81.10?

[the_rock]

Thats fair : ). I have a feeling this could be a bug if you never needed that sort of rule before. Personally, I cannot recall anyone having to do so going back to even before R54.

[the_rock]

Just to be 100% sure, you are referring to the option I circled in the attached screenshot, correct?

[Tobias_Moritz]

In you screenshot, this option is set to "Before Last".
Have you changed that from default?
@biskit : Do you also have "Before Last" there?

If I remember correctly, default in new databases for this option is "First" for many years. Maybe this is the problem why R&D did not found this problem in yet?

[the_rock]

@Tobias_Moritz ...I am pretty sure by default it has always been "before last", even in older releases. I had a quick look for R77.30 and it also shows "before last". Also checked production R80.;40 and R81 and it shows the same.

[biskit]

Yes - that's the right tick box.  Mine is also set to "before last", which technically should still allow IKE out before hitting the cleanup rule.  Maybe there's something TAC can do to troubleshoot why it isn't working.  They did reply saying "there is no bug with VPN" so I replied back agreeing, and saying the problem is the tick box not doing what it did prior to the upgrade.  I'll see where the SR takes me...  Maybe some language barriers to overcome first 😂

[the_rock]

Well, no language barrier there lol. Before last option does exactly what it says...which literally means that whatever is listed as you said, it would be accepted before clean up rule. Anyway, keep us posted, this is quite interesting. 

[JozkoMrkvicka]

Do you see hits on this manually created IKE rule ? 

Do you have implied rules logging ?

[biskit]

Yes, I see hits on the new IKE rule.

I didn't have implied rule logging on. 

As this is a production environment I'll struggle to go back and test stuff now without a maintenance window.  I've got a case with TAC though (who still think I'm reporting a VPN problem rather than the global properties tick box problem) so I'll see where that leads me...

[JozkoMrkvicka]

Did you install the policy after upgrade to R81.10 without doing any modifications with the policy? Simple policy push after cluster has been upgraded.

But anyway, here we go, freshly released/tested version and first issue within the day(s) ...

I am wondering if there will be any version without bugs...

[biskit]

Yep - policy was installed.  

The gateway had "Initial Policy" after the upgrade, so it needed a policy install...

[Kim_Moberg]

Hi 

Have’nt seen this error before.

what take of R81.10 do you run?

Have been running R81.10 take 335 in EA in production with almost 200 site2site tunnels and didnt see this issue.

I find this release very stable and but again maybe a combo of different setups can trigger different bugs that haven’t seen before.

BTW were you able to download the GA version from supportcenter? I have only been able to find the scalable version to download not the main train version.

[biskit]

I'm on the same version - R81.10 Take 335, which I downloaded straight from CPUSE in the WebUI yesterday.

[RamGuy239]

I have no such issues on R81.10 myself. IKE traffic is hitting rule 0 / implied rule as per usual. Implied_rules.def lives on the management server. Did you do an in-place upgrade of the management server? Or did you go with an advanced upgrade?

Would be interesting to see your $FWDIR/lib/implied_rules.def file. There have been changes to the implied_rules.def in newer versions. Might the upgrade somehow have kept the old version instead of going with the newer one? This tends to be a possible issue when installing Jumbo Hotfixes that makes changes to implied_rules.def. If the JHF installation notices that you are not running a default implied_rules.def it will create a copy like implied_rules_HFA_R81_JUMBO_HF_take34.def and not overwrite the one you have which might cause issues.

Have you verified global properties on the R81.10 management to make sure "Accept Remote Access control connections" are still activated after the upgrade?