topic Re: R81.10 Bug? in Firewall and Security Management

R81.10 Bug?

biskit — Fri, 09 Jul 2021 13:11:16 GMT

Today I was Billy Big Balls and upgraded a customer from R80.20 to R81.10. All went well, except site-to-site VPN tunnels didn't come back up.

"zdebug" showed outbound IKE dropping on the cleanup rule.

Previously this worked via the Global "Accept outgoing traffic from the gateway" tick box.

This box is still ticked, but doesn't seem to be working... hence maybe this is a bug?

I've added a security rule to allow outbound IKE and the tunnels all came back.

So - one to be aware of if you go to R81.10 and your tunnels stay down.

Re: R81.10 Bug?

Danny — Fri, 09 Jul 2021 13:30:48 GMT

Thanks for sharing your experience with us.

Re: R81.10 Bug?

biskit — Fri, 09 Jul 2021 13:32:17 GMT

I've also informed TAC.

Re: R81.10 Bug?

the_rock — Fri, 09 Jul 2021 13:35:58 GMT

Just out of curiosity, can you send us a screenshot of the rule you had to change/add?

Re: R81.10 Bug?

JozkoMrkvicka — Fri, 09 Jul 2021 14:09:10 GMT

Did you install the policy after upgrade to R81.10 without doing any modifications with the policy? Simple policy push after cluster has been upgraded.

But anyway, here we go, freshly released/tested version and first issue within the day(s) ...

I am wondering if there will be any version without bugs...

Re: R81.10 Bug?

biskit — Fri, 09 Jul 2021 14:13:12 GMT

Yep - policy was installed.

The gateway had "Initial Policy" after the upgrade, so it needed a policy install...

Re: R81.10 Bug?

biskit — Fri, 09 Jul 2021 14:44:50 GMT

Hmm, that would give the customer away 😉

The rule was literally

SRC: My cluster object
DST: Azure object
Service: IKE (UDP/500)
Action: Accept

The tunnel came straight back up then. For 3 years prior, that rule hasn't been needed. So I'm assuming at this point that the "accept outgoing traffic from gateway" tick box is no longer doing what it says on the tin in R81.10?

Re: R81.10 Bug?

the_rock — Fri, 09 Jul 2021 14:47:43 GMT

Thats fair : ). I have a feeling this could be a bug if you never needed that sort of rule before. Personally, I cannot recall anyone having to do so going back to even before R54.

Re: R81.10 Bug?

the_rock — Fri, 09 Jul 2021 14:56:16 GMT

Just to be 100% sure, you are referring to the option I circled in the attached screenshot, correct?

Re: R81.10 Bug?

Tobias_Moritz — Fri, 09 Jul 2021 15:02:36 GMT

In you screenshot, this option is set to "Before Last".
Have you changed that from default?
@biskit : Do you also have "Before Last" there?

If I remember correctly, default in new databases for this option is "First" for many years. Maybe this is the problem why R&D did not found this problem in yet?

Re: R81.10 Bug?

the_rock — Fri, 09 Jul 2021 15:08:23 GMT

@Tobias_Moritz ...I am pretty sure by default it has always been "before last", even in older releases. I had a quick look for R77.30 and it also shows "before last". Also checked production R80.;40 and R81 and it shows the same.

Re: R81.10 Bug?

biskit — Fri, 09 Jul 2021 15:29:41 GMT

Yes - that's the right tick box. Mine is also set to "before last", which technically should still allow IKE out before hitting the cleanup rule. Maybe there's something TAC can do to troubleshoot why it isn't working. They did reply saying "there is no bug with VPN" so I replied back agreeing, and saying the problem is the tick box not doing what it did prior to the upgrade. I'll see where the SR takes me... Maybe some language barriers to overcome first 😂

Re: R81.10 Bug?

the_rock — Fri, 09 Jul 2021 15:35:41 GMT

Well, no language barrier there lol. Before last option does exactly what it says...which literally means that whatever is listed as you said, it would be accepted before clean up rule. Anyway, keep us posted, this is quite interesting.

Re: R81.10 Bug?

Kim_Moberg — Fri, 09 Jul 2021 15:57:57 GMT

Have’nt seen this error before.

what take of R81.10 do you run?

Have been running R81.10 take 335 in EA in production with almost 200 site2site tunnels and didnt see this issue.

I find this release very stable and but again maybe a combo of different setups can trigger different bugs that haven’t seen before.

BTW were you able to download the GA version from supportcenter? I have only been able to find the scalable version to download not the main train version.

Re: R81.10 Bug?

biskit — Fri, 09 Jul 2021 16:20:14 GMT

I'm on the same version - R81.10 Take 335, which I downloaded straight from CPUSE in the WebUI yesterday.

Re: R81.10 Bug?

JozkoMrkvicka — Sat, 10 Jul 2021 10:13:05 GMT

Do you see hits on this manually created IKE rule ?

Do you have implied rules logging ?

Re: R81.10 Bug?

biskit — Tue, 13 Jul 2021 09:34:57 GMT

Yes, I see hits on the new IKE rule.

I didn't have implied rule logging on.

As this is a production environment I'll struggle to go back and test stuff now without a maintenance window. I've got a case with TAC though (who still think I'm reporting a VPN problem rather than the global properties tick box problem) so I'll see where that leads me...

Re: R81.10 Bug?

RamGuy239 — Thu, 15 Jul 2021 18:27:02 GMT

I have no such issues on R81.10 myself. IKE traffic is hitting rule 0 / implied rule as per usual. Implied_rules.def lives on the management server. Did you do an in-place upgrade of the management server? Or did you go with an advanced upgrade?

Would be interesting to see your $FWDIR/lib/implied_rules.def file. There have been changes to the implied_rules.def in newer versions. Might the upgrade somehow have kept the old version instead of going with the newer one? This tends to be a possible issue when installing Jumbo Hotfixes that makes changes to implied_rules.def. If the JHF installation notices that you are not running a default implied_rules.def it will create a copy like implied_rules_HFA_R81_JUMBO_HF_take34.def and not overwrite the one you have which might cause issues.

Have you verified global properties on the R81.10 management to make sure "Accept Remote Access control connections" are still activated after the upgrade?