Inline Layer: Bug, by design or....?

[Morten_O]

Hi,

Gateways are running R81.10 with Jumbo 7x-something.
I have a setup with some very old and legacy H.323 devices, that stops working, if the word 'inspection' is even mentioned.
So the rule allowing the H.323 server access to the clients just allows tcp-high-ports, like this.

Everything works, and traffic on e.g. port 1720 (H323) is tcp-high-ports in the log.

As part of housekeeping of the rulebase, an inline layer was created, where the top rule cover traffic from east to west. The H323-servers are part of east and the H323-clients are port of west, so we moved the original rule into the layer, and it looks like this:

Traffic from the servers to the clients now hits this inline rule 8.1, but the H323 devices stopped working.

In the log I could see, that traffic on port 1720 was no longer tcp-high-ports, but H323, which means it is being inspected (the log even told that it was h245 signalling). Then we moved rule 8.1 outside the layer again, and the devices worked again.

Now the question: How come, that this traffic is being inspected when the rule is inside the layer, and not inspected then outside a layer? In the log, port 1720 was matched to H323 not even H323_any, which partly could be explained by the 'Any' in the top layer.

[G_W_Albrecht]

I would suggest to ask CP TAC !

[Timothy_Hall]

Locate all service objects matching port 1720, what is the state of "Match for Any" on the Advanced screen of each of them?  I'm thinking that service H323 is set for Match for Any, and the "Any" in your top/parent rule is causing that service to be selected.  If you explicitly place tcp-high-ports in the service field of the top/parent rule it should work as you expect.

[Morten_O]

I already checked, as that was also my thought. But for some reason I didn't check very well - H323 has match for any and H323_any doesn't have match for any - no idea why it's reverse. But that explains why it's matches H323 and not H323_any. So the parent-rule sees port 1720 and matches on 'Any' and then uses H323, but no matter what the parent-rule thinks, it should be the 'real' rule that makes the final decision and use tcp-high-ports, and not just inherit it from the parent.

I'm wondering if this by design, 'a feature' or a bug? It should be pretty easy to reproduce for CP.

[Timothy_Hall]

Yeah that is a good point, I'd kind of expect it to use the "final" service type found in a sub-rule and not the parent service as well.  Probably has something to do with how the Unified Policy is compiled.

[Bob_Zimmerman · Accepted solution]

H323_any is actually a different protocol handler. The name is a bit confusing, agreed. It doesn't have anything to do with Any, the value you can specify in a rule's Service field.

As for why it's still catching the protocol handler, my bet is services with the protocol not set don't try to set the protocol handler to "None". I think they instead don't set the connection's protocol handler, which leaves it at the default of no handler.

If I'm right, the top level rule matches H323 (since it's set to match for Any), which sets the protocol handler. The inner rule then matches tcp-high-ports which doesn't have a protocol handler set, so it doesn't try to write to the protocol handler again. This leaves it at the default of h323, since that's what was already set when the connection hit the rule with that service object.

As a workaround, you could make service objects for IP protocols 1, 6, 17, and any others you might need and use those in the top-level rule instead of Any.

[the_rock]

I think all the guys made excellent points, though as @G_W_Albrecht suggested, I would verify all the with TAC, so you can get an official CP response.

Andy

[Morten_O]

Opening a SR is like playing the lottery - sometimes you win but most times you lose 🙂

I'll give it a try, and update here, if something interesting comes out.

Thanks

[the_rock]

I see your point about opening a case...never though about it that way, but I get it lol

[Morten_O]

To be honest, I can't really deside if I won or lost.... 😆

The answer from TAC:

"It is by design for sure. Because I confirmed it with my seniors internally. As I said, it is by design."

[the_rock]

@Morten_O lol

Maybe its just me, but that TAC answer would not inspire a whole lot of confidence : - )

[Bob_Zimmerman]

Sounds like my guess above is correct (or they read Checkmates and it sounded plausible? 😜). In the parent layer, instead of using Any, try using services which cover all the protocols you want to be able to use. I personally wouldn't edit the H323 service to remove "Match for Any", since that has broader implications. If you're sure nothing else in that management domain needs the protocol inspection without the explicit service object, it should be fine.

If that changes the behavior, then I guessed right, and you have a workaround. Might be a good RFE, since the current behavior definitely violates the principle of least astonishment.

[Morten_O]

Yes, I'm pretty sure, that changing the services on the parent-layer would solve this, but the H323-devices has been challenged enough, so I'm not going to test any more 🙂 But this would mean, generally not to use 'Any' services in parent-rules. Other services with 'match for any' and with inspection might have the same challenge, e.g. SIP.

I still haven't released the TAC-engineer - I'm not letting him close this case with that vague statement.

It's pretty wild, that the same rule behaves differently, depending on the placement inside a general layer or right above that layer.

Another thought - if I create a specific service with tcp port 1720 (the H323 port), with no inspection and no 'match for any', and use that specific service instead of tcp-high-ports in the inline-rule (8.1), I'm not sure, I can 100% predict what the outcome is - would it use my port-1720 or would it inspect with the H323-service...... If it follows the logic from tcp-high-ports, it would inspect, and not use my 1720-port.

[the_rock]

I would honestly pursue this further with TAC, there has to be "happy medium" somewhere...

[Bob_Zimmerman]

Like I said, I think using a service without a protocol handler set causes traffic which matches a rule with that service to not set the protocol handler for the connection, rather than to unset the protocol handler. That is, it leaves the connection alone rather than setting it to not use a protocol handler.

Once a protocol handler is set for a connection, there is no way to get it back to not having a protocol handler. You have to avoid the handler being set in the first place.

[the_rock]

Thats actually very good point @Bob_Zimmerman 👍

[PhoneBoy]

I believe anything with a Protocol Handler is handled by the firewall in the slow path.
The decision to use slow path must be also be made on the first packet (i.e. the TCP SYN).
Which means: if traffic matches a rule that includes a protocol handler in a prior layer (inline or ordered), that protocol handler must be used for the entire connection, even if it's not necessary in the end. 

[the_rock]

Learned something new...thats excellent to know! 👍

[_Val_]

How does your inline layer 8 cleanup rule look?

[_Val_]

Hi @Morten_O, after reading the responses and reviewing your support case, some points:

1. It is certainly NOT the best practice to use tcp-high-ports service for H323 connectivity

2. The difference between H323 and H323_any is documented and elaborated in sk20371, please review it and let me know if it is clear enough.

3. As you did not show your Layer 8 cleanup rule, I might assume it does not exist or has a "Drop" default action. In such cases, some of the traffic that the parent rule would otherwise accept will be dropped. 

4. You can also refer to the VoIP ATRG article, if you need, which covers H323 rather in depth, among other protocols.

I also find the TAC response a bit less than convincing, although "this is by design" is the correct answer in this case. If you seek a more elaborate justification, feel free to seek it through the SR, and ask for escalation, if the answer is not provided.

That said, I would rather chase the appropriate service definition in the policy rules if I were you.

[Morten_O]

Hi @_Val_,

thanks for joining.

Yes, in the ideel world, everything should be inspected and survive, but not all devices are coded optimal, and unfortunately this is the case for these H323-devices (fortunately they are to be switched to SIP-devices soon)

The clarification of H323 and H323_any is fine.

The layers always ends with a manually added drop rule with log - nothing in relation to H323 servers or clients was dropped, when we had the issue.

I know the ATRGs, but it really doesn't explain the behavour we saw.

"That said, I would rather chase the appropriate service definition in the policy rules if I were you."  Sorry, but I'm not really sure what you mean - it's hard to get it to work without the tcp-high-ports - well, the port-range could probably be narrowed down a little.

Let's see what happens next in the SR 🙂

[_Val_]

Inline layer cleaning rule with drop makes the layer super restrictive. Try changing it to accept and see if the layer starts working for you. The quote you used is a suggestion to use H323 services instead of tcp-high-ports