Traceroute logs on Layered policy - R81.10

[Scottc98]

I have this weird issue I have been trying to get around in my R81.10 lab.   I have a layered policy that I have been using for all internet traffic (Main rule No 13) .   

source = is all of my LAN networks

destination = Non-private IP addresses (negated RFC-1918 ips)  ** NOTE** also used the "Internet" application service with same results

This directs all traffic destined for the internet to a inline layer called "TO_Internet Layer_V2"

On this inline layer, I have a rule (13.11) that looks for the service "traceroute" from any source to any internet destination (note: used 'any', Internet application object and the current "All_Internet" one in the screenshot.

It seems that no matter what I do, the logs for any traceroute traffic out to the internet hits the main master rule for the layer (Rule 13) and not the actual rule within the layer itself (13.11).

All other flows that route through this internet inline layer policy seems to update its hit count and record against is respective sub rule # (13.x).

Has anyone seen anything like this before?   It looks like if I move the traceroute rule out of the inline layer and place as a normal policy rule (i.e above rule #13), it matches......but if I use it in inline layer itself, it just will not hit the actual rule (13.11).

MGMT and GW at R81.10 Take 22.    

Thanks in advance.  

[the_rock]

I would try to restrict rule 13.11 a bit, if you can. So, instead of any source, put some subnets there and maybe restrict dst as well. By the way, I see that rule 13.11 has zero hits, so does not look like it was ever hit. Technically though, the way layering works, as Im sure you know, is that traffic would hit parent rule and then get accepted on given correct child rule, in this case 13.11. Since thats failing n your case and only being accepted on parent rule 13, I would definitely try what I mentioned and see if that helps. Logically, to me anyway, it appears child rule (13.11) is actually less restrictive than parent rule, except for the service.

Andy

[Scottc98]

No change if i get more grandular with the rule itself: 

Source: LAN Networks (Same as main layer)

Dest: Internet (Same as main layer)

Service:  Traceroute

I have a NTP rule below this (13.13) that has been working with no issues (attached below) and it hits with no issues.    Its seems like its something explicit to the 'traceroute' service object used.  

[the_rock]

2 quick question for you.

1 - What is negated group in layered rule 13?

2 - if you search in logs for service traceroute for say last 7 days, does ONLY rule 13 show up?

Andy

[Scottc98]

1) The negated group I used was a list of all RFC-1918 blocks (i.e anything but 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12).  The "internet" application object and the "all_internet" one was also tested

2)  Correct.   Any traceroutes to external internet destinations only gets logged under rule 13.  

[Alex-]

Traceroute isn't using a port but rather an Other Service with UDP (Protocol 17) and a pattern matching declaration. That might be a limitation of inline logging.

[Timothy_Hall]

What does the cleanup rule (if any) look like for layer To_Internet_Layer_V2?  If you don't have one and it is hitting the implicit cleanup rule, I imagine it would only log the parent rule 13 in that case.

Also make sure that you are using the appropriate OS-specific traceroute/tracert for your test traffic.  As noted by @Alex- standard Unix/Linux traceroute uses a UDP datagram as the "probe" packet with a low TTL, while Microsoft Windows uses a ICMP echo request with a low TTL instead. Not sure why Microsoft chose to make this different but whatever...

[Scottc98]

@Timothy_Hall 

The Inline layer has a deny all rule at the end (Any/Any/Any/Drop).   The traceroutes are initiated from a MAC laptop (OSX 10.15.7) terminal window (i.e. traceroute www.yahoo.com with no other flags set)

I wonder if @Alex-  is on to something on the possible limitation of the inline logging.   Its obviously hitting the defined traceroute service object per the output but just against the parent rule (13) and there is no other rule within the layer it could match.   If I cut that exact traceroute rule out and put above the inline parent rule, it matches (see attached).  

[Timothy_Hall]

When you put the traceroute rule back into the inline layer, initiate test traceroute traffic and find the log, if you then double-click to open the log card and click the "Matched Rules" tab there, does it show the sub-rule match then?

[Scottc98]

Good call @Timothy_Hall    

I looked at the card and it has an very interesting line

The first one looks like any other inline card; it shows the rule 13, the layer being the main policy, the rule name matching (i.e "Internet-OUTBOUND" in my use case and the action is "inline".    That all looks normal......until the next line

For this one:

*******

Rule - Blank (no data)

Layer - "TO_Internet Layer_V2" 

Rule Name - "CPNotEnoughDataForRuleMatch"

Action - Accept

*****

No idea on that rule name there (seems like some internal one) 

Regardless, the flow has to be referencing the 13.11 Traceroute rule but just not setting the hit/log against it.   As soon as I disable that explicit 13.11 rule, I can no longer traceroute outbound.   

[the_rock]

@Timothy_Hall gave you good place to look...just an idea, if you cant find that rule name in smart console, what if you open guidbedit and search the whole table? I'd be curious to see if it comes up there at all...

Andy

[Scottc98]

When you click on the "CPNotEnoughDataForRuleMatch" from the log card in Smartconsole, it directs you back to Rule 13 where the master layer rule sits.

Unable to query this object in guidbedit....search comes up empty

[Alex-]

Have a look at sk113479.

[the_rock]

Thats really good sk...I went through my emails and saw I had same issue last year with a customer and that sk fixed it.

[Scottc98]

Thanks @Alex-  and @the_rock !   I should have looked further at that card to see that SK mentioned.   

I ended up turning on the extended reason setting (fw ctl set int up_log_extended_reason_for_incomplete_match 1) on the GW and found this:

Connection terminated before the Security Gateway was able to make a decision: Insufficient data passed.
To learn more see sk113479.
First possible rule:
Layer: TO_Internet Layer_V2, Rule: 7.
Missing classifier objects:
1: APPLICATION

Rule 7 in my internet layer was my category block rule.    As soon as I moved my traceroute rule above that, it started hitting.

Anyone know if that 'extended reason' flag has any ill effects to keep on?   That seems like a great 'default' to keep 🙂

Thanks again! 

[the_rock]

I had my customer keep that option and they never had a problem for more than a year now. Also, furthermore, we did ask TAC guy about it and he said it was not intrusive at all to keep it, which to me, logically anyway, makes sense. I personally dont see any issues with that setting at all.