About "CPNotEnoughDataForRuleMatch" and connection reset

[JPR]

Hi there,

I've (partly) asked about this before (https://community.checkpoint.com/t5/Security-Gateways/quot-CPNotEnoughDataForRuleMatch-quot-and-quot...), but now I have another related question regarding this behvavior.

I have a service that connects to an external ip address, but every time the connection gets terminated by a reset from the destination. The log in my firewall says "Accept", however, it is getting  "terminated before the Security Gateway was able to make a decision: No SSL applicative data."  ("CPNotEnoughDataForRuleMatch").

As I got told in my other post (see link above) the behavior is by design and expected, however, I do have a question to why it happens.

The connection in question gets HTTPS Inspected and the log is as follows:

And the "Accept" ("CPNotEnoughDataForRuleMatch") log looks as below:

I tried to establish the connection with a Wireshark running on the client (not the firewall) and as far as I can see the handshake completes, but then it gets disconnected by a reset from the destination:

I have the same service on another endpoint WITHOUT HTTPS Inspection and there it connects fine.

So my question is: Is it possible that the packet somehow gets "malformed" in the HTTPS Inspection process and therefore the destination sends a reset back to us and kills the connection? Or is something different going on? I really can't figure it out!

Looking forward to your comments 🙂

Thanks!

[Lesley]

The error states there is no data in the connection, that is true according to the capture so the error is valid. 
You state without https inspection connections it works so we should focus on that. Could start is to see if the chosen encryption ciphers are accepted on both sides. In the capture the source is the fw right? That is starting with syn?

[JPR]

No, that's the client. I haven't done a capture on the firewall.

It seems like the Handshake and thus cipher suites goes through and it is after that, that the destination endpoint sends a reset packet. I just can't figure out why.

And again if I bypass HTTPSi it works, so my theory was that it had something to do with that...

[PhoneBoy]

Like @Lesley said, you should check from the gateway (specifically from the gateway to the remote site) with tcpdump.

[the_rock]

Check out below links, hope it helps. In short, this is literally never CP issue, as 3 way handshake is not completing, but its not because of the fw, but rather the fact that server did not send back syn-ack.

Andy

https://community.checkpoint.com/t5/Security-Gateways/quot-CPNotEnoughDataForRuleMatch-quot-and-quot...

https://community.checkpoint.com/t5/General-Topics/When-does-CPEarlyDrop-occur-with-ACCPET-action/m-...

[JPR]

I just don't understand why it then works if I bypass HTTPSi... 😞

[the_rock]

But then it sounds like its httpsi issue...

[the_rock]

Now that I think about it, say if you have httpsi enabled, you can always add bypass rule for affected IPs/destinations.

Andy

[JPR]

Yeah, you're right.

Are there any way of troubleshooting that myself or should I involve TAC? And is it okay to bypass destinations that causes problems? Obviously, it wont perform HTTPS Inspection on traffic between whatever source and destination hosts I have defined, but is it a known issue that HTTPSi in some situations will break the connection? It is not a general problem, but I do experience it occasionally.

Thanks.

[the_rock]

Yea, you said it exactly how I would put it. Not a generic problem, but it happens from time to time. Well...if destination causes problems, then I would NOT bypass it, better to involve TAC. However, if it does not cause issues, I would do it.

Andy

[JPR]

Well, it happens all the time for this specific destination and for now I have just created a bypass rule for it. I will consider getting TAC involved.

Thanks 🙂

[the_rock]

sounds good!

[the_rock]

Hey @JPR 

Any updates from TAC?

Andy

[JPR]

Hey,

Thanks for asking, but haven't gotten so far yet. Already have a couple TAC cases regarding some other stuff, so just waiting on those to get done before I open new ones 😉

[the_rock]

No worries, just keep us posted when you do.

Andy

[JPR]

I haven't been in contact with TAC, however, I tried enabling the extra logging as explained here: https://support.checkpoint.com/results/sk/sk113479

When doing that and then telnetting to an ip on 443 that gives me the "CpNotEnoughDataForRuleMatch" I get this expanded explanation:

I'm not sure I fully understand what it means, even after reading the explanation in the aforementioned SK.

Then I tried disabling rule 153 it mentions as a first possible rule match, but then it just says the next rule (154) is the first possible rule match. What those rules have in common, though, is that the source is a Network Group that consist of various hosts, networks and other network groups.

So to try to figure out if that particular group was the culprit, I created a new rule above the first instance where that group was used as source now with a single host as source and that particular ip (the one that otherwise fails with "CpNotEnoughDataForRuleMatch") as destination and action as Accept - and now it works, no error or anything!

It seems as that Network Group is what is causing this issue, but I can't figure out why. Do any of you have an idea as to why that could be the case? It is a group that are being used in many rules and generally it doesn't seem to cause any issues.

Looking forward to your ideas! 🙂

[the_rock]

Its essentially the same thing 🙂

Its long way of saying 3 way handshake is NOT completing, but its not the fw issue.

Andy

[JPR]

Ok, but how do you conclude it isn't a fw issue?

If that host is a member of that network group the conncetion fails with "CpNotEnoughDataForRuleMatch". If I make a specific rule with that host (not part of a group), it works, and I don't get the "CpNotEnoughDataForRuleMatch". To me it really does seem like a fw issue, but there might be something I don't get 🙂

[the_rock]

How do I know its not a fw issue? Valid question...my answer? Very easy 🙂

Here is how...when you see that log, do tcpdump and/or fw monitor and follow the packet. 100% it will show you that connection is going through the fw, but its not coming back, meaning the other side is NOT responding with syn-ack.

Andy

[JPR]

I did a trace on the fw with fw monitor and it seems as it does send a syn-ack back, however, it is malformed:

[the_rock]

In my experience, that would usually mean that it does not "accomodate" (for the lack of the better word) the actual protocol norms. You can try zdebug to confirm if fw is dropping anything on that IP, but I highly doubt it would show anything.

Andy

say if IP is 10.10.10.10, you can run fw ctl zdebug + drop | grep 10.10.10.10 (ctrl + c to stop)

Andy

[JPR]

Yeah, there's no drops doing fw ctl zdebug + drop | grep <ip>, so that checks out...

[PhoneBoy]

To completely avoid issues with "CpNotEnoughDataForRuleMatch" traffic has to match the rulebase on the very first packet.
That means the service in the rule must be a simple TCP/UDP service (e.g. https) and not an Application (say YouTube).
Rules for these "simple" services should be early in the rulebase due to how column-based matching evaluates the rulebase.

[JPR]

I still don't understand why it says it fails with "CpNotEnoughDataForRuleMatch"  and that this is the first possible rule match:

If I make a new rule above that with a single host (that is also part of that group in the rule above) and sets destination as that ip that fails then it goes through just fine. I don't get what causes that. It might not be the rulebase but something else. but it most definetely seems to matter if it is part of that group or not and something the firewall is doing (and migh very well be a configuration issue, I just don't know where to start look...).

The rule 

[the_rock]

I would still involve TAC when you get a chance to verify all this.

Andy

[JPR]

I'll do that, but thanks for the input anyway 🙂

As I went through it all, I see I only get the extended log message with "missing classifier objects"  with the "Insufficient data passed." error and not when it's the "No SSL applicative data." error (both "CpNotEnoughDataForRuleMatch", though)  so that's a mixup on my part.

I'll try and remember to update you when I get it past TAC and hopefully get it resolved 🙂

[the_rock]

When you say "missing classifier objects", what do you mean by that exactly?

Andy

[JPR]

That's from my message above.

Basically I did some more testing, and I took the destination ip from a connection that failed with "CpNotEnoughDataForRuleMatch" and the reason "Insufficient data passed." (my original post was "CpNotEnoughDataForRuleMatch" and "No SSL applicative data.").

If I activate the extended logging and try to connect to that specific ip I get this message in the fw:

And that's where I get the "Missing classifier objects" from.

The rule 153 it mentions as first possible rule (match) looks like this:

If I make a new rule with the source as a single host and that particular ip, then it works:

Hope it makes sense. I myself am starting to get a little confused 😄

[PhoneBoy]

Your example demonstrates exactly what I said previously.

You are using a Custom Application/Site in the service column in the "problematic" rule.
These rules require multiple packets to conclusively say whether it matches that rule, a different one, or none at all.
When the connection closes BEFORE that happens, you will get the "CpNotEnoughDataForRuleMatch" message.
Again, 100% expected behavior.

The "simple" rule added above the so-called problematic one is exactly how you resolve this problem.

[JPR]

Okay, thanks, that makes total sense.

Is there a particular reason for that behaviour? Because otherwise I would potentially have to make a lot of separat rules or?

Oh and by the way, then the issue is not the Network Group as source (when it fails) but the use of Custom Application/Site and URLs in the Services/Application column, is that correct?

Again, thanks 🙂