Policy Based Routing issue

[marcyn]

Hello CheckMates,

I'm struggling with an issue with PBR ... and I hope you will be able to put some light on this one.

Customer wants that some traffic should go different path then via default route ... so it's obvious that this is case for PBR.
But it doesn't work as expected.

I was able to recreate similar environment in my lab and it looks like this:
1) I have a Access Control rule:

 So as you can see any traffic FROM debian (10.20.200.222) will be matched.
2) PBR was configured to be Application Aware (it's a shame that this is hidden by default) - so I can create a rule in PBR that will tell gateway that it should match rule 1:

 3) Table 1: router forward traffic to different router (in my case 10.99.99.203)

So ... as you can see ... nothing odd, just a normal, simple PBR config.

This 10.99.99.203 router doesn't have network 10.20.200.0/24, but knows who has this network - based on static-route entry:

 10.99.99.234 = my gateway, where I created PBR, etc.

Ok, now you know how it is configured ... but I will explain this below (just to be sure, you know everything 😛 ) :
1) traffic (for example ping to 8.8.4.4) from 10.20.200.222 arrives on incoming interface on my router (10.99.99.234)
2) Access Control policy knows that it should be accepted and not send to external interface of this gateway, but to interface where 10.99.99.203 is connected
3) then this traffic arrives on router 10.99.99.203 and is accepted and is send to the internet
4) return traffic from 8.8.4.4 arrives on 10.99.99.203 and because this router doesn't have 10.20.200.0/24 network it will not forward this reply directly to "debian" ... but it will forward this traffic to 10.99.99.234 (via static-route)
5) so ... return traffic from 8.8.4.4 to 10.20.200.222 arrives on my gateway on interface where 10.99.99.203 is connected, and then it should go to "debian"

Simple...

But it doesn't work like that.

Take a look how it works (steps 1-5 from the above description):

ad. 1)
root@debian:~# ping 8.8.4.4
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.

ad. 2)
SG (10.99.99.234):
[vs_0][fw_1] eth2:i[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
[vs_0][fw_1] eth2:I[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
[vs_0][fw_1] eth0:o[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
[vs_0][fw_1] eth0:O[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
(traffic leaves 10.99.99.234 and goes to 10.99.99.203)

ad. 3)
router (10.99.99.203) - also Check Point:
[vs_0][fw_2] eth0:i[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53362
[vs_0][fw_2] eth0:I[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53362
[vs_0][fw_2] eth1:o[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53362
[vs_0][fw_2] eth1:O[44]: 203.0.113.203 -> 8.8.4.4 (ICMP) len=84 id=53362
(as you can see there is a NAT)

ad. 4)
[vs_0][fw_2] eth1:i[44]: 8.8.4.4 -> 203.0.113.203 (ICMP) len=84 id=0
[vs_0][fw_2] eth1:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_2] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_2] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
(traffic leaves 10.99.99.203 and goes to 10.99.99.234)

ad. 5)
And .. here is my issue:
[vs_0][fw_1] eth0:i[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
(traffic enters 10.99.99.234 on eth0 interface ... and stays here ... it is not routed to eth2 interface where this debian host exists)

fw ctl zdebug + drop:
@;137323;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1 8.8.4.4:0 -> 10.20.200.222:19315 dropped by fw_first_packet_state_checks Reason: ICMP reply does not match a previous request;

Do you have any idea what is wrong here ?
In case router 10.99.99.203 would have this network 10.20.200.0/24 there will be no issue at all, because return traffic would go directly from this router to "debian" ... but in this case this return traffic needs to go back to 10.99.99.234.

And By The Way ... another issue:
(this rule was disabled in above example ... that's why I had no NAT)
In case Customer would have Manual NAT rule ... not Automatic one ... PBR is doing NAT... and I have absolutely no idea why ?
For Automatic NAT - it will not do NAT for this PBR.
Take a look how my Manual NAT rule looks like:

 So ... normal SNAT rule - any traffic from 10.20.200.0/24 to ExternalZone should be hidden behind Gateway's external IP.
But ... 10.99.99.203 router (this one in PBR rule) ... is behind interface that has InternalZone 🙂
Why this rule matches this traffic ? ... it looks like NAT is matched before routing ... and because of that Gateway decides that 8.8.4.4 is reachable via eth1 (external) interface ... not eth0 (here is this 10.99.99.203).

Anybody ? 🙂

--
Best
m.

[the_rock]

Hey @marcyn 

Grext explanation btw! Here is one question that pops up in my head...what does route to say 8.8.8.8 look like from debian host?

Andy

[marcyn]

Hi Andy,

debian host knows nothing about this "other router" ... it's gateway is of course this one 10.99.99.234.
So any traffic from debian (10.20.200.222) directed to the Internet will go directly to Gateway with IP 10.99.99.234.

--
Best
m.

[the_rock]

K, fair enough, so what does ip r g 8.8.8.8 show on the fw?

[marcyn]

Of course default route 😀

[the_rock]

Just making sure lol. Anyway, I know that zdebug drop you posted may refer to nat or possibly assymetric routing. Let me see if I can try this in the lab.

Andy

[emmap]

The PBR side of things seems to be working, but you have a stateful inspection drop for ICMP. Your final fw monitor though shows your o and O so it seems like it is actually going somewhere - can you validate with tcpdumps on eth0 and eth2 on that device?

[marcyn]

Yes,
That's right ... because I see o and O it is obvious that this traffic was not blocked by FW.
However there should be routing in place ... this traffic should be forwarded to eth2.

Anyway ... you asked about tcpdump - no problem, here you have it:

[Expert@SG:0]# tcpdump -nnei eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:57:02.807595 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 51904, seq 1, length 64
10:57:02.823689 f2:f5:43:c6:50:8b > bc:24:11:fb:ea:30, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 1, length 64
10:57:02.823780 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 1, length 64
10:57:03.819566 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 51904, seq 2, length 64
10:57:03.834611 f2:f5:43:c6:50:8b > bc:24:11:fb:ea:30, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 2, length 64
10:57:03.834679 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 2, length 64

As you can see ther is a pair request+reply ... and another reply.
First reply is FROM f2:f5:43:c6:50:8b  = router behind PBR (10.99.99.203)
And it is addressed TO bc:24:11:fb:ea:30 = debian
And it's exactly what it should be (in my opinion) !

[Expert@CP-GW1:0]# ip a l eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether f2:f5:43:c6:50:8b brd ff:ff:ff:ff:ff:ff
inet 10.99.99.203/24 brd 10.99.99.255 scope global eth0

root@debian:~# ip a l ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether bc:24:11:7e:c1:0c brd ff:ff:ff:ff:ff:ff
altname enp0s18
inet 10.20.200.222/24 brd 10.20.200.255 scope global ens18

But then ... this first reply instead of beeing routed to eth2 where this debian exists ... it is replied but this time:
FROM debian TO router 10.99.99.203 ... why ? ... it is as if this PBR rule interfere with this reply as if it was request...
It looks like this reply bounced back to the router behind PBR.

In case you wonder - arp table in my 10.99.99.234 = "source gateway" (where PBR is declared):
[Expert@SG:0]# arp -an
? (10.99.99.203) at f2:f5:43:c6:50:8b [ether] on eth0
? (10.20.200.222) at bc:24:11:7e:c1:0c [ether] on eth2

Based on arp table ... it's obvious that this gateway exactly knows where this bc:24:11:7e:c1:0c is located... => eth2,
So why it bounced back this reply to f2:f5:43:c6:50:8b (from bc:24:11:7e:c1:0c).

I don't get it ...

Ah ... if you wanted - no problem - tcpdump from eth2:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
11:11:28.963040 bc:24:11:7e:c1:0c > bc:24:11:22:48:4a, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 10360, seq 1, length 64
11:11:29.963344 bc:24:11:7e:c1:0c > bc:24:11:22:48:4a, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 10360, seq 2, length 64

Only requests here ...

bc:24:11:22:48:4a = obviously gateway's IP on eth2 (10.20.200.254) = debian's default gateway


--
Best
m.

[the_rock]

You got me so curious now, I want to try this in the lab. But in the meantime, just a thought...what if you added a route to 10.20.200.0/24 on 10.99.99.203?

Andy

[marcyn]

Andy, if there would be no route to 10.20.200.0/24 on 10.99.99.203, it would not work at all ... as this router doesn't have this network ... and I mentioned that in my opening message 😀 

So of course that 10.99.99.203 has a route to 10.20.200.0/24 with 10.99.99.234 as a gateway to this route.

BR, m.

[the_rock]

Let me try this in the lab tomorrow.

[the_rock]

@marcyn I replicated this and its exact same issue as you. I tested on R82 jumbo 41 (latest) one. Now its good challenge to fix it : - )

Andy

[the_rock]

@marcyn 

Here is how I fixed it

Enabled this kernel parameter -> fw ctl set int fw_allow_asymmetric_routing 1

Andy

[marcyn]

Hi Andy,

Thanks for your work on this one.
Because it's the 3rd environment where the same issue occurs ... it's not a coincidence 🙂

I'm glad that you found a fix ... but ...
1) why asymmetric routing ? ... request:  incomming = eth2, outgouing = eth0, and as we saw reply goes from eth0 and should go to eth2 ... I don't see here asymmetric routing 🙂

SG (10.99.99.234):
10:04:19.480446 In bc:24:11:7e:c1:0c ethertype IPv4 (0x0800), length 100: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 58102, seq 1, length 64
10:04:19.480730 Out bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 58102, seq 1, length 64
10:04:19.497326 In f2:f5:43:c6:50:8b ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64
10:04:19.497416 Out bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64

CP-GW1 (10.99.99.203):
10:04:19.530303 In bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 58102, seq 1, length 64
10:04:19.530597 Out 62:93:df:1c:11:3b ethertype IPv4 (0x0800), length 100: 203.0.113.203 > 8.8.4.4: ICMP echo request, id 26668, seq 1, length 64
10:04:19.546660 In 26:cb:19:cd:cc:57 ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 203.0.113.203: ICMP echo reply, id 26668, seq 1, length 64
10:04:19.546713 Out f2:f5:43:c6:50:8b ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64
10:04:19.546957 In bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64    <-- ???

So based on the above:
1) seq1 with id = 58102 was received on incomming interface of my 1st gateway (10.99.99.234)
2) this packed was then forwarded by PBR to 2nd machine (10.99.99.203) /we can clearly see the sam id on incomming interface of 10.99.99.203/
3) then 10.99.99.203 sends this packet to the internet (the is different id = 26668) and it receives reply to id = 26668
4) this reply is then send from 10.99.99.203 to 10.99.99.234 with id = 58102, and it is received by 10.99.99.234 on eth0 interface with valid and expected id = 58102

For me there is no asymmetric routing here...

5) then 10.99.99.203 for some reason receives the same reply with id = 58102 ... but as an incomming messega on eth0 interface from 10.99.99.234:
[Expert@CP-GW1:0]# arp -na | grep bc:24:11:fb:ea:30
? (10.99.99.234) at bc:24:11:fb:ea:30 [ether] on eth0

And this a mistery for me...

Anyway ... I decided to test your fix (I don't get it why it could fix this ... but ok, why not to test).
And unfortunately on R81.20 it's not working:
[Expert@SG:0]# fw ctl set int fw_allow_asymmetric_routing 1
Set operation failed: failed to get parameter fw_allow_asymmetric_routing
set: Operation failed
I checked and there is no such parameter indeed.

But ... Customer has R82 ... so ... I can check this out on R82 as well 😉
But2 ... I would also like to understand this, why asymmetric rounting ... 🙂

--
Best
m.

[marcyn]

Unfortunately on R82 JHF41 this kernel parameter is not supported in my machine as well...

[Expert@SG:0]# cpinfo -y FW1 | grep Take

This is Check Point CPinfo Build 914000250 for GAIA
HOTFIX_R82_JUMBO_HF_MAIN Take: 41
[Expert@SG:0]# fw ctl get int fw_allow_asymmetric_routing
Get operation failed: failed to get parameter fw_allow_asymmetric_routing
get: Operation failed
Killed
[Expert@SG:0]# fw ctl set int fw_allow_asymmetric_routing 1
Set operation failed: failed to get parameter fw_allow_asymmetric_routing
set: Operation failed
Killed
[Expert@SG:0]#

So ever that I don't get it why this fixed it for you Andy ... it will not work for me 🙂

--
Best
m.

[the_rock]

Will check my lab later on and update you.

Andy

[the_rock]

Hey mate,

So sorry, I pasted wrong kernel parameter, that does not work in R82 either, just tried it. I meant this one:

fw ctl set int fw_allow_out_of_state_tcp 1

Andy

[marcyn]

Hello Andy,

I don't like this solution to allow oos ... and I don't see any relation with this issue.
But because this is a lab (at least my environment 🙂 ) .. why not to just check what will be the resuls ... so I checked and it didn't help me.

So question still remains:
[vs_0][fw_1] eth0:i[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0

wtf ?

It looks like I'll need to contact TAC with this one ...

But still, thank you Andy for your afford and time.

--
Best
m.

[the_rock]

I agree 10%, trust me, I would not do that either in production : - )

Let us know what TAC says.

Andy

[emmap]

That is weird, it looks like your debian server is bouncing the reply back to the gateway - which then explains the drops, because the gateway is seeing the same packet twice and dropping it the second time around, this is normal behaviour as far as the gateway is concerned. 

[marcyn · Accepted solution]

Hello guys,

I have good news ... it looks like there is a solution 🙂

But first let me answer to @emmap :
Yes, this is weired ... but it does not look like my debian server is bouncing this reply ... but my SG (where I have PBR rule) - debian doesn't reseive this reply at all ... this reply hits 2nd router, then goes to 1st router (where debian is connected to), and then it goes back to 2nd router (so 1st router bounces this reply).

And now back to the solution:
As I mentioned I've created TAC case describing this issue even with more details then here.
It was couple of days ago ... until today there is no answer ... yet 🙂

But ....
I have another case with PBR that I didn't mentioned here to you ... maybe you will be interested in as well.
Case is about: "how to force Check Point that it will not do NAT for traffic related to PBR".

Let's assume that I have PBR configured as I described here in this post.... so I have enabled ABR and I have rule in fw where any traffic from debian should by matched by PBR and go to 2nd router.
And ... I don't want Check Point to do any NAT for this traffic.
Easy ? I thought so at the beginning.... but it looks like Check Point is not able to do that in one particular case (and unfortunately Fortigate has no issues with that at all 😜 ).
What's this particular case ? ... => Manual NAT rule.
If I have Manual NAT rule - for example like this one:

What should this rule do ?
NAT for any traffic FROM 10.20.200.0/24 (debian has 10.20.200.222) to ExternalZone.
Only eth1 (external) interface has ExternalZone.
eth0 has InternalZone - this is an interface that has 10.99.99.234 ... and via thius interface 2nd router (10.99.99.203) is reachable.
There is no other NAT rules in my lab configuration.
So I thought that in case traffic from eth2 (10.20.200.0/24) will be send to 10.99.99.203 via PBR (so to interface eth0) ... it will not match this NAT rule.
I was wrong ... unfortunately it looks like NAT is calculated before routing decision ... so even that traffic to 8.8.4.4 will go via eth0 .... it will be considered as traffic for ExternalZone ... sadly 😞

You may say ... so what ? let's create no-NAT rule before this one ... <-- sure in most cased it will be good idea ... but not in all of the cases. What about ICMP protocol ? And I don't want to create bunch of no-NAT rules before ....
I hope you get my point here...

Anyway ,... why I'm writing this NAT issue here ... in this post ?
HA HA becuase by accident I've found a solution to issue described in this post... 🙂

Take a look:

If you remember ... reply looked like this:
[vs_0][fw_0] eth0:i[44]: 8.8.4.4 -> 10.99.99.234 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=10003 seq=1
[vs_0][fw_0] eth0:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=51307 seq=1
[vs_0][fw_0] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=51307 seq=1
[vs_0][fw_0] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=51307 seq=1

Right ? ... eth0 as incoming and OUTGOING interface ... no routing to eth2...

One minor change in configuration for PBR rule ... .and now it looks like this:
[vs_0][fw_0] eth0:i[44]: 8.8.4.4 -> 10.99.99.234 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=10003 seq=1
[vs_0][fw_0] eth0:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=51307 seq=1
[vs_0][fw_0] eth2:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=51307 seq=1
[vs_0][fw_0] eth2:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
ICMP: type=0 code=0 echo reply id=51307 seq=1

Perfect ! we have routing to eth2 !

What's this minor change ?
I just added additinal matching criteria to my PBR rule ...

SG> set pbr rule priority 100 match from 10.20.200.0/24
SG> show pbr rule 100
PBR rule 100 from 10.20.200.0/24 fwrule 1 table 1
SG> save config

And by accident ...it solved this issue !
root@debian:~# ping 8.8.4.4
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=117 time=17.3 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=117 time=16.2 ms

Dear Check Point .... you should consider some more work on PBR ... because it is illogical in some aspects 🙂

I hope that my lab work will be helpful to somebody that will face similar problem 🙂

BTW
Any ideas about this NAT issue ? 🙂

--
Best
m.

[the_rock]

Excellent @marcyn , thanks a lot for the update!

Andy