topic Re: Policy Based Routing issue in Firewall and Security Management

Policy Based Routing issue

marcyn — Fri, 19 Sep 2025 09:38:06 GMT

Hello CheckMates,

I'm struggling with an issue with PBR ... and I hope you will be able to put some light on this one.

Customer wants that some traffic should go different path then via default route ... so it's obvious that this is case for PBR.
But it doesn't work as expected.

I was able to recreate similar environment in my lab and it looks like this:
1) I have a Access Control rule:

So as you can see any traffic FROM debian (10.20.200.222) will be matched.
2) PBR was configured to be Application Aware (it's a shame that this is hidden by default) - so I can create a rule in PBR that will tell gateway that it should match rule 1:

3) Table 1: router forward traffic to different router (in my case 10.99.99.203)

So ... as you can see ... nothing odd, just a normal, simple PBR config.

This 10.99.99.203 router doesn't have network 10.20.200.0/24, but knows who has this network - based on static-route entry:

10.99.99.234 = my gateway, where I created PBR, etc.

Ok, now you know how it is configured ... but I will explain this below (just to be sure, you know everything 😛 ) :
1) traffic (for example ping to 8.8.4.4) from 10.20.200.222 arrives on incoming interface on my router (10.99.99.234)
2) Access Control policy knows that it should be accepted and not send to external interface of this gateway, but to interface where 10.99.99.203 is connected
3) then this traffic arrives on router 10.99.99.203 and is accepted and is send to the internet
4) return traffic from 8.8.4.4 arrives on 10.99.99.203 and because this router doesn't have 10.20.200.0/24 network it will not forward this reply directly to "debian" ... but it will forward this traffic to 10.99.99.234 (via static-route)
5) so ... return traffic from 8.8.4.4 to 10.20.200.222 arrives on my gateway on interface where 10.99.99.203 is connected, and then it should go to "debian"

Simple...

But it doesn't work like that.

Take a look how it works (steps 1-5 from the above description):

ad. 1)
root@debian:~# ping 8.8.4.4
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.

ad. 2)
SG (10.99.99.234):
[vs_0][fw_1] eth2:i[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
[vs_0][fw_1] eth2:I[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
[vs_0][fw_1] eth0:o[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
[vs_0][fw_1] eth0:O[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53134
(traffic leaves 10.99.99.234 and goes to 10.99.99.203)

ad. 3)
router (10.99.99.203) - also Check Point:
[vs_0][fw_2] eth0:i[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53362
[vs_0][fw_2] eth0:I[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53362
[vs_0][fw_2] eth1:o[44]: 10.20.200.222 -> 8.8.4.4 (ICMP) len=84 id=53362
[vs_0][fw_2] eth1:O[44]: 203.0.113.203 -> 8.8.4.4 (ICMP) len=84 id=53362
(as you can see there is a NAT)

ad. 4)
[vs_0][fw_2] eth1:i[44]: 8.8.4.4 -> 203.0.113.203 (ICMP) len=84 id=0
[vs_0][fw_2] eth1:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_2] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_2] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
(traffic leaves 10.99.99.203 and goes to 10.99.99.234)

ad. 5)
And .. here is my issue:
[vs_0][fw_1] eth0:i[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
(traffic enters 10.99.99.234 on eth0 interface ... and stays here ... it is not routed to eth2 interface where this debian host exists)

fw ctl zdebug + drop:
@;137323;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1 8.8.4.4:0 -> 10.20.200.222:19315 dropped by fw_first_packet_state_checks Reason: ICMP reply does not match a previous request;

Do you have any idea what is wrong here ?
In case router 10.99.99.203 would have this network 10.20.200.0/24 there will be no issue at all, because return traffic would go directly from this router to "debian" ... but in this case this return traffic needs to go back to 10.99.99.234.

And By The Way ... another issue:
(this rule was disabled in above example ... that's why I had no NAT)
In case Customer would have Manual NAT rule ... not Automatic one ... PBR is doing NAT... and I have absolutely no idea why ?
For Automatic NAT - it will not do NAT for this PBR.
Take a look how my Manual NAT rule looks like:

So ... normal SNAT rule - any traffic from 10.20.200.0/24 to ExternalZone should be hidden behind Gateway's external IP.
But ... 10.99.99.203 router (this one in PBR rule) ... is behind interface that has InternalZone 🙂
Why this rule matches this traffic ? ... it looks like NAT is matched before routing ... and because of that Gateway decides that 8.8.4.4 is reachable via eth1 (external) interface ... not eth0 (here is this 10.99.99.203).

Anybody ? 🙂

--
Best
m.

Re: Policy Based Routing issue

the_rock — Fri, 19 Sep 2025 12:12:53 GMT

Hey @marcyn

Grext explanation btw! Here is one question that pops up in my head...what does route to say 8.8.8.8 look like from debian host?

Andy

Re: Policy Based Routing issue

marcyn — Fri, 19 Sep 2025 12:16:20 GMT

Hi Andy,

debian host knows nothing about this "other router" ... it's gateway is of course this one 10.99.99.234.
So any traffic from debian (10.20.200.222) directed to the Internet will go directly to Gateway with IP 10.99.99.234.

--
Best
m.

Re: Policy Based Routing issue

the_rock — Fri, 19 Sep 2025 12:16:54 GMT

K, fair enough, so what does ip r g 8.8.8.8 show on the fw?

Re: Policy Based Routing issue

marcyn — Fri, 19 Sep 2025 12:20:21 GMT

Of course default route 😀

Re: Policy Based Routing issue

the_rock — Fri, 19 Sep 2025 12:22:33 GMT

Just making sure lol. Anyway, I know that zdebug drop you posted may refer to nat or possibly assymetric routing. Let me see if I can try this in the lab.

Andy

Re: Policy Based Routing issue

emmap — Fri, 19 Sep 2025 22:57:12 GMT

The PBR side of things seems to be working, but you have a stateful inspection drop for ICMP. Your final fw monitor though shows your o and O so it seems like it is actually going somewhere - can you validate with tcpdumps on eth0 and eth2 on that device?

Re: Policy Based Routing issue

marcyn — Sat, 20 Sep 2025 09:16:07 GMT

Yes,
That's right ... because I see o and O it is obvious that this traffic was not blocked by FW.
However there should be routing in place ... this traffic should be forwarded to eth2.

Anyway ... you asked about tcpdump - no problem, here you have it:

[Expert@SG:0]# tcpdump -nnei eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:57:02.807595 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 51904, seq 1, length 64
10:57:02.823689 f2:f5:43:c6:50:8b > bc:24:11:fb:ea:30, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 1, length 64
10:57:02.823780 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 1, length 64
10:57:03.819566 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 51904, seq 2, length 64
10:57:03.834611 f2:f5:43:c6:50:8b > bc:24:11:fb:ea:30, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 2, length 64
10:57:03.834679 bc:24:11:fb:ea:30 > f2:f5:43:c6:50:8b, ethertype IPv4 (0x0800), length 98: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 51904, seq 2, length 64

As you can see ther is a pair request+reply ... and another reply.
First reply is FROM f2:f5:43:c6:50:8b = router behind PBR (10.99.99.203)
And it is addressed TO bc:24:11:fb:ea:30 = debian
And it's exactly what it should be (in my opinion) !

[Expert@CP-GW1:0]# ip a l eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether f2:f5:43:c6:50:8b brd ff:ff:ff:ff:ff:ff
inet 10.99.99.203/24 brd 10.99.99.255 scope global eth0

root@debian:~# ip a l ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether bc:24:11:7e:c1:0c brd ff:ff:ff:ff:ff:ff
altname enp0s18
inet 10.20.200.222/24 brd 10.20.200.255 scope global ens18

But then ... this first reply instead of beeing routed to eth2 where this debian exists ... it is replied but this time:
FROM debian TO router 10.99.99.203 ... why ? ... it is as if this PBR rule interfere with this reply as if it was request...
It looks like this reply bounced back to the router behind PBR.

In case you wonder - arp table in my 10.99.99.234 = "source gateway" (where PBR is declared):
[Expert@SG:0]# arp -an
? (10.99.99.203) at f2:f5:43:c6:50:8b [ether] on eth0
? (10.20.200.222) at bc:24:11:7e:c1:0c [ether] on eth2

Based on arp table ... it's obvious that this gateway exactly knows where this bc:24:11:7e:c1:0c is located... => eth2,
So why it bounced back this reply to f2:f5:43:c6:50:8b (from bc:24:11:7e:c1:0c).

I don't get it ...

Ah ... if you wanted - no problem - tcpdump from eth2:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
11:11:28.963040 bc:24:11:7e:c1:0c > bc:24:11:22:48:4a, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 10360, seq 1, length 64
11:11:29.963344 bc:24:11:7e:c1:0c > bc:24:11:22:48:4a, ethertype IPv4 (0x0800), length 98: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 10360, seq 2, length 64

Only requests here ...

bc:24:11:22:48:4a = obviously gateway's IP on eth2 (10.20.200.254) = debian's default gateway

--
Best
m.

Re: Policy Based Routing issue

the_rock — Sat, 20 Sep 2025 10:25:33 GMT

You got me so curious now, I want to try this in the lab. But in the meantime, just a thought...what if you added a route to 10.20.200.0/24 on 10.99.99.203?

Andy

Re: Policy Based Routing issue

marcyn — Sat, 20 Sep 2025 16:53:48 GMT

Andy, if there would be no route to 10.20.200.0/24 on 10.99.99.203, it would not work at all ... as this router doesn't have this network ... and I mentioned that in my opening message 😀

So of course that 10.99.99.203 has a route to 10.20.200.0/24 with 10.99.99.234 as a gateway to this route.

BR, m.

Re: Policy Based Routing issue

the_rock — Sat, 20 Sep 2025 16:59:27 GMT

Let me try this in the lab tomorrow.

Re: Policy Based Routing issue

the_rock — Mon, 22 Sep 2025 00:44:58 GMT

@marcyn I replicated this and its exact same issue as you. I tested on R82 jumbo 41 (latest) one. Now its good challenge to fix it : - )

Andy

Re: Policy Based Routing issue

the_rock — Mon, 22 Sep 2025 01:31:54 GMT

@marcyn

Here is how I fixed it

Enabled this kernel parameter -> fw ctl set int fw_allow_asymmetric_routing 1

Andy

Re: Policy Based Routing issue

marcyn — Mon, 22 Sep 2025 08:21:42 GMT

Hi Andy,

Thanks for your work on this one.
Because it's the 3rd environment where the same issue occurs ... it's not a coincidence 🙂

I'm glad that you found a fix ... but ...
1) why asymmetric routing ? ... request: incomming = eth2, outgouing = eth0, and as we saw reply goes from eth0 and should go to eth2 ... I don't see here asymmetric routing 🙂

SG (10.99.99.234):
10:04:19.480446 In bc:24:11:7e:c1:0c ethertype IPv4 (0x0800), length 100: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 58102, seq 1, length 64
10:04:19.480730 Out bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 58102, seq 1, length 64
10:04:19.497326 In f2:f5:43:c6:50:8b ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64
10:04:19.497416 Out bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64

CP-GW1 (10.99.99.203):
10:04:19.530303 In bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 10.20.200.222 > 8.8.4.4: ICMP echo request, id 58102, seq 1, length 64
10:04:19.530597 Out 62:93:df:1c:11:3b ethertype IPv4 (0x0800), length 100: 203.0.113.203 > 8.8.4.4: ICMP echo request, id 26668, seq 1, length 64
10:04:19.546660 In 26:cb:19:cd:cc:57 ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 203.0.113.203: ICMP echo reply, id 26668, seq 1, length 64
10:04:19.546713 Out f2:f5:43:c6:50:8b ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64
10:04:19.546957 In bc:24:11:fb:ea:30 ethertype IPv4 (0x0800), length 100: 8.8.4.4 > 10.20.200.222: ICMP echo reply, id 58102, seq 1, length 64 <-- ???

So based on the above:
1) seq1 with id = 58102 was received on incomming interface of my 1st gateway (10.99.99.234)
2) this packed was then forwarded by PBR to 2nd machine (10.99.99.203) /we can clearly see the sam id on incomming interface of 10.99.99.203/
3) then 10.99.99.203 sends this packet to the internet (the is different id = 26668) and it receives reply to id = 26668
4) this reply is then send from 10.99.99.203 to 10.99.99.234 with id = 58102, and it is received by 10.99.99.234 on eth0 interface with valid and expected id = 58102

For me there is no asymmetric routing here...

5) then 10.99.99.203 for some reason receives the same reply with id = 58102 ... but as an incomming messega on eth0 interface from 10.99.99.234:
[Expert@CP-GW1:0]# arp -na | grep bc:24:11:fb:ea:30
? (10.99.99.234) at bc:24:11:fb:ea:30 [ether] on eth0

And this a mistery for me...

Anyway ... I decided to test your fix (I don't get it why it could fix this ... but ok, why not to test).
And unfortunately on R81.20 it's not working:
[Expert@SG:0]# fw ctl set int fw_allow_asymmetric_routing 1
Set operation failed: failed to get parameter fw_allow_asymmetric_routing
set: Operation failed
I checked and there is no such parameter indeed.

But ... Customer has R82 ... so ... I can check this out on R82 as well 😉
But2 ... I would also like to understand this, why asymmetric rounting ... 🙂

--
Best
m.

Re: Policy Based Routing issue

marcyn — Mon, 22 Sep 2025 10:09:16 GMT

Unfortunately on R82 JHF41 this kernel parameter is not supported in my machine as well...

[Expert@SG:0]# cpinfo -y FW1 | grep Take

This is Check Point CPinfo Build 914000250 for GAIA
HOTFIX_R82_JUMBO_HF_MAIN Take: 41
[Expert@SG:0]# fw ctl get int fw_allow_asymmetric_routing
Get operation failed: failed to get parameter fw_allow_asymmetric_routing
get: Operation failed
Killed
[Expert@SG:0]# fw ctl set int fw_allow_asymmetric_routing 1
Set operation failed: failed to get parameter fw_allow_asymmetric_routing
set: Operation failed
Killed
[Expert@SG:0]#

So ever that I don't get it why this fixed it for you Andy ... it will not work for me 🙂

--
Best
m.

Re: Policy Based Routing issue

the_rock — Mon, 22 Sep 2025 11:21:59 GMT

Will check my lab later on and update you.

Andy

Re: Policy Based Routing issue

the_rock — Mon, 22 Sep 2025 12:06:17 GMT

Hey mate,

So sorry, I pasted wrong kernel parameter, that does not work in R82 either, just tried it. I meant this one:

fw ctl set int fw_allow_out_of_state_tcp 1

Andy

Re: Policy Based Routing issue

marcyn — Mon, 22 Sep 2025 12:57:41 GMT

Hello Andy,

I don't like this solution to allow oos ... and I don't see any relation with this issue.
But because this is a lab (at least my environment 🙂 ) .. why not to just check what will be the resuls ... so I checked and it didn't help me.

So question still remains:
[vs_0][fw_1] eth0:i[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:I[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:o[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0
[vs_0][fw_1] eth0:O[44]: 8.8.4.4 -> 10.20.200.222 (ICMP) len=84 id=0

wtf ?

It looks like I'll need to contact TAC with this one ...

But still, thank you Andy for your afford and time.

--
Best
m.

Re: Policy Based Routing issue

the_rock — Mon, 22 Sep 2025 13:00:12 GMT

I agree 10%, trust me, I would not do that either in production : - )

Let us know what TAC says.

Andy

Re: Policy Based Routing issue

emmap — Tue, 23 Sep 2025 01:47:17 GMT

That is weird, it looks like your debian server is bouncing the reply back to the gateway - which then explains the drops, because the gateway is seeing the same packet twice and dropping it the second time around, this is normal behaviour as far as the gateway is concerned.