DHCP relay via 2nd firewall not working

[David_Herselman]

We configured central DHCP relaying, using new services, by following SK104114 and it works perfectly if we are relaying locally  (receiving DHCP broadcasts on a locally attached VLAN and sending unicast queries to the DHCP servers).

We've however deployed a second firewall and the firewall that is local to the DHCP server drops the packet although the policy matches with 'Allow'.

My artwork:

Policy:

Debug information:

fw ctl zdebug -T -e "accept host (10.150.50.1) and host(192.168.141.220) and port(67);" -m fw + vm drop

[-- Stateful VM inbound: Entering (1596318170) --];
@;1931295932; 1Aug2020 23:42:51.041041;[cpu_7];[fw4_0];Before VM: <dir 0, 192.168.141.220:67 -> 10.150.50.1:67 IPP 17> (len=337) (ifn=23) (first seen) (looked up) ;
@;1931295932; 1Aug2020 23:42:51.041044;[cpu_7];[fw4_0];fw_filter_chain: fwconn_chain_conn_exists returned 1 (conn=<dir 0, 192.168.141.220:67 -> 10.150.50.1:67 IPP 17>, is new 0), chain 0xffffc2005afc5bc8;
@;1931295932; 1Aug2020 23:42:51.041046;[cpu_7];[fw4_0];fw_cluster_ttl_anti_spoofing: conn=<dir 0, 192.168.141.220:67 -> 10.150.50.1:67 IPP 17>;
@;1931295932; 1Aug2020 23:42:51.041049;[cpu_7];[fw4_0];fw_conn_post_inspect: Packet accepted (fast path);
@;1931295932; 1Aug2020 23:42:51.041049;[cpu_7];[fw4_0];fw_filter_chain: Final switch, action=ACCEPT;
@;1931295932; 1Aug2020 23:42:51.041050;[cpu_7];[fw4_0];fw_filter_chain: packet accepted;
@;1931295932; 1Aug2020 23:42:51.041052;[cpu_7];[fw4_0];After VM: <dir 0, 192.168.141.220:67 -> 10.150.50.1:67 IPP 17> (len=337) ;
@;1931295932; 1Aug2020 23:42:51.041052;[cpu_7];[fw4_0];VM Final action=ACCEPT;
@;1931295932; 1Aug2020 23:42:51.041052;[cpu_7];[fw4_0]; ----- Stateful VM inbound Completed -----
;
@;1931295932; 1Aug2020 23:42:51.041054;[cpu_7];[fw4_0];
[-- Stateful POST VM inbound: Entering (1596318170) --];
@;1931295932; 1Aug2020 23:42:51.041055;[cpu_7];[fw4_0];fw_post_vm_chain_handler: (first_seen 32, new_conn 0, is_my_ip 0, is_first_packet 0);
@;1931295932; 1Aug2020 23:42:51.041057;[cpu_7];[fw4_0];Before POST VM: <dir 0, 192.168.141.220:67 -> 10.150.50.1:67 IPP 17> (len=337) (ifn=23) (first seen) (looked up) ;
@;1931295932; 1Aug2020 23:42:51.041058;[cpu_7];[fw4_0];fw_post_vm_chain_handler: executing handler function dhcp_reply_code;
@;1931295932; 1Aug2020 23:42:51.041064;[cpu_7];[fw4_0];fw_post_vm_chain_handler: handler function returned action DROP;
@;1931295932; 1Aug2020 23:42:51.041066;[cpu_7];[fw4_0];fw_log_drop_ex: Packet proto=17 192.168.141.220:67 -> 10.150.50.1:67 dropped by fw_post_vm_chain_handler Reason: Handler 'dhcp_reply_code' drop;
@;1931295932; 1Aug2020 23:42:51.041068;[cpu_7];[fw4_0];After POST VM: <dir 0, 192.168.141.220:67 -> 10.150.50.1:67 IPP 17> (len=337) ;
@;1931295932; 1Aug2020 23:42:51.041068;[cpu_7];[fw4_0];POST VM Final action=DROP;
@;1931295932; 1Aug2020 23:42:51.041069;[cpu_7];[fw4_0]; ----- Stateful POST VM inbound Completed -----
;

fw monitor -TP -e "accept dport=67 and host(10.150.50.1) and host(192.168.141.220);" -p all
in chain (23):
0: -7fffffff (0000000000000000) (00000000) SecureXL inbound (sxl_in)
1: -7ffffffe (0000000000000000) (00000000) SecureXL inbound CT (sxl_ct)
2: -7ffffff0 (ffffffff89bb93f0) (00000001) tcpt inbound (tcp_tun)
3: -7f800000 (ffffffff8a3db810) (ffffffff) IP Options Strip (in) (ipopt_strip)
4: -7d000000 (ffffffff89bda870) (00000003) vpn multik forward in
5: - 2000000 (ffffffff89be6120) (00000003) vpn decrypt (vpn)
6: - 1fffffa (ffffffff89bbb0d0) (00000001) l2tp inbound (l2tp)
7: - 1fffff8 (ffffffff8a3d95d0) (00000001) Stateless verifications (in) (asm)
8: - 1fffff7 (ffffffff8a3d90f0) (00000001) fw multik misc proto forwarding
9: - 1fffff2 (ffffffff89a390e0) (00000003) vpn tagging inbound (tagging)
10: - 1fffff0 (ffffffff89bd6020) (00000003) vpn decrypt verify (vpn_ver)
11: - 1ff (ffffffff89f9b510) (00000001) NAC Packet Inbound (nac_tag)
12: 0 (ffffffff8a4c88d0) (00000001) fw VM inbound (fw)
13: 1 (ffffffff89bdb750) (00000003) vpn policy inbound (vpn_pol)
14: 2 (ffffffff8a3dbc60) (00000001) fw SCV inbound (scv)
15: 3 (ffffffff89bd3990) (00000003) vpn before offload (vpn_in)
16: 5 (ffffffff8a032830) (00000003) fw offload inbound (offload_in)
17: 10 (ffffffff8a4ba610) (00000001) fw post VM inbound (post_vm)
18: 100000 (ffffffff8a470f20) (00000001) fw accounting inbound (acct)
19: 7f730000 (ffffffff896bb4f0) (00000001) passive streaming (in) (pass_str)
20: 7f750000 (ffffffff8a20fdc0) (00000001) TCP streaming (in) (cpas)
21: 7f800000 (ffffffff8a3db7c0) (ffffffff) IP Options Restore (in) (ipopt_res)
22: 7fb00000 (ffffffff898f5bd0) (00000001) Cluster Late Correction (ha_for)
out chain (20):
0: -7f800000 (ffffffff8a3db810) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: -78000000 (ffffffff89bda850) (00000003) vpn multik forward out
2: - 1ffffff (ffffffff89bce5c0) (00000003) vpn nat outbound (vpn_nat)
3: - 1fffff0 (ffffffff8a206e50) (00000001) TCP streaming (out) (cpas)
4: - 1ffff50 (ffffffff896bb4f0) (00000001) passive streaming (out) (pass_str)
5: - 1ff0000 (ffffffff89a390e0) (00000003) vpn tagging outbound (tagging)
6: - 1f00000 (ffffffff8a3d95d0) (00000001) Stateless verifications (out) (asm)
7: - 1ff (ffffffff89f40400) (00000001) NAC Packet Outbound (nac_tag)
8: 0 (ffffffff8a4c88d0) (00000001) fw VM outbound (fw)
9: 10 (ffffffff8a4ba610) (00000001) fw post VM outbound (post_vm)
10: 2000000 (ffffffff89bd1330) (00000003) vpn policy outbound (vpn_pol)
11: 1ffffff0 (ffffffff89bb98b0) (00000001) l2tp outbound (l2tp)
12: 20000000 (ffffffff89be15a0) (00000003) vpn encrypt (vpn)
13: 60000000 (ffffffff89bb9170) (00000001) tcpt outbound (tcp_tun)
14: 7f000000 (ffffffff8a470f20) (00000001) fw accounting outbound (acct)
15: 7f700000 (ffffffff8a2077f0) (00000001) TCP streaming post VM (cpas)
16: 7f800000 (ffffffff8a3db7c0) (ffffffff) IP Options Restore (out) (ipopt_res)
17: 7f850000 (ffffffff898f4e10) (00000001) Cluster Local Correction (ccl_out)
18: 7f900000 (0000000000000000) (00000000) SecureXL outbound (sxl_out)
19: 7fa00000 (0000000000000000) (00000000) SecureXL deliver (sxl_deliver)
monitor: monitoring (control-C to stop)
[vs_0][fw_4] 2Aug2020 0:16:09.083304 bond2.4:id2 (tcpt inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083309 bond2.4:id3 (IP Options Strip (in))[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083310 bond2.4:id4 (vpn multik forward in)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083314 bond2.4:id5 (vpn decrypt)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083316 bond2.4:iD6 (l2tp inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083317 bond2.4:iD7 (Stateless verifications (in))[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083319 bond2.4:iD8 (fw multik misc proto forwarding)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083321 bond2.4:iD9 (vpn tagging inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083328 bond2.4:iD10 (vpn decrypt verify)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083329 bond2.4:iD11 (NAC Packet Inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083330 bond2.4:iD12 (fw VM inbound )[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083663 bond2.4:ID13 (vpn policy inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083666 bond2.4:ID14 (fw SCV inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083668 bond2.4:ID15 (vpn before offload)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083669 bond2.4:ID16 (fw offload inbound)[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67
[vs_0][fw_4] 2Aug2020 0:16:09.083675 bond2.4:ID17 (fw post VM inbound )[337]: 192.168.141.220 -> 10.150.50.1 (UDP) len=337 id=26608
UDP: 67 -> 67

If I edit the default objects 'dhcp-request' and 'dhcp-reply' to remove the 'Match' strings, as shown here, it breaks DHCP relaying altogether:

I presume this is a bug?

Regards

David Herselman

[HeikoAnkenbrand]

Hi @David_Herselman;

You have unicast dhcp traffic from the first gateway (dhcp helper) through the second gateway. You must create a dhcp rule in both directions from dhcp helper to the dhcp server and in the reverse direction.

In your case:

Source                            Destination                       Service               Instal on

DHCP Server                 DHCP Helper IP               dhcp-replay      jb1-cluster
                                         

[David_Herselman]

Which is covered by the above rules 'from internal to DHCP servers' and 'from dhcp servers to internal'. The packet is accepted according to the policy but it appears the 2nd gateway receiving the DHCP unicast response back from the DHCP servers doesn't find a DHCP relay request it initiated (as 1st security gateway generated it) and subsequently invalidates the packet although the policy accepts it. The dhcp request however was relayed through the 2nd firewall so it should accept the reply back so that the 1st security gateway can receive the offer for the relayed request.
fw_filter_chain: Final switch, action=ACCEPT;
POST VM Final action=DROP;

I tried adding a policy rule, installed exclusively on 'jb1-cluster', to allow udp:67 without protocol analysis from the dhcp servers towards the 1st security gateway but this then breaks dhcp relay on both security devices as one should only use the new service objects. Using 'dhcp-replay' is an older protocol definition which immediately breaks dhcp-request and dhcp-reply objects...

[HeikoAnkenbrand · Accepted solution]

Use the same rule and create a service.

Source              Destination                      Service                             Instal on
DHCP Server  DHCP Helper IP               new_dhcp_replay           jb1-cluster

Create the new udp service for DHCP without protocol settings. Therefore, no protocol analysis should be used and the connection should not be blocked via "POST VM Final action=DROP".

[David_Herselman]

Hi Heiko,

I did try that but the first firewall then simply ignores the broadcast and doesn't send anything on to the 2nd firewall which has the DHCP servers.

I understand ClusterXL DHCP relay to require policies with the new service objects that do protocol analysis...

Regards

David Herselman