Packet doesn't leave the FW. It ate my packets!

[Matthew_Do]

Need your expertise!

Packet from R80.10 firewall doesn't leave the box?

The FW (its external interface at 10.0.0.111) can ping its default gateway (10.0.0.1) which is behind a NAT going out to the internet. However, the FW can't reach google or ComCast DNS (75.75.75.75 or 8.8.8.8). These IPs are reachable by other host in 10.0.0.0/24 subnet). Traceroute at the FW indicates the packet doesn't leave the box! Did the FW eat the packets? Internal network is ok, I could ssh into the FW.

----

See attached picture for the trouble firewall.....am I missing something?

hoangsa@ubun2svr:~$ ping 8.8.8.8 -c 2
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 time=11.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=121 time=10.4 ms

--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 10.407/11.127/11.848/0.728 ms
hoangsa@ubun2svr:~$ ping 75.75.75.75 -c 2
PING 75.75.75.75 (75.75.75.75) 56(84) bytes of data.
64 bytes from 75.75.75.75: icmp_seq=1 ttl=58 time=10.5 ms
64 bytes from 75.75.75.75: icmp_seq=2 ttl=58 time=9.48 ms
--- 75.75.75.75 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 9.486/10.041/10.597/0.564 ms
hoangsa@ubun2svr:~$

[Matthew_Do]

very strange.... able to access the internet only after all rules are removed.

HQ-FW1> fw unloadlocal

Uninstalling Security Policy from all.all@HQ-FW1
Done.
HQ-FW1> fw stat
HOST POLICY DATE
localhost - - - : >eth0 <eth0 >eth1 <eth1 >eth2
HQ-FW1> ping 8.8.8.8 -c 1
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 time=10.5 ms

:::::::.......

HQ-FW1> traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
1 10.0.0.1 (10.0.0.1) 16.733 ms 16.681 ms 16.705 ms
2 96.120.25.229 (96.120.25.229) 26.779 ms 26.769 ms 26.719 ms
3 162.151.173.129 (162.151.173.129) 26.713 ms 26.661 ms 26.685 ms
4 68.86.184.146 (68.86.184.146) 26.540 ms 26.640 ms 26.618 ms
5 be-121-ar01.area4.il.chicago.comcast.net (69.139.203.169) 27.454 ms 27.412 ms 28.348 ms
6 be-33491-cr02.350ecermak.il.ibone.comcast.net (68.86.91.165) 28.307 ms 13.031 ms 39.374 ms
7 be-10563-pe01.350ecermak.il.ibone.comcast.net (68.86.82.158) 30.555 ms 39.274 ms 30.509 ms
8 as15169.350ecermak.il.ibone.comcast.net (75.149.230.110) 39.178 ms 30.349 ms 75.149.230.26 (75.149.230.26) 39.173 ms
9 * * *
10 108.170.233.86 (108.170.233.86) 39.120 ms 216.239.51.144 (216.239.51.144) 30.267 ms 216.239.42.108 (216.239.42.108) 39.041 ms
11 209.85.250.7 (209.85.250.7) 30.730 ms 72.14.235.245 (72.14.235.245) 39.036 ms 108.170.236.101 (108.170.236.101) 39.009 ms
12 google-public-dns-a.google.com (8.8.8.8) 20.045 ms 20.026 ms 19.953 ms
HQ-FW1>

...

[Matthew_Do]

and this is what happened when the policy is re-applied ......... (FW got drunk?)

HQ-FW1> fw fetch 172.16.0.25

Fetching FW1 Security Policy From: 172.16.0.25

Local Policy is Up-To-Date.
Reinstalling Local Policy.

Installing Security Policy Basic-Rules on all.all@HQ-FW1
Using username "admin".  <<<< I had to restart ssh session.  Rules are re-applied, the prompt is back but not the GUI
This is HQ-FW1. Authorized personnel only.
admin@172.16.0.111's password:
Last login: Sat Sep 22 23:12:16 2018 from 172.16.0.9
HQ-FW1> fw stat
HOST POLICY DATE
localhost Basic-Rules 22Sep2018 23:25:37 : [>eth0] [<eth0] [>eth1] [<eth1] [>eth2]
HQ-FW1>

[_Val_]

Hi Matthew Do‌, there is nothing strange. FW GW is a security device. Policy is important.

Please make sure you apply appropriate policy allowing you require. Also, if this is a newly installed device, check the topology is properly defined, and you have both internal and at least one external interfaces probably defined on FW object.

To see why packets are not forwarded, you ca use "fw ctl zdebug drop | grep <IP in question>" command on cli to understand why packets are dropped.

BTW, I have moved your post into more appropriate area and removed unnecessary tag

[Timothy_Hall]

Traffic originating from the firewall itself is a bit of a special case, and must still be allowed in the Firewall/Network Policy Layer by an explicit rule or the implied rule "Accept outgoing packets originating from gateway" since that traffic still passes through inspection points o and O.  As Val mentioned topology and antispoofing are enforced as well and must be properly defined.  Because traffic originated from the firewall itself does not pass through iI where destination NAT operations occur, it is impossible to NAT the destination address of this type of traffic but the source IP address can most definitely still be NATted.  When you unload policy NAT is disabled too, so I suspect you are source NATting traffic originated from the firewall to something the Comcast modem is not expecting.  Probably not relevant here, but also keep in mind that traffic originated from the firewall or bound for the firewall itself is never accelerated by SecureXL and will always go F2F.

Please see the slides from my CPX Barcelona/Vegas presentation where I talk about how to troubleshoot what I call the "Roach Motel" situation where packets enter the firewall, but mysteriously disappear (get "eaten") with no logging indication of what happened to them:

Best of CheckMates CLI 

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Matthew_Do]

Thank you gentlemen... below is the reason but I don't know how to fix yet. The reason I tried the ping from FW is internal host (172.16.1.22) couldn't ping any thing from the internet although the packet is routed to the FW (172.16.1.111)

The FW ext interface is 10.0.0.111.

[Vladimir]

Please see my post above.

You did not configure your Forewall's topology properly: your External Interface is not defined.

It should look something like this:

In step 3, chose the interface connected to your ISP's router.

In step 7, you can chose "According to topology: ExternalZone"

Settings in Step 8, "Anti-Spoofing" are likely the reason you are seeing the messages shown in your screenshot above.

You still should keep it on "Prevent", but for experiment, you may disable "Perform Anti-Spoofing based on interface topology" load the policy ignoring the warning and see if you are getting replies.

Then revert to normal Anti-Spoofing settings.

[Matthew_Do]

Hi Vladimir, this is helpful but still doesn't resolve my issue. The FW was reconfigured as your pictures suggested & published but still get the same output when a ping was attempted at internal host (172.16.0.7):

The same result for both cases: (enable or disable perform Anti-Spoofing....)

;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:35062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 205.185.216.10:80 -> 10.0.0.111:10062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:35062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:10005 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:35062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:35062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:35062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:35062 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 205.185.216.10:80 -> 10.0.0.111:10062 IPP 6>, dropped by do_inbound, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:10005 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 69.89.207.99:123 -> 10.0.0.111:123 IPP 17>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.248.45.254:80 -> 10.0.0.111:35063 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.248.45.254:80 -> 10.0.0.111:35063 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.248.45.254:80 -> 10.0.0.111:35063 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:10005 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.248.45.254:80 -> 10.0.0.111:35063 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:10005 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.248.45.254:80 -> 10.0.0.111:35063 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;

;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.252.10.254:80 -> 10.0.0.111:10074 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:35006 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:10075 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:35006 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:10075 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:10075 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:10075 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:35006 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:10075 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.249.107.254:80 -> 10.0.0.111:10075 IPP 6>, dropped by handle_spoofed_susp, Reason: Address spoofing;
;[cpu_0];[fw4_0];fw_log_drop_conn: Packet <dir 1, 8.8.8.8:0 -> 10.0.0.111:35006 IPP 1>, dropped by handle_spoofed_susp, Reason: Address spoofing;

[Vladimir · Accepted solution]

Did you install the policy after publishing the changes?

[_Val_]

Show us the topology info for your external interface. It is still incorrect.

[Matthew_Do]

One of the root causes was...

[_Val_]

Antispoofing drops mean your topology is not properly defined. Look into interface topologies to fix the issue.

[Matthew_Do]

Can some expert please explain or share link to the answer?
I'm still unclear about the difference and when it would be appropriate to use one of the following options:
1) CP GWY: NAT > Advanced > Add automatic address translation rules > to hide this gateway behind another gateway
2) CP GWY: NAT > Hide internal networks behind the gateway's external IP

[Vladimir]

1. If you have two gateways, i.e. Internal and External, the setting "CP GWY: NAT > Advanced > Add automatic address translation rules > to hide this gateway behind another gateway" allows you to specify that either:

a). you'll be hiding your Internal Gateway behind IP address of the External interface of your External Gateway

In this case, you have to specify the actual External Gateway, behind who's IP you are hiding and the NAT rule will be installed on that unit:

b). you'll be hiding your Internal Gateway behind IP address of another NAT capable device, such as your ISPs router.

So if you have a single gateway with internal network 10.0.0.0/24 and external network 192.168.1.0/24 connected to the ISP router's internal port and the external port having a single public IP (for example 20.20.20.1/30) assigned to the router, you can specify that in this setting:

What you cannot do is hide your gateway behind it's own external IP, as this will look like dog chasing it's own tail.

2. The "CP GWY: NAT > Hide internal networks behind the gateway's external IP" simply allows you to hide all of your internal networks behind external IP of your single gateway.

I.e. it is the same thing any home router is doing by overloading single public IP with the internal network range on its way out to the Internet.

It is seldom used in business environments that have multiple public IPs or entire network ranges from ISPs, as a more prudent approach is to "Hide" or "Statically NAT" individual hosts or networks behind either external interface of the gateway (for general access to the Internet), or particular IPs (for inbound traffic, such as mail, FTP, etc..).

[Vladimir]

and that your policy does not drop the ICMP replies.

[Matthew_Do]

Thanks. Is your CP FW behind another FW?

These are the cfg on my FW. However, this CP FW is behind another FW (NAT) and SNAT is being used at 2 places (CP and ComCast FW). In your case, probably public IP is assigned to eth1 (ext interface) and there is no NAT after that.

In addition, not only ICMP can't get out but also http, udp etc.

Probably, I need to study Timothy Hall's suggestion above..