Facing weird issue with Route based tunnels and traffic is getting dropped

[Blason_R]

Hello,

This is again I stuck in same issue and not sure how to troubleshoot. I built a VTI tunnel and traffic is getting dropped with below messages.

I turned off

vpn accel

fwaccel off

disabled few parameters with fw ctl set int but in vain.

Any idea how do I handle this?

PPAK 0: Get before set operation succeeded of simple_debug_filter_off
[Expert@LPCPNETCORE-FW:0]# fw ctl zdebug + drop | grep 10.255.255.1
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';
@;3131804750;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=1
10.255.255.1:476 -> 192.168.2.7:0 dropped by fw_send_log_drop Reason:
Rulebase drop - dropped due to 'drop optimization';

[the_rock]

Not sure if it could be related, but maybe check below settings. If its enabled, maybe disable it and install the policy to see if it changes the behavior.

Andy

[Expert@CP-gw:0]# fw ctl get int fwkern_optimize_drops_support
fwkern_optimize_drops_support = 1
[Expert@CP-gw:0]#

[Blason_R]

Nah man - That is already done but no luck.

Drop Optimization is not enabled. In fact you and I had a previous thread on similar topic, eventually it ended or resolved on it own 😞

[the_rock]

One thing I find odd from the drops is that it keeps saying rulebase drop, yet, there is no rule mentioned. 192.168.2.7, thats host on the other end?

[Timothy_Hall]

Hmm sounds to me like the firewall expired or has otherwise lost state for the existing ICMP "connection" between 10.255.255.1 and 192.168.2.7, which was probably initiated by 192.168.2.7,   However 10.255.255.1 does not know this and is still trying to use the dead connection to send replies.  Because it does not match anything in the state table, a rulebase lookup occurs and column-based matching has thrown out all possible rules with an accept action thus leaving only drops, and does not bother to figure out which rule will drop it to save resources as it doesn't matter anyway, since I suspect 10.255.255.1 is not allowed to initiate pings to 192.168.2.7.

fw ctl conntab | grep 10.255.255.1 can be used to see the current idle timer for that ICMP "connection", if Aggressive Aging is enabled the connection may get expired early if the firewall is running short of resources.

[Blason_R]

Well forgot to mention or not sure if that would be useful. 10.255.255.1 is a tunnel interface of peer and I have 10.255.255.2. 

192.168.2.7 is my lan server. And they are initiating a connection from that peer hence 10.255.255.1 which is an egress interface of that device is being used and those are the packets getting originated from the peer.

Thats a good insight and let me try looking 

[the_rock]

Whats the other side? AWS, Azure?

[Blason_R]

Nope a SDWAN router from versa.