Policy installation failed: TCP connectivity failure port 18191

[DDPBharat]

Hi Team,

I have setup a LAB to learn checkpoint so far it is going well, but now i have stuck with one issue where i have setup as Head Office (CP_HO) Firewall and Branch office (CP-Branch) Firewall. The check point management server is behind the (CP_HO) Firewall when i am trying to push policy package from Management server to CP-Branch firewall i am getting error Policy installation failed: TCP connectivity failure port 18191. Do not know what's issue here request you please guide me.

Toplogy and policies screen shot are in the attached.

[the_rock]

What that means is that SIC (secure internal communication) is breaking on port it communicates with management server, 18191. So, what I would do is when you are pushing the policy, run this command on the gateway (in expert mode) -> fw ctl zdebug + drop | grep 18191 and see what you get. Be free to message me privately and I can help you out.

[DDPBharat]

Hi the_rock,

Thanks for the reply below are logs for the same.

[Expert@CP-Branch:0]# fw ctl zdebug + drop | grep 18191
@;3943;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3943;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3943;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3959;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3960;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3960;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3961;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3961;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3965;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3965;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3965;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3965;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3966;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3969;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3970;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3970;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3970;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3975;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3980;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3981;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;3981;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4040;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4041;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4044;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4047;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4048;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4075;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4078;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4080;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4183;[vs_0];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.200.2.1:43396 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4187;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;
@;4192;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.200.2.1:10065 -> 10.200.3.1:18191 dropped by fw_handle_old_conn_recovery Reason: old packet rulebase drop;

[the_rock]

what IP addresses are 10.200.2.1 and .3.1? Are those firewall and mgmt server? If so, then looks like its definitely something in the rulebase blocking it. Have a look at below:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[DDPBharat]

Hi the_rock

The 10.200.1.1 and 10.200.1.2 are HO-Firewall WAN-IP adddress and 10.200.3.1 and 10.200.4.1 are Branch firewall WAN IP address.

[the_rock]

I would definitely check out the articles I provided. Otherwise, message me privately and we can do remote. Im in EST time zone.

[kb1]

Hi do you think you could help me with that as well? i have a similar issue with 18191 failure and it would be great if you can have a remote session with me as well sometime this week est time.

[the_rock]

sure np

[kb1]

How does Thursday 8 pm est sound?

[the_rock]

No worries, just message me privately and we can set it up.

[networkkid23]

Having a similar issue:

@;515697;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 0.0.0.0:68 -> 255.255.255.255:67 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY;
@;515794;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 0.0.0.0:68 -> 255.255.255.255:67 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY;
@;515911;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=17 0.0.0.0:68 -> 255.255.255.255:67 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY;
@;517002;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 0.0.0.0:68 -> 255.255.255.255:67 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY;
@;517119;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=17 0.0.0.0:68 -> 255.255.255.255:67 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY;
@;517216;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=17 0.0.0.0:68 -> 255.255.255.255:67 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY;

[the_rock]

DEFAULT POLICY drops everything. Run fw unloadlocal, make sure routing and rules are good, try again.

Andy

[networkkid23]

So I did the unloadlocal and re-initialized SIC and it is now established. However when trying to install a policy I get the same TCP error. The logs for the dropped traffic is different now:

@;213441;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=6 10.1.123.200:26760 -> 10.1.123.201:18191 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 5;
@;213441;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=6 10.1.123.200:26760 -> 10.1.123.201:18191 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 5;
@;213508;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=6 10.1.123.200:26760 -> 10.1.123.201:18191 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 5;
@;213659;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=6 10.1.123.200:26760 -> 10.1.123.201:18191 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 5;

Rule 5 is the Cleanup rule. However I see accepted traffic in the Logs & Monitor section. 

10.1.123.200 and 10.1.123.201 is the bridge between 2 firewalls. 

[the_rock]

K, so lets think about this logically for a second. Sooo...good job in removing default filter (thats first step) and re-establishing the SIC. NOW...based on the drops you sent, it clearly tells us that SIC port is dropped, as we only care about dst port, source port makes no difference.  Can you send a screenshot of the rule that allows this communication, because clearly, if its dropping on clean up rule, it cannot find any needed rule(s) to pass the traffic on, whether its regular or layered rules.

Andy

[networkkid23]

So I would think rule 3 should allow it. 10.1.123.200 is an interface on CP1 firewall  trying to get to an interface on CP2 firewall 10.1.123.201 and I have internal networks behind NAT'ed. 

The dropped traffic log also attached, not sure how to analyze that. 

[networkkid23]

Also what I find strange is that this was working a coupld days back. It is a home lab environment so I power off the VMs when I am finished. When I resumed today, this issue is occurring.

[the_rock]

Just to make it easy and less "painful", if I were you, I would have mgmt and gateways in the same rule as both src and dst

Meaning src-> mgmt server and both gateways  -> dst -> same as source -> service any -> accept

Thats it

Andy

[networkkid23]

I made the change to the rule but same issue. 

@;359170;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=6 10.1.123.200:26830 -> 10.1.123.201:18191 dropped by fw_send_l og_drop Reason: Rulebase drop - on layer "Network" rule 5;
@;359364;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=6 10.1.123.200:26830 -> 10.1.123.201:18191 dropped by fw_send_l og_drop Reason: Rulebase drop - on layer "Network" rule 5;

[the_rock]

Are you able to ping back and forth?

Andy

[networkkid23]

Yes pings work both ways

[the_rock]

K, here is what I would try. I honestly have no clue why this happens to you, but, to be 100% sure its NOT the policy issue, can you create any any allow policy and see if that works? If does work, then its your existing policy package thats the problem.

Andy

[networkkid23]

Tried the any any rule and still the same issue unfortunately

[the_rock]

Im fairly positive that proves its routing thats the issue. Make sure to run netstat -nr and route commands, as well as say ip r g 8.8.8.8 and also ip r g and then IP of something internal you are trying to access.

Andy