snmp over ipsec is not working for some clusters

[KM1895]

hi,

I have been troubleshooting a somewhat strange scenario for a customer. 

They have multiple clusters around the world, monitored by snmp.

On 3 clusters, we see that the monitoring server fails to send/receive any snmp traffic to the standby nodes, for the rest, it works just fine.

I can see the traffic going to the active node and gets decrypted, when i check the logs. but i dont see any traffic on the standby node when i do either a tcpdump or fw ctl zdebug drop. I dont see any drops when running the same on the active node as well.

the forward_to kernel parameter has been added, as these gateways run on R80.40, and ssh access, and other traffic, like ldap, is working. But for some strange reason, snmp is not working as it should on these few standby nodes, while the rest of the organization is just fine.

I find it strange that there should be 3 similar bugs on 3 different locations, so im assuming there is some setting or configuration missing on these nodes, but i could use some input in trying to figure out what that could be.

[_Val_]

Some further clarification is needed:

1. Those not working are only reachable via VPN, as the title hits?
2. Those working, also on VPN or not?
3. Did you try tracing the traffic via tje tunnel, e.i. run fw monitor on it, which is not on the standby node, but on other GWs?

[KM1895]

hi,

1. Both yes and no. I can access firewall directly on ssh, if that is what you mean. But snmp and traffic between sites will always go over vpn.

2. Yes, they are all in the same vpn mesh, but for some reason, 3 clusters doesnt work properly with snmp

3. Havent tried fw monitor yet, just tcpdump and fw ctl zdebug drop, which didnt provide much info. Will see if i can turn off securexl and run an fw monitor, as im fairly sure the traffic gets to the active node, thats is at least what the logs in smartconsole tell me.

[_Val_]

For SSH, is it excluded from VPN tunnel? 

[KM1895]

hi,

Depending on where im accessing it from. When i usually access, its over the vpn, yes, and i can access both active and standby without any issues, same with gaia portal.

[_Val_]

This is very unlikely. VPN tunnel is terminated on an active member. SSH, would it be in the tunnel itself, would not work for the standby member. Check "Advanced" settings on VPN community to see if SSH is listed as an excluded service. Also, check if GW IPs are excluded from VPN domain. Either would allow SSH access to standby, regardless of the VPN tunnel.

[KM1895]

yeah, you are correct, of course..just me staring blindly at this problem for too long. The ssh access is directly to the fw nodes, while snmp is via the tunnel.

[_Val_]

Then it is the answer. If you have routable IP addresses on both members, move SNMP out of the tunnel. Make sure you use properly encrypted SNMP v3 though.

[Timothy_Hall]

I'd advise against disabling SecureXL to run a fw monitor -e as that may impact the gateway's performance and possibly even change the behavior of what you are attempting to troubleshoot.  Use fw monitor -F instead which does not require disabling SecureXL, but you need to be careful how your filter is constructed to avoid getting overwhelmed, all this is covered in my Max Capture video series but you can obtain the key elements of it for free here:  https://community.checkpoint.com/t5/Member-Exclusive-Content/Max-Capture-Why-cppcap-is-Now-My-Go-To-...

[KM1895]

hi,

The amount of traffic going through the firewall is quite low, so no issue disabling SecureXL for running this for a limited time.

I did capture some traffic, but i think i might need to run it again, to set a better filter. I will also try to get access to the monitoring server used for this as well, to see if there is anything there that can shed some light on this as well