This is another one that is not a question, but a solution, the devising of which found me cursing Google while troubleshooting, and so I wanted to share my findings.
The challenge was that a Cisco 5500 series wireless LAN controller (WLC) needed to communicate with a WLC Anchor located in a guest network, across separate interfaces of a Check Point firewall.
If you are unfamiliar with this design, I'll summarize it by calling it a "pseudo-tunnel" that allows the WLC to communicate with the Anchor, which is the component that allows the WiFi clients to connect. So, you have a DMZ containing guest or corporate WiFi clients, properly cordoned off from the internal subnets, but the device that is responsible for shepherding these clients needs to connect to the inside via a proprietary and seemingly kludgy tunnel or else all hell breaks loose.
Before going any further, I have to point out some restrictions we were working with:
- We did not have access to the WLC or Anchor
- We could not see the WLC or Anchor logs
- We could not physically or remotely reset the Anchor or the controller
- We did not even know the IP addresses on which the Anchor and WLC expected to establish their tunnels (i.e. no-NAT addresses or hide NAT, in both directions.)
- We could not verify the WiFi was working, aside from the usual context clues
- We were not at liberty to cpstop;cpstart or reboot the firewall
Given the above constraints, we found ourselves trying various combinations of NAT and no-NAT to get it working, on top of the problem of the tunnel protocol itself being nonstandard.
The highly uncommon behavior of the WLC-Anchor tunnel turns it into what I dubbed a "poltergeist connection" on the firewall: a stubborn set of connections that survives pretty much all firewall changes, including policy installation, no matter what you try do to it. At one point I even blocked all tunnel traffic for almost 20 minutes, but the connection remained. So, you can drive yourself nuts enabling and disabling NAT, modifying protocol Advanced properties, etc., because the changes may have no effect due to the poltergeist connections.
Ultimately, I crafted a script to surgically exorcise these connections from the connections table, to effectively "reset" the tunnel via the network without affecting other traffic, so that any other changes we were making (such as no-nat or policy changes) would have an immediate effect. The ability to pause or reboot the WLC function at-will would likely have rendered this unnecessary, but we did not have that luxury.
Hat tip to @Kaspars_Zibarts for the inspiration for the exorcism script, from his CheckMates post. This is what our version looks like, taking the IP addresses of the WLC and the Anchor as arguments:
============= killconns.sh =============
#!/bin/sh
fw tab -t connections -u > table
IPA="$1"; IPB="$2"; IPAHEX=`printf '%02x' ${IPA//./ }`; IPBHEX=`printf '%02x' ${IPB//./ }`; grep "$IPAHEX" table | grep "$IPBHEX" | grep "^<0000000" | awk '{print $1" "$2" "$3" "$4" "$5" "$6}'|sed 's/ //g'|sed
's/</fw tab -t connections -x -e /g'|sed 's/>//g'|sed 's/;//g'
======================================
Occasionally, due to a breakdown in whatever voodoo Cisco is using to keep the tunnels alive, the connection will sputter out on its own, wreaking havoc on the WiFi users, and we can quickly reestablish it from the gateway by bleaching the table with that script.
Now, back to getting the WLC-Anchor tunnel working in the first place. Sparing you the details of each step of troubleshooting, this is what ultimately worked:
- udp/16666-16667 service is normal service with accept replies and match for any
- IP_PROTO_97 needs to NOT have accept replies checked
- NAT/no-NAT should be configured to provide both the WLC and the Anchor the expected IP address
#1 and #2 pertain to the Advanced properties of custom Service objects. #3 is esoteric to the environment, and if you're luckier than we were, you'll know in advance which IP addresses the WLC and Anchor expect to serve as the endpoints of the IP Protocol 97 tunnel. If not, then you'll need to use the killconns script while you troubleshoot. It's been a couple months since I worked on this, but IIRC the two custom services needed to be allowed in both directions.
Hopefully this helps someone facing a similar challenge.
-MAB
