- Products
- Learn
- Local User Groups
- Partners
- More
Welcome to Maestro Masters!
Talk to Masters, Engage with Masters, Be a Maestro Master!
Join our TechTalk: Malware 2021 to Present Day
Building a Preventative Cyber Program
Be a CloudMate!
Check out our cloud security exclusive space!
Check Point's Cyber Park is Now Open
Let the Games Begin!
As YOU DESERVE THE BEST SECURITY
Upgrade to our latest GA Jumbo
CheckFlix!
All Videos In One Space
Hi (check)mates,
We all know that "the firewall" is one of the first things people blame when there is a traffic issue. A security gateway (a "firewall") do a lot of "intelligent stuff" more than just routing traffic (and -in fact- many network devices today do also "a lot of things") so I understand there is a good reason for thinking about the firewall but, at the same time, there is a big number of times where it's not anything related to it, or when it's not directly related.
I'm looking to build a brief list of typical or somewhat frequent issues we face, where "the firewall" is reported as the root of the issue, but finally it isn't.
It's a quite generic topic, and in terms of troubleshooting it's probably even more generic. Probably there are several simple tools that one should use first, like: traffic logs, fw monitor, tcpdump/cppcap, etcetera. But what I would like to point is not the troubleshooting, but the issues themselves. Of course, assuming the firewall side is properly configured (which would be a "firewall issue" but due to a bad configuration).
To narrow down the circle, I'm specially focusing on networking issues, but every idea is welcomed.
Do you think it would be useful to elaborate such list? 🙂 What issues do you usually find?
Something to start
(I'll update this list with new suggested issues):
Lastly, a little humor. 😊
One of battle stories I tell on community Live sessions is about just that. When migrating MDS to new IP addresses, we stumbled on one of CMAs failing to install policies on remote FWs. Immediately I asked about third party, and the customer said firmly, no, we do not have anything else. After 30 minutes of arguing and some traffic traces, I have proven to them they had something else blocking traffic. Two hours later, they have identified it was a Juniper FW they forgot about eons ago...
I never expected it, but I've heard that this is not an uncommon distinction between network and telco people, that telco techs are used to thinking in circuits/loops but network guys are just into the packet flow and forget to make sure it can come back? I guess you get used to it before too long either way, but I definitely saw it happen with a lot of new people.
Does other Vendor firewall interpretation of RFC's count?
(Albeit I sometimes wonder about CP as well - can't remember any examples right now but all VPN related)
i.e. Sonicwall / Check Point ikev1 IPSEC VPN using PKI
SonicWall firewall does not contain the full certificate chain so you have to install subordinate CA's into the Trusted section of CP. (could be the cert issuer not doing something correctly)
SonicWall Default expects IKE ID to be Distinguished Name (DN) CP sends Main IP Address.
SonicWall only accepts a Cert from CP if the Main IP is the first Alternate name added to the cert when generating the key. If this is not the case, one way VPN initation is still possible but fails if CP initiates the connection.
OMG, I'm sure that was hard to diagnose.
I've experienced a similar issue with another vendor several years ago.
I'll try to not open too much the can of worms of "issues with others firewalls" in the original list 😊
In one of the places I worked, we had a major update for the AV on the workstations.
This happened a week apart from upgrading the FWs appliances.
So, since upgrading the FWs (or so we thought), surfing was so sloooooow.
A co-worker of mine chased this for half a year, until we somehow figured out the cause was the web reputation engine in the AV.
The engine couldn't get the signatures updated, so every URl took forever, until the engine gave up.
In a somewhat similar problem of not transferring files, the customer even escalated the ticket. It had been detected by Checkpoint, that the problem was on the ISP side. They did not believe it, they had already asked them if they had made changes, and they had sworn they had not.
Then by showing them with tcpdump that the problem was on the ISP side, with a reset of the connection, they admitted that they had implemented an improvement in the IPS software, they turned it off for our client and the problem went away.
More firewall fun:
Put a smile on my face!
Without recounting most of waht was written about above, here is the one from few years ago:
Client states that traffic going through check Point cluster to one of the multihomed AS/400s was being arbitrarely dropped.
As a proof, they have shared the graphs from the Linux MTR (my traceroute) tool.
On a surface of things, it did look like CP was the culprit.
Ended-up being completely unrelated issue, but to prove that it was not a firewall, I had to create a moc environment and have discovered that they were using MTR default TTL 1 per hop, which was decrementing by 1 on each hop. Check Point's processing traffic on iIoO was decrementing the TTL to 0 resulting in a false positives for the tool only, but not for the actual traffic.
Nice! I've seen something like this before as well, on a poorly written application.
Good story!
Sometimes, it's quite difficult to reproduce an issue, test a device or create an identical lab without adding new conditions.
This reminds me several examples where the customer where trying to test the throughput capacity of the gateways (not only the firewalls), and how this is a very difficult thing specially when talking about a layer-7 security device.
I'll add something around this to the original list 🙂 Thanks for sharing!
In the memes section, you've missed the best one - I have it pinned to the board in my office:
A framed, full color copy of this Dilbert strip has been hanging in my ATC training room for years and is always good for a few laughs.
Well, most times it seems to be something in-between that botches the connection but I have a good story....
Set up a demo computer in a conference room so a company could come in and do a 'dog and pony' show (yah this was 'back in the day'). Computer was working perfectly. Hardwired and ready to go (pre-wireless networks). Don't recall the version of Checkpoint at the time, probably whatever came just after 2.1c. So vendor shows up with a team of like 4 people (why I have no idea). They are really trying to give a hard sell. They try to pull up their website and it doesn't work. 404 Not Found. I get called back into conference room. "It doesn't work!" Hmm, quick perusal of websites says it's fine, just doesn't work with their new fangled website. I tell them I have no idea. Well that got them in a huff and they call me all manner of names, question my competence etc. I just said, "I cannot help you, there is no problem with the machine and nothing is blocking the connection." Fast forward about 40 minutes later, one of their people and my manager decide to try it on my bosses PC....doesn't work there either. So the vendor gets on the line with her support...turns out, they were trying to connect to a URL that their support team had dismantled. Oops. Yah, it wasn't the firewall!
Like others have said, the biggest one we see is that the remote side isn't even listening on the port they claim is blocked.
@Robert_OBrien nah, it is another firewall lol
IPS side effects.
Ten years ago a consultant asked me to please configure all Lync (later Skype) relevant port objects to be without protocol setting. Said and done, disregarding the fact that we hadn´t even enabled IPS/SmartDefense back then.
Recently a new external SBC and an internal analogue gateway (with static NAT IP) was installed and they wanted a fixed SIP tcp and a few SIP udp ports. This time I used the sip-tcp port object with protocol "SIP_TCP_PROTO" because the IPS activation project was coming along with already being in monitoring mode. IPS is important and needs to used as much as possible, shall it not?
The calls didn´t go through. Signaling happened but no sound.
During a three-hour session, walking through different configurations within the VoIP devices, the SBC admins finally noticed that the firewall had replaced the IP address in the SIP message.
The outgoing message from the external SBC to our analogue gateway showed the public IP in the CALL-ID.
The internal gateway behind the firewall received the message with the CALL-ID containing the private IP of the analogue gateway.
Using a different port object without a protocol setting solved the issue.
( I never actually followed up on this with Checkpoint. Let me know if you think I should )
About CheckMates
Learn Check Point
Advanced Learning
YOU DESERVE THE BEST SECURITY