Campfire Stories

[AlekseiShelepov]

Hello CheckMates,

At this pleasant almost-weekend day I would like to invite you all to this cozy thread.

All engineers working with devices connected to networks have some cool stories of their heroic saves (or fails) from the past. And then engineers like to tell these stories to their colleagues, who also like to listen good ones.

Why not share your stories connected with Check Point products with the whole community?

The "root cause" for the current thread is Appliance model from CLI and dmidecode with full model list.

Comment by Dameon Welch Abernathy:

Makes me want to start a new thread: Show me your oldest Check Point "fw ver" and uptime from said box

Comment by Kaspars Zibarts:

hahaha IP530... ! do you have IP440 too by any chance? we had on one of those 1800+ days uptime.. indestructible shame I can't find that screenshot anymore

Start a thread "story from back in the day.." I'm sure many will have some fun thing to say

Let's share some funny, strange, inspirational, scare, and maybe even philosophical moments from the old days. Share screenshots or outputs from old devices that still warm up air around them to this day. Your favorite software versions, when you thought that this feels like the future. The most bizzare and monstrous setups that you saw or created yourself (who said GOST cypher?). Something that you were struck by (IPCALC on CLI).

Anything that would squeeze a nostalgic tear out of old-school guys and may enlighten the younger generation.

[Vladimir]

Does this re-purposed IP440 qualifies?

[PhoneBoy]

I think so

[AlekseiShelepov]

Sure. I hope it has something like Core i9 nowadays.

Here is a piece of history that I found, it still works normally. I think this is the oldest Check Point thing that worked with.

Although I cannot really do a lot of things with these devices. Even migration would be a pretty tricky thing to do. And I don't want even think of upgrading them.

[Vladimir]

I like the uptime value on those:)

As to IP440 box, it is now decomissioned, but it was running original Intel Quad Core CPU for a long time and was used as my home ESXi box, until I've splurged on a pair of Dell 720s. Just cannot get read of it- too many sentimental memories.

[AlekseiShelepov]

Welp... To infinity and beyond!

[PhoneBoy]

The first version of IPSO I used was IPSO 3.1, which I believe had the codename Frigate.

And reminds me of a feature that exists in Gaia today that came from an RFE I had (first implemented in IPSO 3.4).

The impetus for it was how slow it was to work with the IPSO WebUI when you had to add many static routes.
(This was also pre-CLISH days as well)

[PhoneBoy]

I want to hear Moti Sagey‌'s story about the firewall he installed on an active volcano (as mentioned in his CPX intro)!

I'll be honest, I've probably forgotten more campfire-worthy stories than I can remember

One I can share is not even that technical, but still somewhat funny.

I don't remember the exact timeframe, but let's say it was year 2000 or 2001, back when I worked in the TAC for Nokia.

I was the on-call engineer for the weekend (this was pre-Ottawa TAC) and got a call from a customer about a firewall issue they were having.

The customer described their issue in detail, and while I understood what they were saying, I was struggling to understand how it related to a Nokia appliance.

I finally asked the customer: what platform is this firewall running on?

The answer: Windows NT.

To which I replied: you realize you called Nokia Support, right?

I did tell him what I thought the solution to his problem might be, of course.

We both had a good laugh.

[Timothy_Hall]

Hmm had to think about this one for awhile, as I've had quite a few of these over the years.  But I think the best one is the incident that first launched me down the path of firewall performance optimization rather painfully, which is why I haven't really talked about this until now.  So without further adieu:

In early 1999 I worked for a Check Point reseller based in Boulder, Colorado. I installed a Check Point firewall for one of the local news stations here in Denver earlier in that year, and it was pretty awe-inspiring getting to see all the great technology they had as well as their local news studio and such.  I got a phone call now and again from their firewall administrator after the project was completed, as he configured the Firewall-1 product which I think was version 4.0 but might have been 4.1.

However on April 20, 1999 I got a call from him and could immediately tell that something was off in his voice as soon as he started speaking, and the conversation went something like this:

Him: We are experiencing some really bad performance on our firewall

Me: Really?  We didn't see anything like that during our testing and afterwards, what changed?

Him: We just got linked to by every news organization in the world due to a big local story

Me: What's the story?

Him: At least 10 students and a teacher just got gunned down at Columbine High School.

Me: (expletive deleted)

The next few hours after that are a bit hazy with phones ringing and TVs blaring incessantly in the background but they involved doubling the firewall's connections table limit from 25,000 to 50,000 and watching it fill up again almost instantly, doubling it again to 100,000 which (barely) managed to shoulder the connection load but now the single CPU was swamped (no CoreXL back then) and memory was running low.  These numbers may seem laughably small now, but were absolutely gigantic in 1999 when dialup access to the Internet was still the norm and DSL/Cable modems didn't exist yet.  There weren't really any comprehensive tuning guidelines available for Check Point's early products (even on Dameon Welch Abernathy's Firewall-1 FAQ site) so I just had to "wing it", and started maintaining notes about firewall performance optimization which eventually resulted in the first edition of my book 16 years later.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Vladimir]

Well, the Checkpoint in this story was just a piece of the puzzle, as it was serving for remote access, which become unavailable.

A matter of time...

This happened on my 40th birthday, which helped it to become a memorable one.

Developers were testing their DR procedures from our hot backup site and asked someone high-up in IT, who should remain nameless, to help them simplify the test.

Instead of changing the date in the application to generate the data that will not be processed as viable real-time trades, being out of scope, dev crew requested temporary change of time in the entire domain and my colleague, God bless his soul, complied, by rolling it back a day.

Thanks to an independent monitoring systems I had in place, my phone was flooded with alarms to the point of being useless. emails and SMS messages were hammering it claiming that the sky has fallen.

Essentially, AD dropped every single server, service and client due to time discrepancy, among them were the RSA SecurID and the Checkpoint components of the infrastructure. So, deprived from remote access capabilities, I had to schlep to work, recall all IT personnel and handle it by hand on premises.

As a result of this snafu which, thankfully, coincided with weekend, the prize of rubber chicken was introduced in the office and awarded to those inadvertently causing mayhem.

I'm including the stock photo for your viewing pleasure: