My Top 3 Check Point CLI commands

[Moti]

Just had a fun geeky conversation with Dameon Welch Abernathy (AKA Phoneboy) Jony Fischbein , Jeff Schwartz and Michael Poublon (over 100 accumulated years of experience in Check Point products) , on what are our favorite & most useful commands in a Check Point environment.

Below are my 3 , plz add yours in the comments (we will do a poll for the top 5 after getting your feedback ... ).

1) fw ctl zdebug drop 

used to quickly see all dropped connections and more importantly the reason (e.g. anti-spoofing, IPS , FW rule , ....)

2) cpstat fw

quickly see stats of number of connections (accepted,denied,logged) with a breakdown

if the FW was under a high load i would usually run " watch --interval=1 'cpstat fw' " (would see a real-time to see the interface that is causing this)

3) fw tab -s -t connections 

allowed me to quickly see how much load is (and was i.e "peak" ) on the FW 

that's it (i have more , but i want to hear yours ...)

plz add yours in the comments (we will do a poll for the top 5 after getting your feedback ... )

[Tomer_Sole]

Yuri Slobodyanyuk wrote:

Yes and no  . Your solutions are perfectly correct, true, but ... don't forget that I called it 'List of Administrator's errors...' not Checkpoint product errors . So technical solutions existed long before R80.10,

e.g. wrong Policy install ? ... can't be easier than that - just check relevant gateways in "Policy Targets" menu for this policy and this will never happen. 

Removing object in use ? Well, just read the warning and click on the button "Where used" .

And while my list is compiled based on R55 - R77.30 versions, after speaking with thousands of IT guys/gals managing firewalls, I can assure you - they will 'outsmart' any technical safety measures put by bright R&D folks at Checkpoint, no one can beat the 8th layer of OSI, doesn't mean you should never try but  ...  

I encourage you to try to remove a used object from the objects bar in R80.10. Let me know if you found a way to do that.

In general, at Check Point we try to have our admins do the most by clicking the least. Some errors can be fully prevented with a smart backend platform (R80). Others are user best practices which aren't always a problem and are different between organizations - which I mentioned as stay tuned.

Of course there will always be the market for educational programs and partner sessions in which they help their customers make the most out of their Check Point products. We definitely learn from our partners' experience when shaping the future of our product line.

[Moti]

About admin errors (not common ones ) there's one story I have to share that took me 24 hours straight to figure out back in the days of February 2004 in one the largest airlines core FW

There was a worm called mydoom that attacked  www.sco.com and other IP addresses (https://www.google.com/amp/www.networkworld.com/article/2330164/lan-wan/the-worm-attacks--sco-downed...)

1)The security admin at that company created a rule #2 :any--(group called) bad-ip--http --drop

2)The symptom was downtime of traffic (ping worked ok) , if you reboot the firewall all works great for like 8 minutes and then the same down time of traffic 

3)No drops in the logs

4)After troubleshooting with Cisco R&D, check Point R&D no solution found 

5)All of the sudden I was hungry and ate a doughnut 

6)After the doughnut I drank Diet Coke (to balance the calories )

7)Then I had an epiphany they should invent a doughnut + croissant , this could work (again it was back in 2004 )

8)No sleep

9)I decided to see what's inside the group called bad-ip

10)I see a fuc$!ng domain object of www.sco.com (I.e. The fw tried to reverse DNS every packet )

11)Face palm 

12)I deleted it 

13)Everything works till today 

Lesson of the story : I should have patented cronut

[PhoneBoy]

So what you're saying is domain objects caused traffic to fall into the proverbial cronut hole?

Thankfully, they work much better in R80.10

[Moti]

Hehe yes

[Yuri_Slobodyany]

Hahaha , hilarious. 

Going down the memory lane I can recall in 10+ years at Netvision how at least 4 times clients downed their network by using this Dynamic Object - all major enterprises (and the newest case with R77.30 some 2 years ago). The user-friendliness of the SmartConsole tricked them:

" Hmm, my users are abusing the whole line to the Internet with Youtube/Facebook traffic, CEO is not happy - all is slow, but no license for URL Filtering, let's see...."

"Wait a minute, what this standing so prominently alone in the Smart DashBoard Dynamic Object does ?? Cool, it allows to enter facebook.com and youtube.com in its name and use it in Security Rules !!! Eureka ! Next pay raise is a done deal, I am genius. Let's create it as a 1st rule, source is all LANs (or any) ,destination this Dynamic Object(s), install policy and nirvana is close... Should "You were disconnected from the Management server" happen after that? Oh, here is CEO calling, probably to thank me."

"All is lost! No internet for the whole company, Checkpoint firewall failed, let me call my CSP and yell at them to fix it ASAP" 

Me: "What changes were done just before failure?"

Him:"Nothing unusual, was doing some rules"

Me:" Can you connect to the firewall from LAN by ssh?"

Him:"Not really" [then goes short explanation how to disconnect the whole LAN, or run fw unloadlocal] 

Me: "Please let it be a Standalone firewall" [I am happy - it is indeed Standalone so w/o Security policy I can connect to it from the Internet"

Me (after deleting this 1st rule and installing policy with CPU load going from 100% to 3%): "Look, this is not a technical failure this is you doing stuff you have no idea about"

Him:"Really? Ok, sorry for that ..."

Me [wishfully]: "I wish Checkpoint hid this stupid Dynamic Object deep inside Guidbedit cave or not created at all, in all these years I needed this object just once but took care of at least 20 cases of its misuse."

PS. I know what book I can write about Checkpoint - "Diaries of the phone boy or stories from the trenches" [if https://community.checkpoint.com/people/dwelccfe6e688-522c-305c-adaa-194bd7a7becc  doesn't mind franchising his handle  ] 

[PhoneBoy]

I'm assuming above you mean Domain Objects, which definitely has some performance implications and had some bugs.

R80.10 has hopefully solved those issues

Dynamic Objects are actually pretty useful in the sense they are basically placeholders in the rulebase.

On the local gateway, you can define what IP addresses those objects resolve to.

The DShield block list was implemented as a Dynamic Object.

You can use the dynamic_objects command to modify the contents of a given dynamic object without doing a policy installation.

A few pre-defined ones exist such as LocalMachine and LocalMachine_All_Interfaces.

The performance impact of Dynamic Objects is not nearly as bad as Domain Objects, though in pre-R80.10 releases, it did disable SecureXL templates on any rule that is used (as well as any rules that follow).

In R80.10, this limitation does not apply.

[Yuri_Slobodyany]

I did mean Dynamic Object and rest assured - every time you use it in the Rulebase (especially  at the top of the Rulebase) in a busy network your firewall will max out CPU load at 100% (true for all gateways up to and  included R77.30 as I've seen it close up personally, not stories from someone and never mind the hardware - 16 Gb 8 cores + CoreXL + SecureXL still went down, haven't tried with R80 so can't comment) :

While problem with Domain object is its reverse DNS resolving , with Dynamic Object it is forward IP resolving for each passing session   thanks to https://community.checkpoint.com/people/dwelccfe6e688-522c-305c-adaa-194bd7a7becc  for clarification : the real reason for Dynamic Object causing trouble is that when configured only via Dashboard and not via CLI it is left undefined..

Domain Object hasn't been so much of a problem (still happened but rarely) because it is hidden from the eye, so you would not stumble upon it as you do with Dynamic Object.

And of course I know what all these objects were meant to be used for, only that end users find more creative ways to utilize them.

[PhoneBoy]

Use of Dynamic Objects in R77.30 and earlier will disable SecureXL for the first rule it is used and all rules below that one.

If you put a Dynamic Object at the top of your rulebase, that could definitely lead to maxing out the CPU...

[Sven_Glock]

Hi Dameon, 

are you 100% sure?

I am using dynamic objects with R77.30 and fwaccel stat says that access templates are not disabled by the rules using dynamic objects.

[PhoneBoy]

I'm 100% sure on this.

In fact, there was a bug that I personally experienced around this that was fixed in Take 17 of the R77.30 jumbo: Jumbo Hotfix Accumulator for R77.30 (R77_30_jumbo_hf) 

If you're on a recent jumbo and you're still seeing this, I recommend opening a TAC ticket.

[PhoneBoy]

Dynamic Objects do no DNS lookups, they only use what has been configured in the local gateway via the dynamic_objects command.

If they didn't, then the object is basically undefined, and that could be a problem if it's used in the rulebase.

[_Val_]

Second that, it seems your opponent is a bit confused and actually means Domain objects. 

[Aaron_Pritchard]

There are some top notch commands being shared here.
What defines a 'favorite' really depends on the scenario. So i'll go with a couple of what i deem to be 'cool' commands, that you may know now (or may not) but either way probably wished you knew them sooner!:

1 - sed -I s/"text"/"newtext"/ file.name    << Find and replace when 'vi-ing' a file.

2 - watch -n 0.5 -d cpstat fw     << can use cpstat fw or any other, but the '-d' flag allows fothe autorefresh to highlight the changes. perfect for spotting increments in hit counters, of use with 'df-h' to spot a hardrive filling up during upgrade processes.

3 - du -sk * | sort -n  << got a full hardrive? no idea where the large files are? here you go, Merry Xmas!

4 - fw tab -t fwx_alloc -x  << not had to use this for a few years now, but having the gateway suddenly dropping connections due to a full NAT table isnt fun. this isnt the cleanest way to clear the table, but possibly the best knee-jerk fix to get an instant relief on the traffic flow.

5 - fw sam -v -s 10.1.1.1 -f ClusterName -t 7200 -J src 8.8.8.8  << the SAM rule. nothing cooler than an instant block of a malicious IP.

thanks all for sharing. some great stuff here.

Pritchard

CCSM

[Hugo_vd_Kooij]

I haven't seen this one yet:

 - Activate fw worker stats (per instance!)

echo 1 > /proc/cpkstats/fw_worker_0_stats

 - Read fw worker stats

cat /proc/cpkstats/fw_worker_0_stats

Again. Make sure you are aware of the impact that looking at a firewall may have on it's performance.

We could sure use some Heisenberg compensators for this.

[Daniel_]

My favorite (unsupported) command:

vsx_util downgrade

😉

[_Val_]

Actually, not so much unsupported as undocumented. "vsx_util upgrade" option allows you to downgrade VSX cluster version too. 

[Gary_Lipets]

Gateway

fw1> show route destination 10.1.1.1
Codes: C - Connected, S - Static, R - RIP, B - BGP (D - Default),
O - OSPF IntraArea (IA - InterArea, E - External, N - NSSA)
A - Aggregate, K - Kernel Remnant, H - Hidden, P - Suppressed,
U - Unreachable, i - Inactive

S 10.0.0.0/8 via x.18.47.36, eth2-08, cost 0, age 328677

Really helps figuring out which route is going to which interface to run tcpdump on 

[Scott_Reisenweb]

installed_jumbo_take

cpwd_admin list

chsh -s /bin/bash admin

[Alex_Sazonov]

In latest R80.10 we don't have installed_jumbo_take command.

It was integrated in cpinfo:

[Expert@R80-10-MGMT:0]# cpinfo -y all

This is Check Point CPinfo Build 914000176 for GAIA
[IDA]
HOTFIX_R80_10

[CPFC]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 24

[FW1]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 24

Alex

[Scott_Reisenweb]

The question posed has no reference to R80, R77, R65, etc...

[PhoneBoy]

Exactly, as there are a mix of versions that people use.

I was personally hoping for some oldies but goodies as well.

Remember fw putkey?  

[Ofir_Shikolski]

still using it

[Hugo_vd_Kooij]

Ouch. That was another fine mess you could get into. Untill you understoord exactly the various connections and how the gateway object is defined it could lead to a lot of weird connectivity issues.

It still is usefull to uderstand it as some other problems have the same root cause. Multihomes hosts are ..... a challenge.

[Alex_Sazonov]

General question is not, but my comment is for specific version. It will help to understand were it was deprecated and what to use instead of it.

This community will be in internet cache for a long time and maybe for someone running R100 it will be good to know that starting from version X they need to use different command to find installed jumbo take  

[Jejj_Longman]

R65 old school:-)

[Bhushan_Chopade]

My command :

watch cphaprob stat

[Michael_Goehrin]

compare tables of two Clusternodes from Management Server

[Expert@management:0]# fw tab -t connections -s gateway1 gateway2
HOST NAME ID #VALS #PEAK #SLINKS
gateway1 connections 8158 53 1620 69
gateway2 connections 8158 52 1610 52

[Victor_MR]

I love the commands listed here!

Already mentioned several times (some of them are utilities instead of really commands):

- cpview -t (to view the history minute by minute, previously enabled with "cpview history on").

- fw monitor (classic, a basic and very powerful utility to diagnose traffic passing through the Gateway).

- fw ctl zdebug drop

Special mention to:

- fw unloadlocal

- fwaccel stat

- fwaccel stats -s

- cphaprob state

- fw ctl multik stat -l -v -r

- fw ctl affinity -l

- cpwd_admin list

Cheers!

[Smadi_Paradise]

fw debug fwd on TDERROR_ALL_ALL=5

• Adds detailed debug to gateway's main user mode process fwd (firewall daemon). Debug can be found in fwd.elg
• I'm using it for example if a child of fwd doesn't start.
• You can debug any user mode process (a child of fwd) by replacing fwd with that process name.

[Timothy_Hall]

Well looks like this has turned into the most epic Checkmates thread ever and my last 3 commands were well-received, so here are some more that have not been mentioned yet.  The focus for these is recovering from mistakes that normally would require a firewall outage:

1) Disable anti-spoofing on the fly from the gateway.  Should you make a mistake in the anti-spoofing config the results can be dire: lots of traffic suddenly being dropped by the gateway.  Once the specific interface with the problem has been identified, best practice dictates setting the antispoofing setting for that interface to "Detect" and reinstalling policy.  On R80+ Management, quickly reverting the gateway to a known-good policy via the "Installation History" screen is a good choice as well.  At that point you can figure out what went wrong at your leisure.

But what if antispoofing is now dropping all management traffic to/from the SMS itself? A corrected policy can't be pushed, and new logs can no longer be received from the firewall either to figure out what is going on.  The usual way of breaking this catch-22 that involves taking a full outage on the firewall is:

1) Log into firewall (probably on console)

2) Unplug/disable externally-facing interface to protect firewall

3) Run fw unloadlocal (full outage begins)

4) Race back to SmartConsole and push policy with corrected antispoofing

5) Restore external-facing interface (full outage ends)

But there is a better way that does not involve taking a full outage, and these expert mode gateway commands can also be used as a bit of a "panic button" in regards to an antispoofing problem:

fw ctl set int fw_antispoofing_enabled 0
sim feature anti_spoofing off ; fwaccel off ; fwaccel on

All antispoofing enforcement on the gateway is immediately disabled.  While these settings will not survive a gateway reboot, they will survive a policy re-installation and cprestart, so once the issue is corrected make sure to turn anti-spoofing back on like this:

fw ctl set int fw_antispoofing_enabled 1
sim feature anti_spoofing on ; fwaccel off ; fwaccel on

Edit: The above commands work for R80.10 and earlier, for R80.20+ use these commands instead:

fw ctl set int fw_antispoofing_enabled 0

fw ctl set int sim_anti_spoofing_enabled 0 -a

2) Recover from SIC issues between SMS and gateway without an outage.  If SIC somehow gets broken between a single gateway and the SMS and must be reset, the traditional way of dealing with it is to run "cpconfig" on the gateway, reset SIC and enter a new activation key.  Unfortunately doing it this way causes the gateway to not only discard its current SIC certificate, but also discard its current installed security policy and load up the default "InitialPolicy", which blocks almost all traffic and causes a full outage until policy is reinstalled/fetched from the SMS. 

A far more problematic situation though is when the certificate of the SMS itself gets corrupted/changed (or the ICA somehow gets reset brutally) and SIC trust gets instantly broken between all managed gateways and the SMS.  Now you're staring down the barrel of outages on all non-clustered gateways to recover control.  But there is a way to have the gateway discard its SIC certificate and set a new activation key, but without discarding its installed security policy taken from sk86521: Reset SIC without restarting the firewall process:

On the gateway run:

cp_conf sic init ACTIVATIONKEY norestart

cpwd_admin stop -name CPD -path "$CPDIR/bin/cpd_admin" -command "cpd_admin stop"

cpwd_admin start -name CPD -path "$CPDIR/bin/cpd" -command "cpd"

Trust can now be reestablished to to the gateway and policy reinstalled, all without an outage!

3) Regain gateway CLI access via SIC if admin and/or expert passwords are unknown/corrupt.  The typical outage-inducing way to recover from this situation:  Factory reset a gateway appliance (and pray you have a good backup to restore containing passwords that you know) or on open hardware boot from a live Linux distribution DVD/USB media like Knoppix and try to hack the password that way.

The cprid_util command has been mentioned a few times in this thread, but bears repeating for this situation.  If for some reason you cannot log into the gateway CLI, assuming SIC is still established between the gateway and the SMS (and SIC traffic is not being impeded by firewall policy), you can execute commands on the gateway from the SMS via the SIC trust without a password.

Here is a sample command to reset the gateway's admin password, taken from sk106490: How to remotely reset Admin / Expert password on a Security Gateway from a Security Manage...:

$CPDIR/bin/cprid_util -server <IP_address_of_Security_Gateway> -verbose rexec -rcmd /bin/clish -s -c 'set user admin password-hash <password hash generated by grub-md5-crypt command>'

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.