R81 SmartConsole Access issue after install of recent TAKE 81

[Jerry]

Build 564 (new SC was tried), old 563 was attempted as well, same stuff, same error.

my Customer have tried everything, from restarting the CPD to restarting FWM and finally the whole device.

no luck, you have full firewall running, fwd make logs, cod works and your security is in place however, you've got no chance to open your SmartConsole with above error and you have no technical capabilities to bring the FWM alive again. Any clues from the R&D or yourself folks?

Below details of the environment (as usual).:

[Expert@cp:0]# cpstat os

Product Name: SVN Foundation
SVN Foundation Version String: R81
SVN Foundation Build Number: 995000045
SVN Foundation Status: OK
OS Name: Gaia
OS Major Version: 3
OS Minor Version: 10
OS Build Number: -
OS SP Major: -
OS SP Minor: -
OS Version Level:

[Expert@cp:0]# cpinfo -y all

This is Check Point CPinfo Build 914000231 for GAIA
[MGMT]
HOTFIX_R81_JUMBO_HF_MAIN Take: 81
[IDA]
No hotfixes..
[CPFC]
HOTFIX_TEX_ENGINE_R81_AUTOUPDATE
[FW1]
HOTFIX_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE
HOTFIX_NGM_DOCTOR_AUTOUPDATE
HOTFIX_WEBCONSOLE_AUTOUPDATE
HOTFIX_R81_JUMBO_HF_MAIN Take: 81
HOTFIX_R81_MTA Take: 4
HOTFIX_GOT_TPCONF_AUTOUPDATE
HOTFIX_GOT_TPCONF_MGMT_AUTOUPDATE
HOTFIX_TEX_ENGINE_R81_AUTOUPDATE

FW1 build number:
This is Check Point Security Management Server R81 - Build 017
This is Check Point's software version R81 - Build 037
kernel: R81 - Build 037
[SecurePlatform]
HOTFIX_R81_JUMBO_HF_MAIN Take: 81
HOTFIX_ENDER_V17_AUTOUPDATE

***

Any ideas on how to bring the SmartConsole back to live?

[Sorin_Gogean]

Soo, Management went well also with JHF81 (we're with R81).

SmartConsole connected fine to the Managemen, and I can see Log's flowing OK.



Latter tonight (EU timezone) we'll patch 6 appliances, we'll come back with updates on those too.

[Jerry]

exactly plus this appliance is now under the full rebuild as I've exhaused all other options I'm afraid 😞

anyway, bumping to the factory default then going to hike towards R81 take 81 via 77 etc.

let's see how it goes. if that would be a VM ... well, wouldn't really have such performance on 10G upling unless it is a super powerful hypervisor on eSXI server 🙂

Cheers boys and wish me good luck on the rebuild! It is just happening now!

[the_rock]

We "met" yesterday virtually and I can tell you are a solid and hard-working dude : - ). Im sure you got this. By the way, @Jerry @Sorin_Gogean @genisis__ , I looked through my notes for the last few years for all CP customers I worked with since R80 came out initially and found 1 customer that actually had R81 also as standalone (though it was open server) and even with R81 jumbo take 81, they did not have an issue, so I dont believe Jerry its hotfix problem, but rather something else caused database to get corrupted. Its too bad we might not be able to get an answer as to why : - (

[Sorin_Gogean]

@Jerry , sorry you had that luck to encounter that problem .

Our VM's are for Logs and Management, for FWL part we're using clusters of 15600 . 

So when I read that you had issues applying JHF81, I've got some chills since I was supposed to apply the same JHF this weekend 🙂 .

Fingers crossed on your re-built, hopefully you'll not encounter other things 🙂 .

[the_rock]

Lets hope everything works for both of you at the end! Fingers crossed fellas 🤞🤞🤞

[Jerry]

looks like I'm stuck as I'm not going to boot ISO via USB when UC Licensing site is down 😞

ps. anyone notices that EVAL's generation is down just now?

[Sorin_Gogean]

I was able to log-in to UserCenter and Evaluation was accessible - I didn;t generate a license....

As for issues, I got confirmation it was resolved, couple of hours ago...


PS: why you have to boot ISO, don't you have an image on the appliance that you can revert to - like Factory default - and from there move to R81 ?

[the_rock]

That blows 😞

[genisis__]

I just tried and it seems to work for me

 

[Sorin_Gogean]

So, as an update from my side, I've applied JHF81 on R81 secondary appliances from our clusters, and all went well.
Services started properly, and policy push was OK on all.

We'll move traffic over in 15 - 20 min or so, but I don't expect issues 🙂 .

Talk latter.

[the_rock]

Glad it went well @Sorin_Gogean 

🙌

[genisis__]

sounds good.

[JozkoMrkvicka]

Is the hostname of the problematic appliance really just 2 letters "cp" ? I remember from old times that I had to modify the /etc/hosts file to map hostname with correct IP, otherwise the appliance didnt boot correctly.

Wondering if such a short hostname can cause some troubles...

I expect that you tried to use console connection or LOM to see if you got also "server not ready" while trying to access clish...

Did you try to check hard drive integrity over maintance mode ? It could also be some HDD failure. What kind of appliance it is ? Maybe removal of 1 HDD will do the trick.

Another try, maybe the last one, will be to power off the appliance, remove power cords, reseat all HDDs, reseat all SFPs, reseat all cables, wait 10 minutes and boot it up again.

Did you find the article "cannot change shell to CLISH" where the same clish error code is mentioned ?

I know it is too late for troubleshooting, even for post mortem RCA. It all depends on urgency and priority to bring the device back into production.

[the_rock]

@JozkoMrkvicka ...EXCELLENT points, however, I will address what @Jerry and myself discussed over remote session. I dont believe hostname is the issue, as all worked fine on previous jumbo and I actually had hostnames with only 2 letters before on R80 amd R81 and never had a single problem. I recall when R80 just came out, I tried hostname with just a single letter, but it told me it had to be at least 2, so I can only logically assume thats still the case, but maybe someone who works for CP can confirm 100%.

As far as hard drive integrity, thats fair point, but we dont believe based on the messages we saw that was the issue. I know in the old days of Nokia, on the IPSO appliances, you could run fsck -y from the shell and reboot, but that does not work on Gaia, so it has to be done from maintenance mode.

For your last point, we did not do that yesterday, and sadly might be too late now, as @Jerry is probably doing rebuild this weekend anyway.

Cheers mate.

Andy

[JozkoMrkvicka]

The latest version of CPUSE deployment agent should have some kind of GUI available. I assume that the deployment agent is still possible to be installed with rpm bash command (from expert mode). Would be great if someone can check that one.

If such a scenario is possible, even without access to clish, you should be able to install deployment agent in bash and using DA GUI to uninstall the latest jumbo which caused the issue and revert back to previous (working) one.

[the_rock]

Lets see, but I believe Jerry might be past that point...

[Jerry]

indeed too late chaps, on R80.40 fresh already, now the Evan's does not work from UC ... whatta crap 😞 

Thanks anyway.

ps. hostname cp is only here on CheckMates - real hostname differes 😛 Also tried that SK's mentioned. Postgre cracked I believe, nevermind, all is like a new now except ... licenses gone 😞 

[Jerry]

hi guys

few updates from yesterday and today's tweaks:

1. R80.40 up&running and all works like a charm however,

2. R81 does not like 13500 any longer with such outcome when trying to clish-import the TGZ.

3. this very same device was running R81 since its day 1 on the market so why now saying "no way"? any clues?

4. sk166536 shows NOTHING about the 13000 series, although as mentioned earlier this applinace was running R81 take 77 just perfect, now it has to stay on R80.40 take 180 and for some reason I cannot move forward and make it back to R81 which obviously is my aim and holly grail goal for that appliance and it's "secure power" capabilities with sim affinity and Dynamic Despatch (MulitK) , Dynamic Split, etc. On R80.40 it somehow does not give me a full throttle with 10G uncompromised uplink ...

***

cp> installer import local /var/log/upload/Check_Point_R81_T392_Fresh_Install_and_Upgrade_v1.tgz
The package is already located in upload folder. Continuing with the import process...
Note: The selected package will be deleted from this folder once the process is finished
Info: Initiating import of Check_Point_R81_T392_Fresh_Install_and_Upgrade_v1.tgz...
Interactive mode is enabled. Press CTRL + C to exit (this will not stop the operation)
Result: The following results are not compatible with the package:
- Machine's appliance type is Check Point 13500, machine's series name is 13000 Appliances
This installation package is not supported on Cloud environments (Microsoft Azure, Google Cloud, Amazon Web Service and Aliyun)
This installation package may not be supported on your appliance model.
For the latest software images for Check Point appliances, see sk166536
This installation package may not be supported on your server.

***

any idea why jumping from R80.40 to R81 is not such a hassle and how to fix that chaps?

[Tal_Paz-Fridman · Accepted solution]

According to Appliance Support Life Cycle Timeline 13500 does not support R81 and has already reached end of support:

https://www.checkpoint.com/support-services/support-life-cycle-policy/#appliances-support

[Jerry]

yes this is totally clear but I'm not talking "BUSINESS" to you guys, I'm talking GAIA/TECHNOLOGY, as an Employee you should have distinguish that already that not EVERYTHING is about a MONEY but "technical capabilities" and as I've mentioned earlier this post isn't about what's "LEGAL" or "PAID FOR" or ALLOWED or NOT-ALLOWED but about what's DOABLE/POSSIBLE. Don't you think this is a major difference here pal?

I was having R81 running for long long time with no single issue. it is SUPPORTED just simply not-standard-approach seem required here. That's all. But thanks for reminding me that this forum slowly becoming a playground for the R&D and "hungry" money makers developers ...

Cheers!

[Chris_Atkinson · Accepted solution]

Please refer:

https://community.checkpoint.com/t5/Partner-News-and-Updates/Remove-R81-support-for-2012-appliances/...

[the_rock]

I think there is always a fine line with these topics : - ). Just my personal experience, I find with other vendors, specially Fortinet and Cisco, even if you are running unsupported versions, they are way more helpful to find a solution than Check Point TAC. But obviously, that all goes vendor by vendor. Now, I will say, as far as running versions that are not supported on certain models, all vendors have same mentality, they usually wont bother helping much in that situation.

[Jerry]

agree this is exactly what I thought, and being in IT for far more than 2 decades I can tell (knowing CP for more than 2 decades too!) that this is typical but ... unfair. Appliance 2012 P370 so called 13500 runs on R81 like a charm, of course one can say R80.40 is not so much worse off except ... dynamic objects, Split Brain, More useful SmartConsole etc. but I am not the one who complains, I just though that if on Friday this week I was on R81 I could come back to this very same "state of play". Turns out I cannot as from R80.40 I cannot make a "jump". Blimme why. Anyway, looking for 15600 to replace that monsta except it won't have 64GB of RAM and 24 CORE's. Shame really shame.

Shalom everyone.

[Jerry]

see what @PhoneBoy wrote 2y ago:

As @Yaron_Weiler mentioned above, there is a workaround to enable installation on unsupported appliances.
For lab purposes, I presume the 2012 appliances will still work.

so can I please have that "work around" chaps? 🙂 

[the_rock]

Let me see if I can find it, I know I did this before and it worked, just cant recall exactly what line was changed.

[genisis__]

I'm wondering if you originally used a earlier build of R81 which did not have this restriction in place? (taking a total guess here), clearly if you managed to get R81 working previously and did not have to do anything special (you would have remembered that), then all I can think of is earlier build.

[PhoneBoy]

The editing /etc/appliance_config.xml trick (which I don't recall the precise details on) will only work when you do a fresh install on the target appliance BEFORE First-Time Wizard is run.
Meaning, you can't use this trick to do an in-place upgrade.

[_Val_]

@Jerry I understand that you are upset, but I do not find your comment, and also @the_rock addon to be fair. 

The appliance is clearly not just out of engineering support, but out of support completely, for more than 6 months. The engineering support expired two years ago. 

That means, new versions are no longer tested with this appliance, and no issues will be expected to be fixed on it if they ever arise.  

R81 was not tested on that appliance. If you elect to run an outdated appliance, you have to use the supported version on that, at least.

I can only guess if it was an oversite on your end or a continence choice, but it is not okay to blame the vendor for the outcome, in your specific case.

Thanks for your understanding.

[RS_Daniel]

Hello,

AFAIK as it is not supporte by CP officially DA will tell you upgrade not possible. Talking in tech words i installed R81.10 in 4400 and 4600 appliances and are working fine. The problem is that i did it with a clean install using USB and isomorphic, i am not sure if that is possible for you. Honestly i read the post very quickly and maybe i missed something. HTH.

Regards

[Jerry]

hi

I did R81 from ISO/ via isomorphic 1000s of time and indeed this time I did 13500 with R81 from USB boot, then it started to crash and reboot by itself so I had no other way and boot R80.40. Then the whole install went like a magic and all works like a charm except ... no R81 feautures hence my frustration. I've left 13500 on the R80.40, tweaked and tuned and all is fine but I did something else, I've purchased from one of my CP friends 15600 and I've got USB ready to boot R81.20 so no panic, seems the best bet was to replace (later on this week) bloody 64GB of RAM 13500 with 15600 and have at least 2 years of peace 🙂

Cheers!