R82 Jumbo Hotfix Accumulator take #36 has been released today

[eranzo]

Hi All,

R82 Jumbo Hotfix Accumulator take #36 has been released today and is available for download.

Please note the following:

•        Availability:

o   Available to download the via Jumbo documentation (R82) 

o   Available for download via CPUSE by using package identifier.

o   Can be provided by customer support

Content included in this take:

• List of resolved issues in this take can be found in the Jumbo documentation (R82) 

Note:

• Central Deployment allows you to perform a batch deployment of Hotfixes on your Security Gateways and clusters from SmartConsole!! For more information, see sk168597.

Thanks,

Release Operations Group

[the_rock]

Will install it in the lab and report back.

Andy

[the_rock]

Just installed, smart console seems much more responsive.

Andy

[the_rock]

Still all good after 48 hours, no issues.

Andy

[Steffen_Appel]

Installation fails due to the api errors we have since take 25

[the_rock]

@Steffen_Appel Can you please post the errors?

Andy

[Steffen_Appel]

Basically the gaia_api stopped working and therefore the post installation check fails

[Daniel_]

@Steffen_Appel We had an gaia_api stopped issue in R81.20. A configuration file was gone during automatic upgrade and REDIS didn't start after reboot or crash.

With build 300 (and IMHO also 299) it should be solved (and some other bugs too ;-)).  In our SR sk183532 has been mentioned.

[the_rock]

That makes sense @Daniel_ . I checked the sk and it gives another sk as reference

https://support.checkpoint.com/results/sk/sk143612

I verified in my R82 lab and version does show 1.8

Andy

[Steffen_Appel]

Version 300 does not help

[the_rock]

Did you try cprestart/reboot?

[Steffen_Appel]

yes

[the_rock]

I would suggest TAC case then.

[Steffen_Appel]

We have one for a couple of weeks now (regarding the issues after T25).

[the_rock]

Any progress? What did they say?

[Steffen_Appel]

no real progress

[the_rock]

Maybe try escalate it via support portal...

Andy

[Timothy_Hall]

This is an interesting one for Scalable Platforms:

UPDATE: The "fwha_allow_different_corexl_instances" kernel parameter is now added to prevent cluster members from entering a Down state because of firewall instance count mismatches.

It appears that the hard and fast rule requiring CoreXL splits to match between cluster members is no longer applicable if this new variable is changed from the default.  It could be helpful for lab cluster setups of mismatched hardware.

[Steffen_Appel]

Looking at the logs it seems that there is a problem with python

[the_rock]

Are you able/allowed to post some of those logs?

Andy

[Steffen_Appel]

e.g.: 

Traceback (most recent call last):
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/connection.py", line 264, in connect
sock = self.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/connection.py", line 265, in <lambda>
lambda: self._connect(), lambda error: self.disconnect(error)
^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/connection.py", line 627, in _connect
raise err
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/connection.py", line 615, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/config_point/libs/lib_db.py", line 255, in fetch_hashes
db_keys = self.key_scan(pattern="{}.*".format(vsid),sort=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config_point/libs/lib_db.py", line 164, in key_scan
cur, keys = self.__db.scan(cursor=0, match=pattern if pattern else '*', count=count)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/commands/core.py", line 3025, in scan
return self.execute_command("SCAN", *pieces, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/connection.py", line 1086, in get_connection
connection.connect()
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/redis/connection.py", line 270, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
worker.init_process()
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/workers/gthread.py", line 95, in init_process
super().init_process()
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app
mod = importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/CPsuite-R82/fw1/Python/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/config_point/wsgi/app.py", line 10, in <module>
from lib_db import lib_db
File "/config_point/libs/lib_db.py", line 314, in <module>
lib_db = LibDB()
^^^^^^^
File "/config_point/libs/lib_db.py", line 53, in __init__
self.__params_paths[vsid] = self.fetch_hashes(vsid)
^^^^^^^^^^^^^^^^^^^^^^^
File "/config_point/libs/lib_db.py", line 258, in fetch_hashes
raise GenericServerError(str(re))
genericException.GenericServerError: Error 111 connecting to localhost:6379. Connection refused.

[Daniel_]

@Steffen_Appel Looks like "my" REDIS problem

> localhost:6379. Connection refused.

Port 6379 is REDIS.

Have you tried to upgrade GAiA API to build 300 (from sk143612) before? Automatic upgrade to build 299 should also work.

[Steffen_Appel]

Yes see above 🙂

[the_rock]

@Steffen_Appel Let me check that file in my lab, Im curious to see what it looks like.

Andy

[the_rock]

Hey Steffen,

So sorry, totally forgot to update, was so busy with some AWS stuff, just remembered now. So, here is the challenge I have...based on what you sent, appears its related to way more than just a single file. I mean technically, I could send you the content of the folder itself from working lab R82 management, but not sure if replacing all those files might be a wise idea or if it would even do anything to fix the issue. I guess you could back up the current files or even better, make a snapshot, but you let me know.

Best,

Andy

[Steffen_Appel]

Lets see what the TAC comes up with...

[the_rock]

Sounds good. If you do change your mind, happy to send those files offline.

Andy

[Steffen_Appel]

Looking at the trace the problem once again seems to be, that redis is inactive:

edis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

[the_rock]

Whatd does below show on your mgmt server?

netstat -anp | grep 6379

Andy

[Steffen_Appel]

/etc/rc3.d/S07start_confp start

fails as follows:

8957:C 07 Aug 2025 14:14:46.793 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8957:C 07 Aug 2025 14:14:46.793 # Redis version=5.0.4, bits=32, commit=00000000, modified=0, pid=8957, just started
8957:C 07 Aug 2025 14:14:46.793 # Configuration loaded
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 5.0.4 (00000000/0) 32 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 8957
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'

8957:M 07 Aug 2025 14:14:46.796 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
8957:M 07 Aug 2025 14:14:46.796 # Server initialized
8957:M 07 Aug 2025 14:14:46.796 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
8957:M 07 Aug 2025 14:14:46.796 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
8957:M 07 Aug 2025 14:14:46.796 # Can't handle RDB format version 11
8957:M 07 Aug 2025 14:14:46.796 # Fatal error loading the DB: Invalid argument. Exiting.

[the_rock]

What does below show? If anything other than 128, change it to that value and try starting the process again.

Anndy

[Expert@CP-MANAGEMENT:0]# more /proc/sys/net/core/somaxconn
128

[Steffen_Appel]

it is 128

[the_rock]

This message is sort of interesting, though I dont see this line in my lab. Ok, lets see what TAC tells you.

Andy

line Im referring to:

8957:M 07 Aug 2025 14:14:46.796 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

[Steffen_Appel]

seems like /gaia/redis/dumb.rdb is corrupted - after moving it redis seems to start

[the_rock]

Ah, excellent find. I just checked and that file does not even exist in my lab.

Andy

[Steffen_Appel]

But celery is still not starting....

[Steffen_Appel]

and gaia_api process as well on the second node...

[CaseyB]

R82 JHF 36 has been running in our environment for over 24 hours, no issues yet. It did resolve my RADIUS issue from R81.20 (PRJ-60446, PRHF-38975), so that is nice, that is why we did 36 over 34.