GAIA_API Status shows REDIS Stopped

[D_Riddleberger]

GAIA_API Status shows REDIS is Stopped (see photo attached)

GAIA API scripts are not working

Seeing this on various devices running R81.10 HF-150 and R81.20 HF-99

gaia_api status shows build 991255299

cpinfo -y all | grep GAIA_API - shows Take: 6

Reboots do not resolve and

Expert>gaia_api restart -f all (does not resolve)

 

[Lesley]

Can you still reach it on 443? If no did you maybe turned off the ssl portal and then automatic also the API? -> https://support.checkpoint.com/results/sk/sk166692

[the_rock]

Just wondering, if you try to run api stop and api start, does it give any error? If it does and its complaining about a file, I can send you whatever needed from my working lab.

Andy

[D_Riddleberger]

<Lesley> yes we can still access over 443, no recent changes except for periodic policy installs.

<Andy> Checking and trying the api stop/start. Also do you know if there is an $FWDIR/gaia_api.elg file or is everything related to gaia_api also going to be in the api.elg file? We're not finding a gaia_api.elg anywhere so far. 

[the_rock]

I dont see that one anywhere, but you have api.elg* files in $FWDIR/log dir

Andy

[Duane_Toler]

Yes, but it's /var/log/gaia_api_server.log

Restart it with:

gaia_api stop
gaia_api restart -f all

[the_rock]

Good to know! Though, considering Dan indicated fw was rebooted, that 100% would have started the process...

Andy

[Timothy_Hall]

The Gaia API log file is /var/log/gaia_api_server.log.

[the_rock]

Did more checks on this and found some online forums where people were saying this could be related to cpu/memory issue, but also to make sure port 6379 is allowed as well.

Andy

From my lab:

[Expert@R82:0]# netstat -na | grep 6379
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:6379 127.0.0.1:40596 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40544 ESTABLISHED
tcp 0 0 127.0.0.1:40536 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40582 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40490 ESTABLISHED
tcp 0 0 127.0.0.1:40582 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40556 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40508 ESTABLISHED
tcp 0 0 127.0.0.1:40564 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40490 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40610 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40592 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40616 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40552 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40524 ESTABLISHED
tcp 0 0 127.0.0.1:40508 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40592 ESTABLISHED
tcp 0 0 127.0.0.1:40616 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40544 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40536 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40610 ESTABLISHED
tcp 0 0 127.0.0.1:40524 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40596 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:40552 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40556 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:40564 ESTABLISHED
[Expert@R82:0]#

[D_Riddleberger]

ok, so let the weirdness begin.

R81.20 HF-65 SMS and Gateways are all ok no gaia_api issues, stops, starts, reboots, restarts all AOK.

R81.10 HF-150 and R81.20 HF-99 gateways not ok

all have the same build gaia_api status shows build 991255299 (including the HF-65 devices)

Get ready to mark your bingo card, all problematic gateways show...

[Expert@cp_fw1:0]# netstat -na | grep 6379
[Expert@cp_fw1:0]#

[Duane_Toler]

That is quite interesting!  You might have found a bug!  I'd suggest you contact TAC at this point, unless you find something useful in the log file.

[the_rock]

Will check my R81.20 cluster soon...one I sent you is from R82 fw.

Andy

[the_rock]

master member R81.20 jhf 99 cluster:

[Expert@CP-FW-01:0]#
[Expert@CP-FW-01:0]# netstat -na | grep 6379
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:6379 127.0.0.1:53896 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53876 ESTABLISHED
tcp 0 0 127.0.0.1:53894 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53880 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53892 ESTABLISHED
tcp 0 0 127.0.0.1:63792 127.0.0.1:1024 ESTABLISHED
tcp 0 0 127.0.0.1:53886 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:63790 127.0.0.1:1024 ESTABLISHED
tcp 0 0 127.0.0.1:53880 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:53896 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53894 ESTABLISHED
tcp 0 0 127.0.0.1:53878 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53878 ESTABLISHED
tcp 0 0 127.0.0.1:53876 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53882 ESTABLISHED
tcp 0 0 127.0.0.1:1024 127.0.0.1:63790 ESTABLISHED
tcp 0 0 127.0.0.1:1024 127.0.0.1:63792 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53884 ESTABLISHED
tcp 0 0 127.0.0.1:53898 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:53882 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:53892 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:53884 127.0.0.1:6379 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53898 ESTABLISHED
tcp 0 0 127.0.0.1:6379 127.0.0.1:53886 ESTABLISHED
[Expert@CP-FW-01:0]#

[D_Riddleberger]

Thanks Andy,

Can you elaborate a bit more on your comment yesterday about the online forum and port 6379?

Clearly, this is definitive finding in the problematic systems whereas port 6379 is not listening for those systems.

NEW UPDATE - found 'reaped' error in /var/log/messages file (below)

Andy wrote - 'Did more checks on this and found some online forums where people were saying this could be related to cpu/memory issue, but also to make sure port 6379 is allowed as well.'

[D_Riddleberger]

 

[Alex-]

Thanks for the finding. We make minor use of the GAIA API, and started to get Error 500 on upgraded gateways when doing calls locally when the remote calls stopped. Let's hope an SK with possibly a workaround will be published soon.

[Daniel_Kuhl1]

That's an interesting finding. I need to look after it, because I planned to update my R81.20 lab devices to HF-99 later this week. 🤔

[the_rock]

Worked fine for me, never had that problem.

Andy

[Henrik_Noerr1]

We see all t99 nodes affected by this. Until now only gateways where api is not used.

But I think we need to hold back upgrading our MDS if this affects the api.

Any workarounds?

[Hugo_vd_Kooij]

I see the same thing on our MDS farm.

At the moment we don't use the GAIA API there.

We see no impact on the normal Management APII which would get us in a heap of trouble.

(I can't help but notice that QA on Jumbo Hotfix Take 99 seems to have missed a few things.)

[D_Riddleberger · Accepted solution]

Team,

Solution provided, installed and confirmed to resolve.

sk143612 - Gaia REST API: Read and send information to Check Point servers
Check Point REST GAIA API

[the_rock]

Awesome, tx for the update!

Andy

[Hugo_vd_Kooij]

I assume you mean that Build 300 fixes the issue?

[Daniel_Kuhl1]

So the bug was specifically in Build 299 and fixed within Build 300, right?