Re: Firemon - Check Point R80.10 API not working.

Jonathan_Horne · ‎2018-04-06

Currently running Check Point Multi-domain R80.10 JHF 91 and trying to retrieve/pull configurations into Firemon v8.21. On the MDS API status shows good. I am seeing error in api.elg and httpd log. I know there is SK for known issue, but that was resolved in previous JHF. I am not sure if this is a Firemon issue or a Check Point issue. Anyone else seeing or having similar issue? Any help is appreciated. Thanks!

Error Log from Firemon:

Last Updated: 4/2/2018 6:11 PM

Status: Failure

Description: Manual Retrieval

Retrieval Error: Error: Error Summary:

Exception: http status=502 content=<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<title>502 Proxy Error</title>

</head><body>

<h1>Proxy Error</h1>

<p>The proxy server received an invalid

response from an upstream server.<br />

The proxy server could not handle the request <em><a href="/web_api/show-groups">POST&nbsp;/web_api/show-groups</a></em>.<p>

Reason: <strong>Error reading from remote server</strong></p></p>

</body></html>

The API status is good on Check Point and all IP’s are allowed to communicate to it.

API Status:
[Expert@XXXXXXX:0]# api status

API Settings:

---------------------

Accessibility: Require all granted

Automatic Start: Enabled

Processes:

Name State PID More Information

-------------------------------------------------

API Started 12783

CPM Started 876 Check Point Security Management Server is running and ready

FWM Started 7500

Port Details:

-------------------

JETTY Internal Port: 50277

APACHE Gaia Port: 443

--------------------------------------------

Overall API Status: Started

--------------------------------------------

API readiness test SUCCESSFUL. The server is up and ready to receive connections

Notes:

------------

To collect troubleshooting data, please run 'api status -s <comment>'

Here is what I am seeing in the API logs.

Api.elg

ID: 756

Address: http://127.0.0.1:50276/web_api/show-access-rulebase

Encoding: ISO-8859-1

Http-Method: POST

Content-Type: application/json

Headers: {Accept=[*/*], accept-encoding=[gzip, deflate], Cache-Control=[no-cache], connection=[keep-alive], Content-Length=[108], content-type=[application/json], Host=[127.0.0.1:50276], User-Agent=[python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-693.21.1.el7.x86_64], X-chkp-sid=[XsV3X_LLY9jsSZAvMMq8rXje6NWqaw-WYOEagzp9yCg], X-Forwarded-For=[10.178.17.41], X-Forwarded-Host=[10.178.17.33], X-Forwarded-Host-Port=[443], X-Forwarded-Server=[10.178.17.30]}

Payload: {"offset": 100, "limit": 100, "uid": "93b6a23a-5c39-4807-8117-b860cf775ec1", "use-object-dictionary": false}

--------------------------------------

2018-04-05 06:48:03,335 INFO com.checkpoint.management.web_api_is.utils.helpers.ApiCache.<init>:25 [qtp839789802-28] - Cache created and initialized

2018-04-05 06:48:03,335 INFO com.checkpoint.management.web_api.web_services.WebApiEntryPoint.logRequestedCommandInfo:132 [qtp839789802-28] - Executing [show-access-rulebase] of version 1.1

2018-04-05 06:48:04,109 WARN com.checkpoint.management.web_api_is.utils.managers.command_manager.WebApiCommandManager.getSafeStandardReplyClassByCpmClassSimpleName_aroundBody18:203 [qtp839789802-28] - Getting standard reply class for CPM class [CpmiAnyObject] for API version [1.1] failed. Returning default standard reply class.

2018-04-05 06:48:04,122 WARN com.checkpoint.management.web_api_is.utils.managers.command_manager.WebApiCommandManager.getSafeStandardReplyClassByCpmClassSimpleName_aroundBody18:203 [qtp839789802-28] - Getting standard reply class for CPM class [Global] for API version [1.1] failed. Returning default standard reply class.

Httpd2_error_log

[Mon Apr 02 12:44:32.179081 2018] [proxy_http:error] [pid 18072] (70007)The timeout specified has expired: [client 10.178.17.41:34140] AH01102: error reading status line from remote server 127.0.0.1:50277

[Mon Apr 02 12:44:32.179150 2018] [proxy:error] [pid 18072] [client 10.178.17.41:34140] AH00898: Error reading from remote server returned by /web_api/show-groups

[Mon Apr 02 13:04:37.578421 2018] [proxy_http:error] [pid 25356] (70007)The timeout specified has expired: [client 10.178.17.41:55488] AH01102: error reading status line from remote server 127.0.0.1:50277

[Mon Apr 02 13:04:37.578492 2018] [proxy:error] [pid 25356] [client 10.178.17.41:55488] AH00898: Error reading from remote server returned by /web_api/show-groups

[Mon Apr 02 13:09:06.196842 2018] [proxy_http:error] [pid 26534] (70007)The timeout specified has expired: [client 10.178.17.41:35238] AH01102: error reading status line from remote server 127.0.0.1:50277

[Mon Apr 02 13:09:06.196915 2018] [proxy:error] [pid 26534] [client 10.178.17.41:35238] AH00898: Error reading from remote server returned by /web_api/show-groups

[Mon Apr 02 13:28:08.748437 2018] [proxy_http:error] [pid 1065] (70007)The timeout specified has expired: [client 10.178.17.41:56516] AH01102: error reading status line from remote server 127.0.0.1:50277

[Mon Apr 02 13:28:08.748505 2018] [proxy:error] [pid 1065] [client 10.178.17.41:56516] AH00898: Error reading from remote server returned by /web_api/show-groups

Here is a connection attempt from Firemon to Check Point

PhoneBoy · ‎2018-04-06

It's possible the bug mentioned in the SK regressed, in which case a TAC case is warranted.

http://www.checkpoint.com/support-services/contact-support/index.html

Robert_Decker · ‎2018-04-08

Hi,

api.elg shows logs for show-access-rulebase command and Httpd2_error_log shows logs for show-groups command.

The error in Httpd2_error_log indicates that there is a timeout error, probably due to a huge number of group members in one of the group objects. This is normal.

We've released JHF (and SK) on how to deal with such situation with dedicated flags(membership).

Please consult with Firemon if they have implemented this in their scripts.

Robert.

Jonathan_Horne · ‎2018-04-09

Could you please provide SK # and JHF Build?

Also is it normal that I should receive no packages when running $MDS_FWDIR/scripts/web_api_show_package.sh

[Mon Apr 09 07:58:40 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: The parameters that were received:
[Mon Apr 09 07:58:40 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Limit number of object per page: 10
[Mon Apr 09 07:58:40 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Local Ips: [10.178.17.33, 10.178.17.40, 10.178.17.30, 10.200.10.16, 127.0.0.1]
[Mon Apr 09 07:58:42 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Login As root: true
[Mon Apr 09 07:58:42 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Login with 'read-only' flag.
[Mon Apr 09 07:58:50 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.writeTheVersionsToTheLogger()INFO]: Management API running version: 1.1
[Mon Apr 09 07:58:50 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.writeTheVersionsToTheLogger()INFO]: show_package v1.1.3
[Mon Apr 09 07:58:50 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.writeTheVersionsToTheLogger()INFO]: Chosen port: 443
[Mon Apr 09 07:58:50 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Chosen server IP: 127.0.0.1
[Mon Apr 09 07:58:50 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Login response: {"api-server-version":"1.1","last-login-was-at":{"iso-8601":"2018-03-29T10:12-0600","posix":1522339920},"standby":false,"read-only":true,"url":"https:\/\/127.0.0.1:443\/web_api","sid":"xXXsYslHzyFLl45e9rAIBFB_aujPjhvuk0pC1S21uTI"}
[Mon Apr 09 07:58:50 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Run command: 'show-gateways-and-servers' with details level 'full'
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Found 313 gateways from 'show-gateways-and-servers'
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.collectGatewaysInUseAndInstalledPolicies()INFO]: Found 56 gateways that have a policy installed on them
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Run command: 'show-vpn-communities-star' with details level 'full'
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Run command: 'show-vpn-communities-meshed' with details level 'full'
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Found 0 vpn communities
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Run command: 'show-packages' with details level 'full'
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.debug()DEBUG]: Found 0 packages
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.warning()WARNING]: No packages found on the server. Try to login to a user's domain
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.logoutReportAndExit()INFO]: Script finished running with warnings!
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.logoutReportAndExit()INFO]: dirPath: /home/admin/1c095c53-44a0-4476-863e-d44dac5f18e5
[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.ShowPackageTool.logoutReportAndExit()INFO]: tarGzPath: show_package-2018-04-09_07-58-40.tar.gz

Robert_Decker · ‎2018-04-09

are you running on MDS?

if yes, you should use -d switch and specify the requested domain name.

for full options list, use -h switch.

Jeremy_Boselly · ‎2018-04-09

Robert Decker‌

I to would like to know the SK number as well. The most current Check Point API documentation related to the 'show-groups' call ( Check Point - Management API reference ) does not include any information about an argument called "membership". Also when I just tested this argument I received from the Check Point API the error "Unrecognized parameter [membership]".

Jonathan Horne‌

Whenever I've seen a customer open a ticket with this issue we've referred them back to TAC and the resolution has either been for them to upgrade to the newest JHF or the CheckPoint technician manually alters the RAM restrictions on the API. As far as I'm aware the default max RAM allowed for the API is 256MB in R80.10, which can cause timeout's or failures if group membership is large.

As to your last comment, whenever you're running an API call related to non-global domain related data (I realize you're using an internal shell script that mimics an external API call) on an MDS or CMA under an MDS (multi-domain environments), during API login you must include the domain your future requests will be requesting data for.

For example if I want to pull packages related to Domain: X, my login would look like:

Then your future "show-packages" API calls would return the packages for domain X.

Here is the error message where it indicates this is the issue:

[Mon Apr 09 08:00:22 MDT 2018 com.checkpoint.mgmt_api.examples.MyLogger.warning()WARNING]: No packages found on the server. Try to login to a user's domain

Long story short you'll want to use the '-d' argument to specify the domain in question.

Tomer_Sole · ‎2018-04-09

integrated since Jumbo Hotfix Take 70 (January 15).

We will update the SK.

You can always make sure your personal Management Server has the relevant API syntax by opening https://[your Management IP]/api_docs

Jonathan_Horne · ‎2018-04-09

I am running R80.10 with JHF 91, so I should have the fix then.

This is what I get when I trying to run with -d

[Expert@XXXXX:0]# $MDS_FWDIR/scripts/web_api_show_package.sh -d DOMAINIP -u XXXXXXX -p XXXXXX
Exception in thread "main" java.lang.NullPointerException
at com.checkpoint.mgmt_api.examples.ShowPackageTool.setGatewayAndServerPolicy(ShowPackageTool.java:1213)
at com.checkpoint.mgmt_api.examples.ShowPackageTool.buildNewGatewayOrServer(ShowPackageTool.java:1185)
at com.checkpoint.mgmt_api.examples.ShowPackageTool.collectGatewaysInUseAndInstalledPolicies(ShowPackageTool.java:244)
at com.checkpoint.mgmt_api.examples.ShowPackageTool.main(ShowPackageTool.java:158)

Robert_Decker · ‎2018-04-10

please paste here the output of "cpinfo -y all" command on your machine.

robert.

Jonathan_Horne · ‎2018-04-10

CPINFO says it is still R80.10 JHF 70.

[Expert@xxxxxzxx:0]# cpinfo -y all

This is Check Point CPinfo Build 914000176 for GAIA
[IDA]
HOTFIX_R80_10

[KAV]
HOTFIX_R80_10

[CPFC]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 70

[FW1]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 70
HOTFIX_R80_10_JHF_70_IPONLY
HOTFIX_R80_10_JHF_70_958

FW1 build number:
This is Check Point Security Management Server R80.10 - Build 009
This is Check Point's software version R80.10 - Build 043

[SecurePlatform]
HOTFIX_R80_10_JUMBO_HF Take: 70

[NGXCMP]
HOTFIX_R80_10

[EdgeCmp]
HOTFIX_R80_10

[SFWCMP]
HOTFIX_R80_10

[SFWR75CMP]
HOTFIX_R80_10

[SFWR77CMP]
HOTFIX_R80_10
HOTFIX_R80_10_JHF_COMP

[FLICMP]
HOTFIX_R80_10

[R75CMP]
HOTFIX_R80_10

[R7520CMP]
HOTFIX_R80_10

[R7540CMP]
HOTFIX_R80_10

[R7540VSCMP]
HOTFIX_R80_10

[R76CMP]
HOTFIX_R80_10

[R77CMP]
HOTFIX_R80_10

[PROVIDER-1]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 70

[Reporting Module]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 70

[SmartLog]
HOTFIX_R80_10

[CPinfo]
No hotfixes..

[VSEC]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 70

[DIAG]
HOTFIX_R80_10

[MGMTAPI]
No hotfixes..

[CPUpdates]
BUNDLE_R80_10_JUMBO_HF Take: 70

[rtm]
No hotfixes..

Robert_Decker · ‎2018-04-10

Ok, this explains the error message from the show_package tool...

please check why your system is not upgraded as you thought it was.

Robert.

Tomer_Sole · ‎2018-04-10

If these API calls happen through Firemon, and not controlled by you, then you should probably contact Firemon regarding can they utilize the new performance improvement flags - "membership".

By the way I also asked them to look into this thread but it will probably help if you have a support ticket at the Firemon side.

Jeremy_Boselly · ‎2018-04-10

Tomer Sole‌

I'm on the T91:

I did as you suggested and checked "https://[your Management IP]/api_docs", however I'm still not seeing anything related to a 'membership' or 'dereferencing' argument that can be included with the "show-groups" API call.

Does the sk121292 have details on how these new flags should be utilized? If so would it be possible for you to forward me a copy I don't have access to it. Thank you.

Robert_Decker · ‎2018-04-10

Usage

After installing the Hotfix, set the new request fields below to reduce the amount of information returned and improve the performance of the Security Management API:

show-membership - set it to False to stop showing groups membership in response ("groups" field will not be calculated)
- Default value: True
- Supported commands: show-access-rulebase, show-nat-rulebase, show-threat-rulebase, show-threat-rule-exception-rulebase, show-groups, show-application-site-groups, show-service-groups, show-objects, show-unused-objects, where-used, show-simple-gateways, show-hosts, show-networks, show-address-ranges, show-multicast-address-ranges, show-dynamic-objects, show-security-zones, show-opsec-applications, show-dns-domains, show-application-sites, show-services-dce-rpc, show-services-icmp, show-services-icmp6, show-services-other, show-services-rpc, show-services-sctp, show-services-tcp, show-services-udp, show-data-center-objects
dereference-group-members - set it to False to stop group members dereference (they will be shown as list of uids independently of the details level requested)
- Default value: True
- Supported commands: show-access-rulebase, show-nat-rulebase, show-threat-rulebase, show-threat-rule-exception-rulebase, show-groups, show-application-site-groups, show-service-groups, show-objects, show-unused-objects, where-used

Jeremy_Boselly · ‎2018-04-11

https://community.checkpoint.com/people/rdeck5af054c0-4c71-4395-9c31-2a794ff5bc37‌ Perfect, thank you!

Robert_Decker · ‎2018-04-10

The new "membership/dereferencing" flags are intended to improve the performance of API commands.

The SK: sk121292

The JHF: R80.10 Jumbo HotFix - General Availability Take 70 (15 Jan 2018)

Robert.

Raj_Khatri · ‎2018-04-12

We have been experiencing similar issues with Firemon connected to R80.10 on take 70. We have 2 Firemon instances, running 8.18 and 8.21 and both exhibit issues connecting to the MDS when it was running take 42 or 70. For whatever reason, revisions seem to stop coming in and have to kick off a manual CMA retrieval. Restarting MDS seems to fix things for a limited amount of time as well.

With respect to the API memory, the default is 256MB and we noticed retrievals would never completed. We followed sk119553 and after changing to 64-bit and allocating 4GB of RAM, that helped out.

Multiple revisions and queries are being performed via API when a policy is published or installed to a firewall and due to Checkpoint’s slow API process, when simultaneous queries are performed, the Python API query is getting killed instead of a graceful disconnect.

We have an open case with Firemon support regarding this already, so hopefully it gets addressed soon.

Robert_Decker · ‎2018-04-12

As a rule, when you know for sure that you have huge amount of data to process, there are several options:

1. increase the default RAM for API memory - sk119553

2. use new membership flags to improve the performance, as noted above

3. increase mgmt_cli default timeout (3 minutes), starting from R80.10 Jumbo HotFix - Ongoing Take 79

4. increase APACHE server default timeout (5 minutes)

The sk121292 document lists all API commands that may be affected by retrieved data amount.

Robert.

Carlo_Gardener · ‎2018-12-05

I have the same issue within this thread, it has been a challenge for us with API and Firemon, granted the API stopped several times when using Firemon to retrieve data from our R80.10 Mgmt. I have gone through with CP and the folks whom are working with Firemon. It was told to us by one of the engineers to update our Jumbo take due to the API crashing under heavy stress. we went to Take 154 and also increase the API memory size (4096) to the recommended by CP. after doing so the same issue still occurs when invoking Firemon to retrieve data. I have included several things which I have tried and saw different error messages and the API stopping. I Have kicked this back to Firemon due to our API as always work until we involved this tool within our environment.

I have tried several attempts using different outlined methods unsuccessful.

Using a specific account with read-only access (failed)
Using a specific account with SuperAdmin access (failed)
Using company api account (failed)
Patch our Management to the latest Jumbo Take (154) and increased API memory size to re-mediate the API crashing during the data retrieval( no luck- failed )
Open a case between checkpoint and the account which I tested with to see if there are any correlation with the api failing ( no issues , API is functioning as it should , we have been using the company-api functionality in the environment since February 2018 ) this can be ruled out.
Created two separate accounts for the retrieval attempt

Account for CLI access with elevated admin access
Account for API access with read-only access

Created an OPSEC connection for API functionality with read-only access and attempt another retrieval, this also failed with the following message. it is clear that the issue is on the Firemon side.

Starting Check Point R80 retrieval

[ 4099058496][4 Dec 1:05:24] get_pkxld_path: cpshared_filename failed

/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html

InsecureRequestWarning)

Retrieval failed, error: Error Summary:

Exception: http status=503 content=<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<title>503 Service Unavailable</title>

</head><body>

<h1>Service Unavailable</h1>

<p>The server is temporarily unable to service your

request due to maintenance downtime or capacity

problems. Please try again later.</p>

</body></html>

Are you a member of CheckMates?

Firemon - Check Point R80.10 API not working.

Usage