Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
SecNetEng
Contributor

API Show Logs paging returning empty log data

This post is a follow-up to https://community.checkpoint.com/t5/API-CLI-Discussion/API-show-logs/m-p/129384

The log paging method per the documentation is not working per my testing over the last year.

https://sc1.checkpoint.com/documents/latest/APIs/index.html?#web/show-logs

The SDK version was [re-]pulled fresh today.

The first batch of logs is received. The paging mechanism returns:

{
"logs": [],
"logs-count": 100,
"query-id": "<redacted>_b648614a-c0a7-4c46-b248-379b891052bf"
}

That is, "here are 100 more logs...." but the logs list returned is empty.

Specifics:

  • MDS R81.10 JHF take 78
  • MLM R81.10 JHF take 78
  • Log indexing is enabled on MDS and MLM

In certain scenarios I am getting some of the log queries with some log data, but in every case I'm getting one or more empty responses.

Could someone please confirm API log paging is working, and what configuration? (which version and JHF, MDS or SMS)

I have a very old TAC case opened with no resolution. The next step is to rebuild the MLM which our engineer has been working on a process for, for months.

Here is a complete sample script if you would like to try:

#!/usr/bin/env python3
import os
import sys
import json
import getpass

# cpapi is a library that handles the communication with the Check Point management server.
from cpapi import APIClient, APIClientArgs

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

def main():
    """
    take input from user before authenticating
    """
    # getting details from the user
    default_ip = '192.168.1.1'
    default_username = 'administrator'
    api_server = input("Enter server IP address or hostname: [{}] ".format(default_ip))
    if not api_server:
        api_server = default_ip
    username = input("Enter username: [{}] ".format(default_username))
    if not username:
        username = default_username
    if sys.stdin.isatty():
        password = getpass.getpass("Enter password: ")
    else:
        print("Attention! Your password will be shown on the screen!")
        password = input("Enter password: ")

    client_args = APIClientArgs(server=api_server)

    with APIClient(client_args) as client:
        # create debug file. The debug file will hold all the communication
        # between the python script and Check Point's management server.
        client.debug_file = "api_calls.json"
        # The API client, would look for the server's certificate SHA1
        # fingerprint in a file. If the fingerprint is not found on the file,
        # it will ask the user if he accepts the server's fingerprint.
        # In case the user does not accept the fingerprint, exit the program.
        if client.check_fingerprint() is False:
            print("Could not get the server's fingerprint - Check connectivity with the server.")
            sys.exit(1)

        # login to server:
        login_res = client.login(username, password)

        if login_res.success is False:
            print("Login failed:\n{}".format(login_res.error_message))
            sys.exit(1)
        query_data = {}
        query_data['new-query'] = {}
        query_data['new-query']['time-frame'] = 'today'
        query_data['new-query']['filter'] = '(src:10.0.0.0/8 OR src:192.168.0.0/16) AND (dst:10.0.0.0/8 OR dst:192.168.0.0/16)'
        query_data['new-query']['max-logs-per-request'] =100 # optional
        run = True
        query_id = False
        print('Pulling logs...')
        while run:
            print(json.dumps(query_data, indent=2), flush=True) # debug
            logs_res = client.api_call('show-logs',
                                             payload=json.dumps(query_data))
            if 'code' in logs_res.data:
                print(logs_res.data['code'])
                print(logs_res.data['message'])
                sys.exit(1)
            if not query_id:
                query_id = logs_res.data['query-id']
            query_data = {}
            query_data['query-id'] = query_id
            if not logs_res.data['logs']:
                print('EMPTY LOG PULL', flush=True) # debug
                print(json.dumps(logs_res.data, indent=2)) # debug
            else:
                print('LOGS RECEIVED', flush=True) # debug
            if logs_res.data['logs'] or logs_res.data['logs-count'] > 0:
                for log in logs_res.data['logs']:
                    if 'service' in log:
                        print(f"{log['src']},{log['dst']},{log['service']},{log['action']}") # debug
            else:
                print(json.dumps(logs_res.data, indent=2)) # debug
                run = False

if __name__ == "__main__":
    main()

 

12 Replies
PhoneBoy
Admin
Admin

What does $FWDIR/log/api.elg say when you're trying to make these queries?

0 Kudos
SecNetEng
Contributor

On my MDS it's not in $FWDIR/log or $MDSDIR/log. It's at /var/log/opt/CPsuite-R81.10/fw1/log/api.elg.

Here is an example of the api.elg log entry where the paging request is made and the api is returning an empty recordset with the count of 100.

 

--------------------------------------
2023-01-12 13:31:58,279  INFO org.apache.cxf.interceptor.LoggingInInterceptor.log:250 [qtp1602693230-3610324] - Inbound Message
----------------------------
ID: 5955832
Address: http://127.0.0.1:65067/web_api/v1.8/show-logs
Encoding: UTF-8
Http-Method: POST
Content-Type: application/json
Headers: {Accept=[*/*], accept-encoding=[identity], connection=[keep-alive], Content-Length=[65], content-type=[application/json], Host=[127.0.0.1:65067], User-Agent=[python-api-wrapper], X-chkp-sid=[d296ec3742a3291766d29a2e1ad3f376], X-Forwarded-For=[10.14.30.32], X-Forwarded-Host=[192.168.31.100], X-Forwarded-Host-Port=[443], X-Forwarded-Server=[192.168.31.100]}
Payload: {"query-id": "<redacted>_0855d51d-76a2-4ce1-9f93-13b4c11d5d09"}
--------------------------------------
2023-01-12 13:31:58,280  INFO com.checkpoint.management.web_api.web_services.WebApiEntryPoint.logRequestedCommandInfo:247 [qtp1602693230-3610324] - Executing [show-logs] of version [1.8] (references 1.6.1)
2023-01-12 13:31:58,288  INFO com.checkpoint.management.web_api.core.handler.commands.show_logs.ShowLogsRequestHandler.sendQueryAndUpdate:? [qtp1602693230-3610324] - Sending query with token <redacted>_0855d51d-76a2-4ce1-9f93-13b4c11d5d09
2023-01-12 13:32:01,354  INFO org.apache.cxf.interceptor.LoggingOutInterceptor.log:250 [qtp1602693230-3610324] - Outbound Message
---------------------------
ID: 5955832
Response-Code: 200
Content-Type: application/json
Headers: {Content-Type=[application/json], Date=[Thu, 12 Jan 2023 18:32:01 GMT]}
Payload: { 
  "logs" : [ ],
  "logs-count" : 100,
  "query-id" : "<redacted>_0855d51d-76a2-4ce1-9f93-13b4c11d5d09"
}

 

0 Kudos
PhoneBoy
Admin
Admin

Any ideas @Omer_Kleinstern ?

0 Kudos
Omer_Kleinstern
Employee
Employee

Paging @Nitzan_Massad 

0 Kudos
Amir_Senn
Employee
Employee

Did you try to run the API command manually without using Python?

Kind regards, Amir Senn
0 Kudos
SecNetEng
Contributor

Hello @Amir_Senn,

See the previous post I referenced: https://community.checkpoint.com/t5/API-CLI-Discussion/API-show-logs/m-p/129384

This is an issue apparently on the API side, as noted by my post from api.elg.

I have provided full code to reproduce the issue. Hopefully someone is able to verify if this affects all customers or just a subset of customers. TAC case is been open for a very long time without progress, so my purpose is to identify the severity of the issue.

 

Thank you for taking a look.

EDIT: 

SmartConsole API
https://sc1.checkpoint.com/documents/latest/APIs/index.html?#gui-cli/show-logs~v1.9


show logs new-query.time-frame "today" new-query.max-logs-per-request "100" new-query.filter "(src:10.0.0.0/8 OR src:192.168.0.0/16) AND (dst:10.0.0.0/8 OR dst:192.168.0.0/16)" --format json
show logs query-id "<redacted>_6f40bede-6b89-498a-8ab5-bad7a2ebd017" --format json

Output:

> show logs query-id "<redacted>_6f40bede-6b89-498a-8ab5-bad7a2ebd017" --format json

{
"logs" : [ ],
"logs-count" : 100,
"query-id" : "tdposh8a_6f40bede-6b89-498a-8ab5-bad7a2ebd017"
}

0 Kudos
iamnzri
Participant

I have the same problem. I am also using the cpapi and trying to get the second page of a query.

Let me know if I can help further troubleshoot the case.   

0 Kudos
omriHalifa
Employee Alumnus
Employee Alumnus

I have researched your problem. In an attempt to repoduce it, I conducted several tests:
First, I checked the mds machine with R81.20 using the command 'show logs' with paging, 
and received the correct results of logs and logs-count also with paging. 
Additionally, I ran the script you provided and received the correct results and logs, 
so the script looks good and works well. 
Next, I checked the mds machine in an environment similar to yours, R81.10 JHF take 78, 
and both the command and the script returned the correct number of logs and the desired logs with paging.
So, the problem has not reappeared for me.
Do you think there is something else that could be important and affect it? 
In addition, I recommend speaking with technical support team regarding the problem and the environment configuration.
Thank you,Omri Halifa.

0 Kudos
omriHalifa
Employee Alumnus
Employee Alumnus

I have researched your problem. In an attempt to repoduce it, I conducted several tests:
First, I checked the mds machine with R81.20 using the command 'show logs' with paging, 
and received the correct results of logs and logs-count also with paging. 
Additionally, I ran the script you provided and received the correct results and logs, 
so the script looks good and works well. 
Next, I checked the mds machine in an environment similar to yours, R81.10 JHF take 78, 
and both the command and the script returned the correct number of logs and the desired logs with paging.
So, the problem has not reappeared for me.
Do you think there is something else that could be important and affect it? 
In addition, I recommend speaking with technical support team regarding the problem and the environment configuration.
Thank you,Omri Halifa.

SecNetEng
Contributor

Thank you very much @omriHalifa! You showed there is no bug in the API.

@iamnzri welcome to the issue! I have the issue with MDS and MLM. What management and logging are you using?

 

0 Kudos
iamnzri
Participant

Here are the details to my environment:

SmartConsole
Version: R81.10
Build: SmartConsole 81.10.9600.412

cp-mgmt-api-sdk v1.6.0

I've tested a bit with different parameters. I ran the command against the MDS domain and other country-specific domain that we have. It doesn't seem to have made a difference in terms of the number of results. The log count is set to the max no of logs requested but there is no data. Funnily the api call returned a 200 code, which isn't the expected behaviour .

Then I started to change the max log number. It was set to 10, now to a 100. It seems to me that the second page produces results when I set the max log number to 100. 
Maybe this is one of the edge cases that doesn't have enough importance to be handled properly at the backend. In my opinion the backend should maybe give an error status code. 

I think that Checkpoint provides a really good API and it's quite easy to work with. So compliments to the devs. However, maybe this bit of improvement could make it even better. 

Hope this helps. 

SecNetEng
Contributor

Great information @iamnzri I will have been using max-logs-per-request 100 in still get the empty logs[] in the paging queries.

My current workaround is to use a sliding custom-start and custom-end times with smaller and smaller intervals until I'm getting <100 logs back. 😥 Then I change the interval up and down to get as many logs per query as possible. 

I agree Check Point API is good.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events