API show logs

[JozkoMrkvicka]

Hello guys,

I want to use API call "show logs" to show me all logs. I want to see ALL logs, not only last 100. Is that possible ? Using following commands I am able to get only 100 logs, not more (within 7 days period):

mgmt_cli -s sid.txt show logs new-query.filter "src:10.20.30.40" new-query.time-frame last-7-days --format json
mgmt_cli -s sid.txt show logs query-id "$QUERYID" --format json

Variable QUERYID is fetched from first API call.

First 100 logs are shown, but then if I want to go to the "next page" I am getting following from query-id:

{
  "logs" : [ ],
  "logs-count" : 100,
  "query-id" : "WEB_API_3eb4f228-abff-4cbf-83bb-377bcc3272ad"
}

There are for sure more than 100 logs (checked within SmartView and SmartConsole).

Running latest Take of R80.30.

[Tal_Paz-Fridman]

Looking at the Management API Reference Guide it seems 100 is the limit:

https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/show-logs~v1.8%20

I'll forward this to R&D owners to see if this can be improved in future versions.

[JozkoMrkvicka]

My guess would be to do not stress API with huge output - therefore maximum 100 logs per call.

Anyway, I managed to handle it with infinity loop (while true) where I am checking "log-count" value. If it is less than 100, the log search is over.

Also the issue with empty "query-id" was solved, but I dont know how 😄 Maybe the quotes were the issue...

I would like to have the same options like we have in SmartView GUI. For example, you can choose which columns you want to export (not all like in API call). Something like "set log-template" where you will be able to add/remove columns according your needs. Once set, add mandatory parameter in "show logs" to include the log template.

[Amir_Senn]

After your initial command you will also get session id/sid.

Please try this syntax:

mgmt_cli show-logs query-id <query-id> --session-id <session-id>

This should page further. You can repeat this command for further paging.

I think session is alive for 600 seconds after your initial command + credentials.

[mohit_tater]

I am also facing a similar issue with the show-logs API. let's say I want to retrieve more than 100 logs, I make an API call. The first call returns me the log-count(100), list of logs event (size 100) and a query-id.

I use this query-id to make subsequent API call and this call returns me log-count(100) but the list of logs is empty []. Any API call after this point behaves similarly.

payload for the first request:

{

"new-query": {

"time-frame": "custom",

"custom-start": "2021-08-10T12:05:19.000Z",

"custom-end": "2021-09-10T11:20:31+0000",

"filter": "blade:\"Threat Emulation\"",

"type": "logs",

"max-logs-per-request" : 100

}

}

payload for subsequent requests

{

"query-id": "admin-api_cf0eac6a-883c-42e8-xxxxxxxxxxxxxxx"

}

I am not sure if I am doing something wrong here. Please help.

[PhoneBoy]

Recommend a TAC case.

[Amir_Senn]

I think it will show you 100 max logs per request regardless of number of results.

I suggest you try a few things:

Make sure that in SmartConsole you have results for that filter and timeframe.

Try to drop the filter/timeframe and do the query again and see if you have results.

[JozkoMrkvicka]

I have to run the script more times in order to get me the real logs based on filter set.

Sometimes the output from "show logs" is only 100 logs, sometimes 300 logs, sometimes all logs (lets say I have 1543 logs for filter in total).

Looks like the api call is warming up and is fully ready after multiple triggers 😄 Like starting your old car in the winter 😄

[Douglas_Rich]

only 100 Logs?  What's the use case for the API then, if it's only 100 logs it seems not very useful. 

[JozkoMrkvicka]

yeah, even on R81.10 with the latest jumbo, the API "show logs" looks buggy. But as I mentioned, if I run the script more times, it will get more logs than 100. But sometimes every run gives different number of logs.

[PhoneBoy]

Most of our API calls return a limited number of results by design to keep the API server performant.
You have to make multiple calls to the endpoint to retrieve all the logs, as noted.

[Paul_Hagyard]

Given the logic is already in SmartView to export logs in bulk (to 1M), it would make sense to include access for this in the API. Send an API request which runs the export on the management server and provide a request ID. Poll with the request ID to see when it's completed, and when completed return a URL for retrieving the file.

"show logs" doesn't even report the total number of logs matching the query (timeframe and query string), you have to just keep pulling data until a request returns nothing. Just return "logs_remaining: x" in each query.

Quicker to write an Autohotkey kb/mouse macro to login to Smartview and export logs 🙂

[Paul_Hagyard]

Okay, the export does return logs remaining, not just the logs requested, sorry!

"logs-count" : 96,