Hi team,
I am currently working on a script to retrieve all access rules from a Check Point management server using the Check Point API. However, I am encountering an issue where not all rules are being fetched.
Here are the details of the problem:
API Endpoint: /show-access-rulebase
Expected Total Rules: 2566
Retrieved Total Rules: 302
Script Behavior: The script fetches rules in batches of 50 using offsets. Initially, it retrieves some rules, but many offsets return zero rules, and the total retrieved rules are far less than expected.
Example Log Output:
---------------------------------------------------------------------------
Available access layers:
1. Network (1 rules)
2. Network (2566 rules)
Choose an access layer by entering the corresponding number: 2
2024-07-25 18:54:42,440 - DEBUG - Fetching rules with offset: 0
2024-07-25 18:54:42,768 - DEBUG - Fetched 9 rules for offset 0
2024-07-25 18:54:43,839 - DEBUG - Fetched 1 rules for offset 650
2024-07-25 18:54:47,818 - DEBUG - Fetched 0 rules for offset 2600
...
2024-07-25 18:54:49,506 - DEBUG - Fetched 0 rules for offset 3950
2024-07-25 18:54:49,508 - DEBUG - Total rules fetched: 302
---------------------------------------------------------------------------
What I Have Tried:
Thread Pooling: Using a ThreadPoolExecutor with max_workers=10 to fetch rules concurrently.
Retry Mechanism: Implemented retries for offsets that return zero rules.
Logging: Added extensive logging to track which offsets return rules and which do not.
---------------------------------------------------------------------------
Script Snippet:
python
Copy code
def get_all_rules(api_url, sid, layer_id):
rules_endpoint = f"{api_url}/show-access-rulebase"
headers = {
"X-chkp-sid": sid,
"Content-Type": "application/json"
}
payload = {
"uid": layer_id,
"limit": 50
}
rules = []
def fetch_rules(offset):
local_payload = payload.copy()
local_payload["offset"] = offset
logging.debug(f'Fetching rules with offset: {offset}')
try:
response = requests.post(rules_endpoint, headers=headers, json=local_payload, verify=False)
response.raise_for_status()
data = response.json()
fetched_rules = data.get("rulebase", [])
logging.debug(f'Fetched {len(fetched_rules)} rules for offset {offset}')
return fetched_rules
except Exception as exc:
logging.error(f'Error fetching rules at offset {offset}: {exc}')
return []
offset = 0
retries = 3 # Number of retries for each offset
while True:
with ThreadPoolExecutor(max_workers=20) as executor:
futures = {executor.submit(fetch_rules, offset + i * 50): offset + i * 50 for i in range(20)}
fetched_any = False
for future in as_completed(futures):
try:
data = future.result()
if data:
rules.extend(data)
fetched_any = True
print_progress_bar(len(rules), 2566) # Estimation de la progression
except Exception as exc:
logging.error(f'Generated an exception: {exc}')
if not fetched_any:
retries -= 1
if retries <= 0:
break
else:
retries = 3 # Reset retries if any data was fetched
offset += 1000 # Move to the next set of offsets
print() # To move to the next line after the progress bar is complete
# Ensure all rules are retrieved
rules = sorted(rules, key=lambda r: r.get("rule-number", 0))
logging.debug(f'Total rules fetched: {len(rules)}')
return rules
---------------------------------------------------------------------------
Questions:
Are there known limitations or configurations on the API that could cause it to return zero rules for certain offsets?
Is there a recommended best practice for handling large datasets of rules through the Check Point API?
Could there be specific filters or conditions that I need to apply to ensure all rules are fetched?
Any insights or suggestions from the community would be greatly appreciated.
Thank you in advance for your help!
