Well after multiples test I know now why It was not working but still not understand why It acts like that.
So I have 2 functions :
1 standard, without any parallelels call which is like this :
def get_all_rules2(api_url, sid, layer_id):
# Define the endpoint URL for the API request to get the access rulebase
rules_endpoint = f"{api_url}/show-access-rulebase"
# Set the headers for the API request, including the session ID and content type
headers = {
"X-chkp-sid": sid,
"Content-Type": "application/json"
}
# Prepare the initial payload for the API request with the layer ID and limit
payload = {
"uid": layer_id,
"limit": limit
}
rules = [] # List to store all fetched rules
offset = 0 # Initialize offset for pagination
total_rules = count_rules_in_layer(api_url, sid, layer_id) # Get total rules count
iteration = 0 # Initialize iteration counter
i = 0 # Simple counter for debugging purposes
while True:
payload["offset"] = offset # Update offset in the payload
response = requests.post(rules_endpoint, headers=headers, json=payload, verify=False) # API request
response.raise_for_status() # Raise error if request failed
data = response.json() # Parse JSON response
rules.extend(data["rulebase"]) # Add fetched rules to the list
iteration += len(data["rulebase"]) # Update iteration counter
print_progress_bar(iteration, total_rules) # Display progress
print(f"i = {i}") # Print current value of the counter
input("Press Enter to continue") # Wait for user input to continue
# Check if the total number of rules has been reached
if not data.get("total", 0) > len(rules):
break
offset += len(data["rulebase"]) # Update offset for next iteration
print() # Move to the next line after progress bar completion
return rules # Return the list of fetched rules
This one works fine and retrieves me all the rules from my policy but what I tried is to speed up this as It takes more than 150seconds to retrieves all the rulebase
What I did to understand how It works Is I put an input to manually go over the loop and displayed the offset value and rules lenghts after each iteration and here is the output : (Fetch rules 2 run to the function above)
Current parameters:
1. Limit: 50
2. Max workers: 10
3. Retries per offset: 3
4. Total rules expected: 2566
5. Fetch rules
2. Fetch rules 2
6. Exit
Choose a parameter to modify (or fetch/exit): 6
|----------------------------------------| 9 rules / 2566 Complete i = 0
Press enter to continue
|----------------------------------------| 13 rules / 2566 Complete i = 1
Press enter to continue
|----------------------------------------| 16 rules / 2566 Complete i = 2
Press enter to continue
|----------------------------------------| 19 rules / 2566 Complete i = 3
Press enter to continue
|----------------------------------------| 22 rules / 2566 Complete i = 4
Press enter to continue
|----------------------------------------| 25 rules / 2566 Complete i = 5
Press enter to continue
|----------------------------------------| 28 rules / 2566 Complete i = 6
Press enter to continue
|----------------------------------------| 33 rules / 2566 Complete i = 7
Press enter to continue
|----------------------------------------| 39 rules / 2566 Complete i = 8
Press enter to continue
|----------------------------------------| 43 rules / 2566 Complete i = 9
Press enter to continue
|----------------------------------------| 47 rules / 2566 Complete i = 10
Press enter to continue
|----------------------------------------| 51 rules / 2566 Complete i = 11
Press enter to continue
|----------------------------------------| 55 rules / 2566 Complete i = 12
Press enter to continue
|----------------------------------------| 59 rules / 2566 Complete i = 13
Press enter to continue
|----------------------------------------| 64 rules / 2566 Complete i = 14
Press enter to continue
|█---------------------------------------| 70 rules / 2566 Complete i = 15
Press enter to continue
|█---------------------------------------| 76 rules / 2566 Complete i = 16
Press enter to continue
|█---------------------------------------| 81 rules / 2566 Complete i = 17
Press enter to continue
|█---------------------------------------| 84 rules / 2566 Complete i = 18
Press enter to continue
|█---------------------------------------| 88 rules / 2566 Complete i = 19
Press enter to continue
|█---------------------------------------| 94 rules / 2566 Complete i = 20
Press enter to continue
|█---------------------------------------| 100 rules / 2566 Complete i = 21
Press enter to continue
|█---------------------------------------| 106 rules / 2566 Complete i = 22
Press enter to continue
|█---------------------------------------| 111 rules / 2566 Complete i = 23
Press enter to continue
|█---------------------------------------| 115 rules / 2566 Complete i = 24
Press enter to continue
|█---------------------------------------| 119 rules / 2566 Complete i = 25
Press enter to continue
|█---------------------------------------| 123 rules / 2566 Complete i = 26
Press enter to continue
|█---------------------------------------| 127 rules / 2566 Complete i = 27
Press enter to continue
|██--------------------------------------| 131 rules / 2566 Complete i = 28
Press enter to continue
|██--------------------------------------| 134 rules / 2566 Complete i = 29
Press enter to continue
|██--------------------------------------| 136 rules / 2566 Complete i = 30
Press enter to continue
|██--------------------------------------| 137 rules / 2566 Complete i = 31
Press enter to continue
|██--------------------------------------| 138 rules / 2566 Complete i = 32
Press enter to continue
|██--------------------------------------| 139 rules / 2566 Complete i = 33
Press enter to continue
|██--------------------------------------| 140 rules / 2566 Complete i = 34
Press enter to continue
|██--------------------------------------| 141 rules / 2566 Complete i = 35
Press enter to continue
|██--------------------------------------| 142 rules / 2566 Complete i = 36
So as you can see above, after each iteration and each post request with 50 as the limit, I retrieve for the 1st request only 9 rules then 4 then 3 then 3 and so on but never the 50 from the limit.... This is the why It doesn't retrieves the 50 rules which in my opinion is that much....
My other function which do the parallel now works as I fixed it to do some retries if the workers doesn't reach the limit set.
Here is the code :
def get_all_rules(api_url, sid, layer_id):
# Define the endpoint URL for the API request to get the access rulebase
rules_endpoint = f"{api_url}/show-access-rulebase"
# Set the headers for the API request, including the session ID and content type
headers = {
"X-chkp-sid": sid,
"Content-Type": "application/json"
}
# Expected total number of rules (can be adjusted based on your needs)
total_rules_expected = 2566
# Prepare the initial payload for the API request with the layer ID and limit
payload = {
"uid": layer_id,
"limit": limit
}
rules = [] # List to store all fetched rules
rule_ids = set() # Set to track unique rule IDs
def fetch_rules(offset, retries, current_rules=[]):
"""
Function to fetch rules from the API with a given offset and retries.
Uses recursion to handle partial fetching and retries.
"""
local_payload = payload.copy()
local_payload["limit"] = limit - len(current_rules)
local_payload["offset"] = offset
try:
# Send API request to fetch rules
response = requests.post(rules_endpoint, headers=headers, json=local_payload, verify=False)
response.raise_for_status()
data = response.json()
fetched_rules = data.get("rulebase", [])
# If limit is reached or no more retries left, return the current list of rules
if len(current_rules + fetched_rules) == limit or retries <= 0:
current_rules.extend(fetched_rules)
return current_rules
else:
# Recursively fetch more rules
return fetch_rules(offset + len(fetched_rules), retries - 1, current_rules + fetched_rules)
except Exception as exc:
print(f'ERROR: Error fetching rules at offset {offset}: {exc}')
return current_rules
offset = 0
total_fetched_rules = 0
while True:
# Use ThreadPoolExecutor to fetch rules in parallel
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Create a set of futures for parallel fetching
futures = {executor.submit(fetch_rules, offset + i * limit, retries_per_offset): offset + i * limit for i in range(max_workers)}
fetched_any = False
# Process each completed future
for future in as_completed(futures):
try:
data = future.result()
if data:
for rule in data:
rules.append(rule)
total_fetched_rules += len(data)
fetched_any = True
print_progress_bar(total_fetched_rules, total_rules_expected) # Estimate progress
except Exception as exc:
print(f'ERROR: Generated an exception: {exc}')
# Break the loop if no more rules are fetched or the expected total is reached
if not fetched_any or total_fetched_rules >= total_rules_expected:
break
offset += max_workers * limit # Move to the next set of offsets
print() # Move to the next line after the progress bar is complete
print(f'Total rules fetched: {len(rules)}')
# Ensure all rules are retrieved and sorted by rule number
rules = sorted(rules, key=lambda r: r.get("rule-number", 0))
print(f'Total rules fetched after sorting: {len(rules)}')
return rules
Here are few outputs when I play with the parameters like workers to parallel, limits and retries :
As you can see, I tried first with onlz 1 worker (no parallel then) and I retrieve almost all the time only 1 rule per request...... why is that ??
Current parameters:
1. Limit: 15
2. Max workers: 1
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 5
Offset = 0 retries = 20 fetched_rules = 7 current_rules = 0
Press enter to continue
Offset = 7 retries = 19 fetched_rules = 2 current_rules = 7
Press enter to continue
Offset = 9 retries = 18 fetched_rules = 2 current_rules = 9
Press enter to continue
Offset = 11 retries = 17 fetched_rules = 2 current_rules = 11
Press enter to continue
Offset = 13 retries = 16 fetched_rules = 1 current_rules = 13
Press enter to continue
Offset = 14 retries = 15 fetched_rules = 1 current_rules = 14
Press enter to continue
Press enter to continue 2
OUTSIDE Data len = 15
|----------------------------------------| 15 rules / 2566 CompleteOffset = 15 retries = 20 fetched_rules = 1 current_rules = 0
Press enter to continue
Offset = 16 retries = 19 fetched_rules = 1 current_rules = 1
Press enter to continue
Offset = 17 retries = 18 fetched_rules = 1 current_rules = 2
Press enter to continue
Offset = 18 retries = 17 fetched_rules = 1 current_rules = 3
Press enter to continue
Offset = 19 retries = 16 fetched_rules = 1 current_rules = 4
Press enter to continue
Offset = 20 retries = 15 fetched_rules = 1 current_rules = 5
Press enter to continue
Offset = 21 retries = 14 fetched_rules = 1 current_rules = 6
Press enter to continue
Offset = 22 retries = 13 fetched_rules = 1 current_rules = 7
Press enter to continue
Offset = 23 retries = 12 fetched_rules = 1 current_rules = 8
Press enter to continue
Offset = 24 retries = 11 fetched_rules = 1 current_rules = 9
Press enter to continue
Offset = 25 retries = 10 fetched_rules = 1 current_rules = 10
Press enter to continue
Offset = 26 retries = 9 fetched_rules = 1 current_rules = 11
Press enter to continue
Offset = 27 retries = 8 fetched_rules = 1 current_rules = 12
Press enter to continue
Offset = 28 retries = 7 fetched_rules = 1 current_rules = 13
Press enter to continue
Offset = 29 retries = 6 fetched_rules = 1 current_rules = 14
Press enter to continue
Press enter to continue 2
OUTSIDE Data len = 15
|----------------------------------------| 30 rules / 2566 CompleteOffset = 30 retries = 20 fetched_rules = 3 current_rules = 0
Press enter to continue
Offset = 33 retries = 19 fetched_rules = 3 current_rules = 3
Press enter to continue
Offset = 36 retries = 18 fetched_rules = 2 current_rules = 6
Press enter to continue
Offset = 38 retries = 17 fetched_rules = 1 current_rules = 8
Press enter to continue
Offset = 39 retries = 16 fetched_rules = 1 current_rules = 9
Press enter to continue
Offset = 40 retries = 15 fetched_rules = 1 current_rules = 10
Press enter to continue
Offset = 41 retries = 14 fetched_rules = 1 current_rules = 11
Press enter to continue
Offset = 42 retries = 13 fetched_rules = 1 current_rules = 12
Press enter to continue
Offset = 43 retries = 12 fetched_rules = 1 current_rules = 13
Press enter to continue
Offset = 44 retries = 11 fetched_rules = 1 current_rules = 14
Press enter to continue
Press enter to continue 2
OUTSIDE Data len = 15
|----------------------------------------| 45 rules / 2566 CompleteOffset = 45 retries = 20 fetched_rules = 1 current_rules = 0
Press enter to continue
Offset = 46 retries = 19 fetched_rules = 1 current_rules = 1
Press enter to continue
Offset = 47 retries = 18 fetched_rules = 1 current_rules = 2
Press enter to continue
Offset = 48 retries = 17 fetched_rules = 1 current_rules = 3
Press enter to continue
Offset = 49 retries = 16 fetched_rules = 1 current_rules = 4
Here some outputs with differents parameters and the time taken following the parameter used :
Current parameters:
1. Limit: 15
2. Max workers: 5
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 5
|████████████████████████████████████████| 2568 rules / 2566 Complete
ABBBBBBBBBBBBBBBTotal rules fetched: 2568
AAAAAAAATotal rules fetched: 2568
Total number of rules fetched: 2568
Time taken to fetch all rules: 217.33 seconds
Current parameters:
1. Limit: 15
2. Max workers: 5
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Current parameters:
1. Limit: 15
2. Max workers: 10
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit):
An error occurred: invalid literal for int() with base 10: ''
Current parameters:
1. Limit: 15
2. Max workers: 10
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 5
|████████████████████████████████████████| 2568 rules / 2566 Complete
ABBBBBBBBBBBBBBBTotal rules fetched: 2568
AAAAAAAATotal rules fetched: 2568
Total number of rules fetched: 2568
Time taken to fetch all rules: 197.03 seconds
Current parameters:
1. Limit: 15
2. Max workers: 10
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 2
Enter new max workers: 1
Current parameters:
1. Limit: 15
2. Max workers: 1
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 5
|████████████████████████████████████████| 2568 rules / 2566 Complete
ABBBBBBBBBBBBBBBTotal rules fetched: 2568
AAAAAAAATotal rules fetched: 2568
Total number of rules fetched: 2568
Time taken to fetch all rules: 434.68 seconds
Current parameters:
1. Limit: 15
2. Max workers: 1
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 2
Enter new max workers: 20
Current parameters:
1. Limit: 15
2. Max workers: 20
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 5
|████████████████████████████████████████| 2568 rules / 2566 Complete
ABBBBBBBBBBBBBBBTotal rules fetched: 2568
AAAAAAAATotal rules fetched: 2568
Total number of rules fetched: 2568
Time taken to fetch all rules: 605.79 seconds
Current parameters:
1. Limit: 15
2. Max workers: 1
3. Retries per offset: 20
4. Total rules expected: 2566
5. Fetch rules
6. Fetch rules 2
7. Exit
Choose a parameter to modify (or fetch/exit): 6
|████████████████████████████████████████| 2569 rules / 2566 Complete
Total number of rules fetched: 2569
Time taken to fetch all rules: 314.62 seconds
As we can see above, 1 worker is slower that 5 workers that do parallel calls to the api but 20 workers is way slower also.
Anyway, the real question is why do I retrieve only 1 rule most of the time from the API calls.
even when I do it without parallel calls, I only retrieves max 7 rules...... which sound really not much.
I had a look inside the api.elg log file but can't see issues tho.
@PhoneBoy , I had a look on the sk you sent but I'm running on R81.20 and sk seems outdated.
@Alex- Hmmm I'm thinking about maybe downloading the whole policy like every day and just retrieves from show changes the last revision of the day and compare with the actual policy to have the full one, but still not sure how the show changes works and how to do that.. but It looks a bit complicate..
@the_rock maybe the show package can help me to have the whole policy, I tested and It works, but also It takes some times to retrieves the whole policy..
What I want to do is : user put a string and the script will search all the rules that match in their names the string from the user and display the rule and the rule just before this one.
I just saw that It seems possible to put a filter in the command show policy package so maybe it would be faster to search directly for this rule yith the filter and then to do a second search to have the rule with ID minus 1 to have the rule just before.
