In short - no, there is no limitation on the amount of the rules that a security policy can have. We can observe this in several aspects:
Using the GUI - R80 SmartConsole does not load all the rules in the policy but takes chunks of pages. This allows the user to browse a rulebase without reaching a memory limit.
Install Policy - the policy installation process compiles the policy to GW files. While the more rules you have the longer it will take to install the policy, every policy installation will eventually succeed. R80 brings an improvement to some environments, depending the capabilities of the Management server, by utilizing more of the RAM and cores during policy installation.
Networking - rulebase performance is affected more with broken acceleration templates based on specific capabilities of some rules (time objects, service with resource, etc.) and less with the size of the policy. While the size does introduce a performance impact, it is negligible comparing to the content of actual rules and their placement in policies.
Ease of management - this is where the size of a security policy could matter. The larger your rulebase, the less convenient it will be to organize it and keep its sections structure. Pending R80.10 Gateways, you can prepare your policy for easier management by splitting rulebases to inline and ordered layers, and as a result allow reusable chunks of rules, and control the permission profiles within your policy.
Hope this helps
