As previously we using L3 Switch (Name: POC L3) where 4 interface which comes from two internal switches (ABC and XYZ Switch) is terminated with upstream L3 Switch (Name: POC L3) internal VLAN interface.
➡️ Two internal switches (ABC and XYZ) both having one bond interface with mode LACP.
➡️ On Checkpoint external interface also having a bond interface with network 192.168.100.0/22.
➡️ Our requirement is to place Checkpoint Firewall (23500) in between internal switch (ABC and XYZ) and L3 Switch (Name: POC L3).
We face below challenges
As per our requirement, we can not add two bonds on one VLAN, If we added then it will be creating two networks, for example, bond1.100, bond2.100 with respect to VLAN 100.
➡️ So we added 4 interfaces to one BOND as bond1 with mode LACP and then create a VLAN and added that bond1 but we face the issue because all interface is active because of mode LACP.
➡️ So on this scenario, If we remove the cable from any one internal switch, for example, ABC switch then XYZ switch working fine, we able to access server as well ssh from LAN/WAN.
As the first scenario does not work as expected so we change the configuration.
➡️ We create a two BOND, bond 1 and bond 2 with respect to the internal switch and then bridge between bond 1 and bond 2 so in this scenario, we only assign one IP address on the bridge interface as 172.16.100.1/22 which meet our requirement.
So in this scenario we able to ping internal server that behind ABC and XYZ switch.
➡️ we check the logs found that the packet drop on Checkpoint internal bond interface, showing error "missing OS route".
➡️ Base on the logs I suspect as the ARP cache is overflowing. The most possible reason - too much traffic on the network (generated by some application, by some hosts, or by related factors). (sk108587 )
➡️ By default, the arp cache size is 4096 ARP cache slot so maximum we can increase up to 4096*4=16384 arp cache slot.
1. If we use two Checkpoint device and give cluster VIP address as 172.16.100.1/22 then we will not face challenges with bond interface configuration.
2. If we place the checkpoint gateway above the POC L3 Switch then we not face challenge related to bond.
3. We need to increase the arp cache size up to 16386 arp cache slot before that we need to check the below command to verify whether arp entry will increase or not after live the checkpoint gateway.
arp -an | wc -l (to check the line on arp cache size)
Check the message file as "neighbour table overflow"
command : dmesg | grep -i “table overflow”
From MAX POWER Book : One interesting side effect of not having a private transit network between the internal interface of the firewall and your internal core router is that the firewall will have to maintain far more IP address to MAC address mappings in its ARP cache than it otherwise would. This is especially true if there are hundreds or thousands of workstations/servers located on VLANs directly attached to the firewall’s various interfaces.
Such a large network will, of course, have far more broadcast traffic flooded to all stations on the network including the firewall, resulting in reduced overall network
However, one situation that can occur in this case will severely damage the firewall’s performance (almost to the point of looking like rolling outages in some cases): an
overflow of the firewall’s ARP cache.
Please suggest us what about the possible reason for getting the error "missing OS route".
NOTE: Message file is overwritten so we unable to find "neighbor table overflow" also we do not have any output or arp cache size during the issue.
Now we again plane to place the Checkpoint with the proper plane so need all your help.