Intermittent Internet with using OSPF and two Cisco Router

[CharlesLZ]

Hello Check Point Team

I am currently facing an intermittent Internet scenario when Check Point 1600 Spark is connected with 2 Routers and using OSPF. When I disconnect one Interface (LAN 5) from Check Point apparently everything is stable. I need to check with you if you have faced with this kind of scenario with Dynamic Routing and how the firewall could handle this scenarios with 2 Routers with OSPF and intermittency.

• The Security Rule is Accept
• For the server that I made a test, I ran a zdebug but no drops were found.
• Statefull Inspection is disabled on SmartConsole for testing 
• 2 Routers, the SMB Spark 1600 and a SwitchCore are part of the OSPF Area 0

Check Point has this interface configs for cost and the Routers does not have Cost Configuration. What takes the decision for trafic is Check Point SMB Spark1600

LAN5 Cost 30

LAN6 Cost 20

So right now LAN6 is sending traffic to the Router 2

No Priority OSPF configuration is set right now, just Cost Configuration

I will attach an image.

I will appreciate your help

Have a nice day !

[Chris_Atkinson]

Which software version/build is the gateway deployed with and could you share how the NAT is configured ?

Also in the device section under advanced settings, how is the "Multiple ISP Route Refresh" option set out of interest?

[CharlesLZ]

Hello Chris

The SMB is running R80.20.50 (992002773), we are currently not using a NAT, the NAT to Internet is performed by the SDWAN Routers. 

I have check the "Multiple ISP Route Refresh" and it is now set as false, is this right?

[Chris_Atkinson]

You can try testing this configuration, for more information please refer: sk167433

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[CharlesLZ]

Thanks Chris

The Checkpoint does no have Internet interfaces configured, there are all LAN Interfaces with OSPF, the sk should work too. There is no way to have an Internet Interface to be part of OSPF 

I will check the SK, I have a question, based on the PNG image, have you seen a similar scenario or your colleagues?. Which other SK could you share with us to try ?

[Chris_Atkinson]

Are the OSPF neighbourships stable throughout or you see them flapping?

Side note are either the Cisco or Check Point configured to perform MSS clamping in case one or more paths has a lower MTU than standard?

[the_rock]

Can you verify there are no any significant drops on that eth5 interface? Try running ifconfig -a eth5 and ethtool -S eth5 or LAN5 (if thats the actual name)

[Sorin_Gogean]

Hello @CharlesLZ ,

So the end-users are complaining of Internet access issues, and you supposed it's CKP GW issue ?

From the two routers, how is the Internet access looking ? Do you see in any at the routers OSPF flapping or going down at the time end-users complain ? 

About the "Internet not working" complain, is that for specific sites or ? 

Also what I'm missing, is what routes are the R1 and R2 learn from the ISP, or how you configured the balance between those 2 and multiple ISP's ?

From R1 and R2 what routes are you sending over OSPF ? 

You show/say that if R2 is the ONLY active one, all is good, what about dropping the R2 to CKP connection and leaving the R1 as active one. Like forcing all the traffic over R1, does that work without issues ?

I think you might have an issue at the R1 and R2 level, and I'm missing details but I don't think it's an OSPF issue. 
OSPF is between the Routers (R1 and R2) and the CKP GW, therefore I doubt it that the paths are flapping like crazy so it will behave like that, or you have a cable issue 😊? 

Ty and let us know more details,

[CharlesLZ]

Hello Sorin

The enduser experience Teams disconnections sometimes and they do a Speedtest and it appear different updtream and downstreams, when we disconnect LAN5 on Check Point we are now seeing upstreams and downstreams good. The customer needs R1 and R2 Routers are for SDWAN and High Availability (HA) , also  each Cisco router has 3 ISP links; on our side, the Check Point side, LAN6 has a lower cost and it takes as preferred  path, so the default route and customer's routes are being known by LAN6. If we check the CheckPoint's routing table all the relevant routes and the default route are known through LAN6.

The Cisco provider are telling us that the Routers they are configured as Active Active, we don't know what this means and as a SDWAN solution if this is the best way to configure, the thing that we are seeing is that if Check Point has only 1 Interface to a SDWAN Router this works good and since last week Wednesday the end users they have not reported any issue with their Internet.

Right now I am doing a research and I'm thinking to try sk167433 and disable SecureXL in the next session, but this a particular scenario, I need to adjust my troubleshoot to verify packets if are being lost or if it's a OSPF issue, an Acceleration Issue or a SDWAN issue.

I also thinking to do a tcpdump with -e flag to confirm that no packets are leaving with the LAN5 address, to discard that LAN5 is not being used for traffic.

Best Regards

[the_rock]

Not a bad idea to disable sxl for testing. You can also try below if you wish to do it for specific ports/ IPs

https://support.checkpoint.com/results/sk/sk104468

Also, you can do fw monitor -F command

fw monitor -F "srcip,srcport,dstip,dstport,protocol" -F "same just the other way around"

example, say src ip is 1.1.1.1 and dst is 2.2.2.2 and dst port is 444, it would looks like this

fw monitor -F "1.1.1.1,0,2.2.2.2,444,0" -F "2.2.2.2,0,1.1.1.1,444,0"

Good site my colleague made long time ago to help with running captures on different platforms:

https://tcpdump101.com/#

Cheers mate.

[Sorin_Gogean]

Hello @CharlesLZ ,

I really don't see an CheckPoint issue, since you have the R1 and R2 as active-active, it means that you're using all 6 Internet connections towards internet (still we're missing the part on how those links are utilized).

Anyway, like I said, I don't see an CKP issue here, since it's sounds more like an routing issue.

SO if your enduser goes to Internet (Teams and browsing) there is a NAT happening on the CKP level - I/we assume - so the public IP from CKP is going Lan6 then SDWan R2 and exits through one of those 3 ISPs, but as you say, the RETURN traffic comes through R1 SDWan through one of those 3 ISPs and then Lan5 and then CKP and back to the enduser. 

There you have the different upstream/downstream path like you say.  And that would be expected if the public IPs announced to the ISPs would prefer the return traffic through some internet uplinks than others.

Now if you don't do NAT on CKP level, and NAT is done on the SDWan level, and this asynchronous routing is still happening, half of mu above statement still remains and that is WRONG set-up on SDWan and ISP side.

You still didn't answer what happens if you drop on purpose the Lan6 and stick with Lan5 so traffic is routed through R1.....
Also you didn't told us if when issues are seen, if the OSPF is flapping or not, of if all is normal, routes are stable since hours/days, but you have this different upstream/downstream issue, then it's most likely that your return traffic is prefering an ISP from R1 box than the IPS from R2 box through where it got out to Internet.

How I see it, is that you need to tell up to where are the Public IP's and where are private IPs so we know how NAT is happening, and also I would really not prefer in OSPF an path more than the other.... Not sure why would you do that since your intention is to use all 6 Internet connections in same way - or almost in same percentage.

Without CKP, if you set a client on the same HSRP network and tell it to address the HSRP as GW, how is that client/traffic performing ? This would show if you have a problem at SDWan level and you're looking WRONG at CKP level.

Thank you,