I need your help with regard to a complex issue that I have inherited and now I need to provide solutions for it.
We have two geographically separated sites (represented as SITE A and SITE B), and on both Sites, we are running Check Point Firewalls. Each Firewall is connected to a Router (ROUTER 1 - SITE A, and ROUTER 1 - SITE B). The two routers are then connected to other Routers in the user space through our MPLS network.
Below the Firewalls, there are another routers (ROUTER 2 - SITE A, and ROUTER 2 - SITE B), these two routers also know each other through our internal MPLS network.
Below the two routers (ROUTER 2 - SITE A, and ROUTER 2 - SITE B), there is our Datacenter where all our servers reside.
ROUTER 2 - SITE A has the default route pointing to FIREWAAL SITE A
ROUTER 2 - SITE B has the default route pointing to FIREWAAL SITE B
The issue that we are currently facing is that when our users (offices, Stores, etc) access our servers in the Datacenter, the traffic to the server can get in through SITE A, and the return traffic from the server can go back through the Firewall on SITE B on its way back to the User who requested it. Unfortunately, the traffic gets dropped because it did not get in from the Firewall in SITE B.
This issue is happens because Servers in our Datacenter have different default gateways (some have default gateway to SITE A, and others to SITE B)
I would like get your contribution based on your experience on possible solution to solve this problem. Your help will be appreciated.