ElasticXL Traffic Distribution question

[marcyn]

Hi ChackMates,

There is one question that bothers me ... how does exactly work traffic distribution in ElasticXL ?
And I'm not talking about that we have General Mode in ElasticXL.

I'm talking about how does it work in ... general 😉
Let's suppose that we have 2 gateway nodes ... one is obviously SMO.
Each and every traffic first goes to SMO (as it works like pivot in classic ClusterXL Unicast Mode).
Then SMO decides that it will consume this traffic or maybe this 2nd node will do that.

This is a theory ... but how exactly does it work ?

1) LAN Client sends traffic to Internet
2) Traffic goes from Client to switch (this one where LAN segment is connected to) then to SMO ... then SMO decides that 2nd node should process this traffic
3) Traffic goes back to the same switch and then to port on that switch where 2nd node is connected

Does it work like that ?
I don't believe that in case SMO decides that 2nd node should process this traffic, then this traffic will be send via sync interface .... because it will have no sense at all and it could be a huge bottleneck.

Is it works as I described ... how exactly does it work since we only have one MAC in ElasticXL cluster ... so 2nd node does not have it's own MAC ... so how does this switch will know that traffic that will go back from SMO (let's say port1 on this switch) should go to 2nd node (let's say port2 on this switch) ?

Is there any technical documentation about this process that describes this traffic distribution a little bit more deaply ?

--
BR
m.

[genisis__]

I suspect it works like Nokia IPSO clustering.

- Packet comes in to the Master Node

- Master node makes a decision based on its load to send to slave node.

- Master node will then issue a gratuitous arp indicating the traffic should be sent to the slave.

[marcyn]

AFAIK in ElasticXL we have "one MAC address" ... so GARP will not change a thing.
Anyway ... my question is here - do we have some more detailed information regarding this ... how this traffic flow exactly works ?

Maybe a little example:
1) ETH3 interface on SMO (pivot) and on the 2nd node (let's assume for simplicity that we have a 2-node ElasticXL) .... is connected to the same switch (e.g. port1 and port2 respectively).
2) So if traffic directed to ElasticXL (to ETH3 interface) reaches this switch ... then the pivot must tell this switch - I'd like this traffic to port1 (read: the 2nd node is silent when the network asks "who has this MAC address" .... only SMO "it's me" responds to this) - and this makes it clear why the switch will ALWAYS send traffic to port1 (to SMO = pivot).
3) What I wonder is how it works later... when SMO says "I will not process this traffic, the 2nd node should handle this traffic"... how is the switch supposed to know that what came back to it from SMO on port1 should be forwarded to the 2nd node on port2..... in such a case the 2nd node must somehow "advertise itself on the network with its MAC"... This is the aspect that interests me... i.e. how does it technically work that the switch knows that the traffic should be put on port2 (as we don't have multiple MACs here).

Above assumes that it actually works that traffic from SMO will go back to a switch in case pivot will decide that 2nd should process this traffic - if this assumption is wrong, please correct me 🙂

--
Best
m.

[genisis__]

Guess we need someone who know ElasticXL in detail to respond.  In the scenrio above, how about a port mirror and tcpdump it, to see what's going on at a packet level?

[_Val_]

No, this is only partially correct. The Pivot/SMo receives all incoming traffic and forwards most of it via the same production interface to other cluster members, with forward_mac_magic

No GARPS are used

[matanbe_chkpcp]

Hello,

You got most of it right, but the only missing piece is that in ElasticXL, each interface on each member has a different MAC address.
So when the SMO decides to distribute the packet to the second member, it changes the destination MAC address to match that of the target member (using the same interface the packet arrived on, not via Sync).

Only the SMO sends GARPs, so all traffic arrives at the SMO and is then distributed as needed.

I hope that makes things clearer.

Thank you,
Matan

[marcyn]

Hi @matanbe_chkpcp

Thank you for the reply.
So it works like that:

1) client send a packed on ETH3, in case it doesn't know MAC there is arp-who-has message and only SMO replies to this broadcast message - quite an obvious .... because of that packet arrives on SMO ETH3
2) so in client arp table there is MAC of SMO - again quite an obvious - because of that on client side there is only MAC on SMO (client doesn't see MAC of anuther member of ElasticXL)
3) next SMO decides that it will not process this traffic, but 2nd node should do it

--- to that point - all clear, the next step is what I'm asking about 😉 ---

4) SMO knows that it has 00:00:00:00:00:11 on ETH3 and 2nd node has 00:00:00:00:00:22 on ETH3 .... so in case 2nd node should process this PARTICULAR traffic it should go to 00:00:00:00:00:22 ... so what exactly SMO does here ?
Does it sends RST to this PARTICULAR traffic, and GARP that will tell switch and all sorroundings that now this IP is not at 00:00:00:00:00:11 but on 00:00:00:00:00:22 ... and then client after receiving RST will repeat and now it will go to 2nd node ?
If so ... what about if another packet would like to go on this ETH3 interface at exactly the same time ... in that case it should go to 00:00:00:00:00:22 which is not SMO ... but as we know all packets should first go to SMO ...
I just want to undertand this process more deeply.
As for now I undestood that only this PARTICULAR packet that SMO decides that it should go to 2nd node should go there ... nothing else ... so that each and every other packets first go to SMO for a decision.

@matanbe_chkpcp can you describe this process more deeply ... based on the above note from my side ?

--
BR
m.

[Bob_Zimmerman]

To confirm, the members definitely have unique MACs on their interfaces:

[Expert@DallasticXL-s01-01:0]# cphaprob state | grep Mode
Cluster Mode:   HA Over LS

[Expert@DallasticXL-s01-01:0]# g_all 'ifconfig bond2.100 | grep HWaddr'
1_01:
bond2.100   Link encap:Ethernet  HWaddr 00:1C:7F:81:02:FC
1_02:
bond2.100   Link encap:Ethernet  HWaddr 00:1C:7F:81:42:FC

When the SMO receives a frame it decides should be processed by another member, it will just retransmit the frame with the destination set to the selected member's MAC. The attached switches then forward it to that member.

Note that this consumes extra throughput slots on the traffic-bearing interfaces.

[marcyn]

Hi @Bob_Zimmerman 

Yes, members have different MAC, sure.

What I meant before, when I wrote about one MAC was a bit of ... shortcut 🙂

I meant that clients only see one MAC (this one from SMO).

And @_Val_ already answered my question ... it's based on MFF mechanism, now it's clear. SMO just send traffic on L2 directly to chosen member.

Thanks

--

BR, m.

[_Val_ · Accepted solution]

ElasticXL is using the classic ClusterXL Pivot mode, also know as Load-Sharing Unicast Mode to handle the traffic. SMO is a pivot in this case.

More info about Unicast mode can be found in the ClusterXL admin guide.

In a nutshell, traffic is received by the pivot/SMO and mostly forwarded to other cluster members for processing, through the same production interface it is received and not through the sync network. Magic Forwarding MAC is used for the forwarding. The member who received a forwarded packet will then set it directly to the next hop after filtering.

[marcyn]

Hi @_Val_ 
Ok, thank you for this info.

So it's based on MFF (MAC-Forced Forwarding) ... and works the same way as in ClusterXL Unicast Mode.
Until now all I heard was like "it works similiar to ClusterXL Unicast Mode" ... but it looks like that at least in this aspect it works the same way.

Thanks.

--
BR
m.

[_Val_]

Both statements are correct in a specific context.

When it comes to forwarding traffic, ElasticXL is working the same way as ClusterXL Unicast.

However, cluster management, member discovery, and provisioning are completely different from CusterXL. Same, same, but different 🙂

[marcyn]

Hi @_Val_ 

Yes, of course - I was writing about traffic distribution 🙂
What makes ElasticXL so special is what you wrote in your last sentence.

--
BR
m.

[_Val_]

I also need to add, with just two cluster members in a dual-site setup with ElasticXL you can have an HA pair, then you do not have to worry about forwarding traffic. That is impossible to achieve with the classic Unicast mode

[genisis__]

Are there any considerations to have two sets of clusters in the same networks (excluding SYNC)? 

[marcyn]

You can't have more then one ElasticXL cluster in the same sync ..  but in case any other network there should be no issues.

Another question would be if that makes a sense... maybe, I will not judge that 🙂

Of course they would need to have different IPs ... so one cluster would be gateway for some part of such a network and another for different machines ... what for ... but as I wrote I will not judge this idea 🙂

--

BR, m.

[_Val_]

Not sure I fully understand the question. If you are asking about two different ElasticXL clusters in the same broadcast domain, there is no issue with that, due to encrypted CCP, same as with classic ClusterXL.

[genisis__]

That's pretty much what I expected Val.  I don't know why but I think I read somewhere encrypted CCP is not enabled when using ElasticXL and VSNext, maybe I'm dreaming it up!

[marcyn]

I believe you are talking about that traffic inside sync is in clear-text. That's why in one sync broadcast domain there can be only one ElasticXL.

But cluster traffic inside other interfaces should be encrypted ... maybe some employee can confirm that to be perfectly sure ?

--

BR, m.

[emmap]

Any Check Point cluster needs to have an isolated link for their Sync traffic, that's not special to EXL. 

[genisis__]

Thanks - That sounds like what I read.  So basically SYNC (BOND) must be in a unique VLAN per cluster.  All other interfaces can co-exists within the same broadcast domain without causing conflict.

[Bob_Zimmerman]

ElasticXL uses VMACs, though. There's minor concern if you connect eth1-01 on two clusters to a single broadcast domain, they could potentially pick the same VMACs for the members.

Incidentally, it looks like eth1 and eth1-01 (and eth2/eth1-02, eth3/eth1-03, ...) may get the same VMAC. That could cause some "fun" issues if you give the interfaces to different VSs, then you want them to talk over them.

[genisis__]

Oh what joy!

I would hope Checkpoint resolves this ASAP.

[Bob_Zimmerman]

I can't be 100% sure there is an actual issue, since my ElasticXL members only have onboard interfaces. I don't have a box to try with both onboard interfaces and a card. The VMAC assigned to my eth2 decodes as belonging to eth1-02, the VMAC assigned to eth3 decodes as belonging to eth1-03, and so on.

[Mike_Jensen]

Does this introduce the possibility of latency or delay for sessions being handled by the non SMO since this extra step is required?

[PhoneBoy]

Since the SMO has to receive the traffic first before forwarding to the other member, yes.