ClusterXL and port 263

[Don_Paterson]

I noticed that an R81.20 HA ClusterXL (main train) didn't use port 256 to sync after a member reboot. It was not used at all.

Instead port 263 was used as the only port along with port 8116.

Is that the full sync port?

https://support.checkpoint.com/results/sk/sk52421

Clustering
TCP	263	not predefined	Full Synchronization between Security Group Members in a Scalable Platform (by CXLD daemons over the internal network 192.0.2.0/24)

Protocol	Port number	Service Name and Comment	Usage
Firewall
TCP	256	FW1 - Check Point Security Gateway Service	Connections to Security Gateway Service (to FWD daemon): Fetching topology information by Security Gateway (by FWD daemon) from Security Management Server or Domain Management Server (CMA) Full Synchronization between ClusterXL members (by FWD daemons) IPS packet capture Used by the following commands: fw isp_link <server_ip> fw getver <server_ip> fw logswitch -h <server_ip> fw lslogs <server_ip> fw fetchlogs <server_ip>
Clustering
TCP	256	FW1 - Check Point Security Gateway Service	Full Synchronization between ClusterXL members (by FWD daemons)

[the_rock]

My R81.20 cluster member

[Expert@CP-FW-02:0]# netstat -anp | grep 256
tcp 0 0 0.0.0.0:256 0.0.0.0:* LISTEN 8232/fwd
tcp 0 0 0.0.0.0:42569 0.0.0.0:* LISTEN 8232/fwd
[Expert@CP-FW-02:0]# netstat -anp | grep 263
tcp 0 0 0.0.0.0:263 0.0.0.0:* LISTEN 8189/cxld
unix 3 [ ] STREAM CONNECTED 26362 7986/sxl_statd
unix 3 [ ] STREAM CONNECTED 26364 7986/sxl_statd
unix 3 [ ] STREAM CONNECTED 26360 7986/sxl_statd
unix 3 [ ] STREAM CONNECTED 26363 7986/sxl_statd
unix 3 [ ] STREAM CONNECTED 26361 7986/sxl_statd
[Expert@CP-FW-02:0]#

[Don_Paterson]

What if you do a tcpdump for host FW-01 (other member) -w and then reboot the other member. 

When it is back up then cpmonitor the capture file. 

?

[the_rock]

As a matter of fact, did that last night and it was exact same outcome.

Best regards,

Andy

[Don_Paterson]

Can you say ElasticXL   😉

Just a couple of reference to make this post more interesting:

https://support.checkpoint.com/results/sk/sk180742

https://support.checkpoint.com/results/sk/sk97638 

Security Gateway Software Blades and Features

Daemon	Section	Information
cxld	Description	Runs the cluster Full Sync (R81 and higher).
Path	$FWDIR/bin/cxld
Log file	$FWDIR/log/cxld.elg
To Stop	cpstop
To Start	cpstart
Debug	Runs with debug by default

[the_rock]

I cant say elasticXL, as I have no clue in the world what that even is lol

Andy

[Don_Paterson]

https://community.checkpoint.com/t5/General-Topics/R82-ElasticXL/m-p/194697

[the_rock]

Just looked it up before you gave the link...well, its definitely not applicable, since R82 is not even EA yet.

Andy

[Don_Paterson]

The port is used by the SP (Scalable Platform), as per sk52421.
Since TCP 256 was originally used and it seems there has been a change to 263 it may be that they are preparing for ElasticXL, or simply unifying sync mechanisms across platforms.

[_Val_]

R82 is on EA for over 3 weeks: https://community.checkpoint.com/t5/Product-Announcements/R82-EA-Program-Production/ba-p/198695

[Don_Paterson]

I did some debugs and I cannot see port 256 being part of the full sync.

I see a some evidence that 263 is the port used when the member wants to do a full sync.

[12516 3934414848]@FW-02[9 Dec 10:41:14] fwasync_get_maxbuf: maxbuf=4194304
fwsync: starting full sync with cluster member 10.10.10.2 (ctrl-c anytime to abort)

[12516 3934414848]@FW-02[9 Dec 10:41:14] fwsyncn_connect: server 10.10.10.2 alive 1

fwsyncn_connect: trying to connect to 10.10.10.2 with CXLD_PORT
[12516 3934414848]@FW-02[9 Dec 10:41:14] fwclient_do_connect_e: server 10.10.10.2 port 263 sicname CN=FW-01,O=xyz.abc.aic5ip cmd 57

[12516 3934414848]@FW-02[9 Dec 10:41:14] fwasync_conn_params_ex: fd: <24>, my addr: <10.10.10.3,61458>, peer addr: <10.10.10.2,263>

[Don_Paterson]

From sk180742, this is a simple way to see full sync port 263 is used.

tailf $FWDIR/log/cxld.elg | grep 263

[cxld 9315 3934513920]@FW-01[9 Dec 18:51:35] fwasync_conn_params_ex: fd: <36>, my addr: <10.10.10.2,263>, peer addr: <10.10.10.3,35059>
[cxld 9315 3934513920]@FW-01[9 Dec 18:51:35] PM_session_init: given session I(CN=FW-01,O=SMS.abc.com.aizyx;CN=FW-02,O=SMS.abc.com.aizyx;263;syncn).
[cxld 9315 3934513920]@FW-01[9 Dec 18:51:35] PM_policy_query: input session I(CN=FW-01,O=SMS.abc.com.aizyx;CN=FW-02,O=SMS.abc.com.aizyx;263;syncn).

with grep -i instance it covers more detail on "full sync".

Note:
I used sk37030 debug steps as part of stopping and starting sync (forcing full sync) and included the tailf cxld.elg into that procedure to capture the output above.

fw ctl setsync off

fw -d fullsync 10.10.10.2 2

[the_rock]

Yup, just did that tail command, nada, nothing

Andy

[Don_Paterson]

Try this in a separate session:

fw ctl setsync off

fw -d fullsync 10.10.10.2 2

I added note and sk reference to my post to cover that.

[the_rock]

Already done, no change. Only still see connection on port 256.

[_Val_]

Checking with the internal teams if the documentation needs to be updated. Will keep you posted.

[Don_Paterson]

Thanks Val.
I checked on an R80.40 cluster and there was no use of port 263, like in R81 and above. Only 256 was used.

Also, $FWDIR/log/cxld.elg did not exist on the R80.40 cluster member that I tested on.

[_Val_]

We know. We expect documentation to be updated soon.

[Timothy_Hall]

Right I believe cxld was added in R81 to get the full sync process out of the kernel and into process space (or perhaps take that responsibility away from fwd), similarly to the Firewall Worker Instances being pulled out of the kernel with User Space Firewall (USFW) in fwk processes.  Starting in R81.20 on the Quantum Lightspeed appliances only, even SecureXL is mostly located outside the kernel (called UPPAK).  Eventually UPPAK will be the default mode for SecureXL on all appliances, very curious to see what kind of performance hit the fastpath will take as a result.

[Martin_Raska]

I have the same, R81.20

fwasync_conn_params_ex: fd: <34>, my addr: <192.168.254.1,263>, peer addr: <192.168.254.2,56792>
[cxld 30169 3933345856]@chkp-demo-gw-1[2 Dec 15:40:09] fwsyncn_init: starting new sync client on port 263

[the_rock]

I am on R81.20 jumbo 41 in the lab, I dont have that problem.

Andy

[Martin_Raska]

I have 

HOTFIX_R81_20_JUMBO_HF_MAIN Take: 14

[Sergei_Shir · Accepted solution]

Hi

This is the short answer (the complete answer will be added to the relevant ClusterXL Admin Guides and relevant SK articles)

(1)

In R81 and higher versions (due to merge of the Scalable Platform code):

To perform a Full Sync with a peer cluster member, the CXLD daemon on a cluster member connects to the TCP port 263 on the peer cluster member

If this connection fails, then the cluster member falls back to the previous mechanism - the FWD daemon connects to the TCP port 256 on the peer cluster member

(2)

In R80.40 and lower versions:

To perform a Full Sync with a peer cluster member, the FWD daemon on a cluster member connects to the TCP port 256 on the peer cluster member