maestro vsx pnotes

wsitu — Tue, 06 Feb 2024 10:34:21 GMT

Hi experts,

we have the following configuration and experiencing policy installation failure consistently. please advise how to proceed with troubleshooting and/or solution. thank you

dual orchestrator mho-140 with 4 sgm. configured with 5 virtual systems. r81.20 jumbo 26

#/var/log/messages:

Feb 6 04:18:32 2024 mho-ch01-01 spike_detective: spike info: type: thread, thread id: 55092, thread name: fw_full, start time: 06/02/24 01:18:25, spike duration (sec): 6, initial cpu usage: 99, average cpu usage: 99, perf taken: 0

Feb 6 04:18:38 2024 mho-ch01-01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fw_full, start time: 06/02/24 01:18:25, spike duration (sec): 12, initial cpu usage: 99, average cpu usage: 97, perf taken: 1

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-111500-1_01: State change: ACTIVE -> DOWN | Reason: VSX PNOTE due to problem in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 kernel:[fw4_0];fwkdrv_process_vsx_global_data_in_kernel: Updating smo task to 1
Feb 6 04:18:38 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xac1f514 due to failure to read from the socket
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeeb74 due to failure to read from the socket
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xac1f514 due to failure to read from the socket
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeeb74 due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeeb74 due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. errno = 104 (Connection reset by peer)
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeec7c due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. errno = 104 (Connection reset by peer)
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xac38a24 due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 1

Feb 6 04:18:39 2024 mho-ch01-01 sgm_pmd: SGM has left monitoring task
Feb 6 04:18:40 2024 mho-ch01-01 kernel:[fw4_0];fwkdrv_process_vsx_global_data_in_kernel: Updating smo task to 2
Feb 6 04:18:40 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:18:40 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:18:40 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:42 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:18:42 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:18:42 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:45 2024 mho-ch01-01 fw: SMOApiExecHaDoHookCmd: Creating /opt/CPsuite-R81.20/fw1/CTX/CTX00005/conf/fw_updated
Feb 6 04:18:46 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 1

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 3

Feb 6 04:18:48 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 3

Feb 6 04:18:52 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Aligning the local GW LACP bonds to chassis monitor GW LACP bonds
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond4 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond4 First slave System MAC address 00:1c:7f:81:08:01 slaves {'eth2-08': '2', 'eth1-08': '1'}, local GW: VS 0 bond4 First slave System MAC address 00:1c:7f:81:08:01 slaves {'eth2-08': '2', 'eth1-08': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond5 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond5 First slave System MAC address 00:1c:7f:81:09:01 slaves {'eth2-09': '2', 'eth1-09': '1'}, local GW: VS 0 bond5 First slave System MAC address 00:1c:7f:81:09:01 slaves {'eth2-09': '2', 'eth1-09': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond6 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond6 First slave System MAC address 00:1c:7f:81:0a:01 slaves {'eth2-10': '2', 'eth1-10': '1'}, local GW: VS 0 bond6 First slave System MAC address 00:1c:7f:81:0a:01 slaves {'eth2-10': '2', 'eth1-10': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond7 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond7 First slave System MAC address 00:1c:7f:81:0c:01 slaves {'eth2-12': '2', 'eth1-12': '1'}, local GW: VS 0 bond7 First slave System MAC address 00:1c:7f:81:0c:01 slaves {'eth2-12': '2', 'eth1-12': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond1 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond1 First slave System MAC address 00:1c:7f:81:05:01 slaves {'eth1-05': '1', 'eth2-05': '2'}, local GW: VS 0 bond1 First slave System MAC address 00:1c:7f:81:05:01 slaves {'eth1-05': '1', 'eth2-05': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond2 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond2 First slave System MAC address 00:1c:7f:81:06:01 slaves {'eth1-06': '1', 'eth2-06': '2'}, local GW: VS 0 bond2 First slave System MAC address 00:1c:7f:81:06:01 slaves {'eth1-06': '1', 'eth2-06': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond3 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond3 First slave System MAC address 00:1c:7f:81:07:01 slaves {'eth1-07': '1', 'eth2-07': '2'}, local GW: VS 0 bond3 First slave System MAC address 00:1c:7f:81:07:01 slaves {'eth1-07': '1', 'eth2-07': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond8 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond8 First slave System MAC address 00:1c:7f:81:31:01 slaves {'eth1-49': '1', 'eth2-49': '2'}, local GW: VS 0 bond8 First slave System MAC address 00:1c:7f:81:31:01 slaves {'eth1-49': '1', 'eth2-49': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond9 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond9 First slave System MAC address 00:1c:7f:81:35:01 slaves {'eth1-53': '1', 'eth2-53': '2'}, local GW: VS 0 bond9 First slave System MAC address 00:1c:7f:81:35:01 slaves {'eth1-53': '1', 'eth2-53': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond10 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond10 First slave System MAC address 00:1c:7f:81:37:01 slaves {'eth1-55': '1', 'eth2-55': '2'}, local GW: VS 0 bond10 First slave System MAC address 00:1c:7f:81:37:01 slaves {'eth1-55': '1', 'eth2-55': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond11 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond11 First slave System MAC address 00:1c:7f:81:0e:01 slaves {'eth2-14': '2', 'eth1-14': '1'}, local GW: VS 0 bond11 First slave System MAC address 00:1c:7f:81:0e:01 slaves {'eth2-14': '2', 'eth1-14': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Operation was completed
Feb 6 04:18:53 2024 mho-ch01-01 lbd: align_the_local_LACP_bonds_to_chassis_monitor_bonds result, output: 'Operation was completed
', exit code: 0
Feb 6 04:18:54 2024 mho-ch01-01 lbd: Report LACP_SYNC pnote OK, all LACP bonds were successfully verified
Feb 6 04:19:01 2024 mho-ch01-01 confd: setrlimit(RLIMIT_NOFILE, 512): Operation not permitted
Feb 6 04:19:01 2024 mho-ch01-01 xpand[66780]: Performing database cloning
Feb 6 04:19:01 2024 mho-ch01-01 xpand[66780]: load_config>(/config/db/cloned_db,empty)
Tue Feb 06 01:19:03 PST 2024: [asg_update_diag_config]: merging asg_diag_config from SMO with latest asg_diag_config
Feb 6 04:19:13 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:13 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-112004-1_01: State change: DOWN -> ACTIVE | Reason: USER DEFINED PNOTE

Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-115704-1_01: State change: DOWN -> ACTIVE | Reason: Member state has been changed due to issue in Virtual System 0

Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:18 2024 mho-ch01-01 kernel:[fw4_0];fwkdrv_process_vsx_global_data_in_kernel: Updating smo task to 0
Feb 6 04:19:18 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:18 2024 mho-ch01-01 sgm_pmd: SGM became port monitoring task
Feb 6 04:19:18 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:18 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:18 2024 mho-ch01-01 asg_send_alert: [1_1] SGM 1 in Chassis ID 1 state changed to UP
Feb 6 04:19:18 2024 mho-ch01-01 sgm_pmd: Updating Orchestrators matrix...
Feb 6 04:19:18 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:18 2024 mho-ch01-01 asg_send_alert: [1_3] SGM 3 in Chassis ID 1 state changed to DOWN.
Feb 6 04:19:19 2024 mho-ch01-01 lbd: Local member turned to "blade_for_task".
Feb 6 04:19:19 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:19 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0x9
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-Mgmt1 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-10 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-12 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-14 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-49 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-05 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-53 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-55 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-06 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-07 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-08 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 asg_send_alert: [1_2] SGM 2 in Chassis ID 1 state changed to UP
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-09 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting eth1-Sync cluster link state to down(current cluster state is up)
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Running: cphaprob link_state set local eth1-Sync -d full -s 10000M down
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-Mgmt1 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-10 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-12 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-14 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-49 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-05 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-53 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-55 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-06 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-07 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-08 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-09 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting eth2-Sync cluster link state to down(current cluster state is up)
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Running: cphaprob link_state set local eth2-Sync -d full -s 10000M down
Feb 6 04:19:20 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:20 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:21 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:21 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xb
Feb 6 04:19:22 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:23 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:23 2024 mho-ch01-01 sgm_pmd: Running distutil update...
Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: Re-generating /etc/groups
Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: /etc/groups: all 1_01,1_02,1_03,1_04

Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: /etc/groups: chassis1 1_01,1_02,1_03,1_04

Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: /etc/groups: chassis_active 1_01,1_02
Feb 6 04:19:34 2024 mho-ch01-01 PAM-tacplus[5407]: auth failed: 2
Feb 6 04:19:38 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:38 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:39 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:39 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xb
Feb 6 04:19:40 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:40 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:43 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:43 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:43 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:43 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xb
Feb 6 04:19:44 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:45 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:45 2024 mho-ch01-01 expert: SSH connection by admin user to Expert Shell with client IP 10.1.1.1 at 01:19 02/06/2024
Feb 6 04:19:58 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:58 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:58 2024 mho-ch01-01 asg_send_alert: [1_3] SGM 3 in Chassis ID 1 state changed to UP
Feb 6 04:19:59 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:59 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xf
Feb 6 04:20:00 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:20:00 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.

-----------------------------------------------------------------------------------------------------------------------------------------------
#fwk.elg

[6 Feb 4:18:38][fw4_0];[vs_0];CLUS-120103-1_01: VSX PNOTE ON
[6 Feb 4:18:38][fw4_0];[vs_0];CLUS-111500-1_01: State change: ACTIVE -> DOWN | Reason: VSX PNOTE due to problem in Virtual System 5
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: ACTIVE ---> DOWN (cluster state changed to: DOWN)
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: [SMO]: Current blade (1_1) is no longer SMO Mgmt Master (on 99964150)
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: fwha_update_member_running_dr_task: Changed DR manager from member (1_01) to (1_02)
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: fwha_update_member_running_smo_task: Changed fwha_smo_member from member (1_01) to (1_02)
[6 Feb 4:18:39][fw4_0];[vs_0];CLUS-100101-1_01: Failover member 1_01 | Reason: VSX PNOTE
[6 Feb 4:18:40][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_2's state: ACTIVE ---> DOWN (State: ACTIVE -> DOWN.)
[6 Feb 4:18:40][fw4_0];[vs_0];FW-1: fwha_update_member_running_dr_task: Changed DR manager from member (1_02) to (1_03)
[6 Feb 4:18:40][fw4_0];[vs_0];FW-1: fwha_update_member_running_smo_task: Changed fwha_smo_member from member (1_02) to (1_03)
[6 Feb 4:18:40][fw4_0];[vs_0];CLUS-211500-1_01: Remote member 1_02 (state ACTIVE -> DOWN) | Reason: VSX PNOTE
[6 Feb 4:18:40][fw4_0];[vs_0];CLUS-100202-1_01: Failover member 1_02 | Reason: Available on member 1_02
[6 Feb 4:18:42][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_4's state: ACTIVE ---> DOWN (State: ACTIVE -> DOWN.)
[6 Feb 4:18:42][fw4_0];[vs_0];CLUS-211500-1_01: Remote member 1_04 (state ACTIVE -> DOWN) | Reason: VSX PNOTE
[6 Feb 4:18:42][fw4_0];[vs_0];CLUS-100404-1_01: Failover member 1_04 | Reason: Available on member 1_04
[6 Feb 4:18:43][fw4_0];[vs_0];CLUS-211505-1_01: Remote member 1_03 (state ACTIVE -> ACTIVE(!)) | Reason: VSX PNOTE
[6 Feb 4:18:48][fw4_0];[vs_0];CLUS-214904-1_01: Remote member 1_03 (state ACTIVE(!) -> ACTIVE) | Reason: Reason for ACTIVE! alert has been resolved
[6 Feb 4:18:52][fw4_0];[vs_0];CLUS-120103-1_01: VSX PNOTE OFF
[6 Feb 4:18:52][fw4_0];[vs_0];report_lacp_sync_pnote_problem: registering LACP_SYNC pnote with problem
[6 Feb 4:18:52][fw4_0];[vs_0];CLUS-120109-1_01: LACP_SYNC PNOTE ON
[6 Feb 4:18:52][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: READY ---> DOWN (cluster state changed to: DOWN)
[6 Feb 4:18:54][fw4_0];[vs_0];CLUS-120109-1_01: LACP_SYNC PNOTE OFF
[6 Feb 4:18:54][fw4_0];[vs_0];fwha_ch_should_pull_config: registering pull_config pnote with problem
[6 Feb 4:18:54][fw4_0];[vs_0];CLUS-120109-1_01: pull_config PNOTE ON
[6 Feb 4:18:54][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: READY ---> DOWN (cluster state changed to: DOWN)
[6 Feb 4:18:54][fw4_0];[vs_0];fwha_ch_should_pull_config: pulling configuration for local SGM
[6 Feb 4:18:56][fw4_0];[vs_0];fwha_mbs_pull_config_start_notify_cb: pull config was initiated
[6 Feb 4:18:56][fw4_0];[vs_0];fwha_mbs_pull_config_start_notify_cb: waiting for all VSs pull config to be done before going up
[6 Feb 4:19:12][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_4's state: DOWN ---> READY (State: DOWN -> READY.)
[6 Feb 4:19:12][fw4_0];[vs_0];CLUS-216703-1_01: Remote member 1_04 (state DOWN -> ACTIVE*) | Reason: Trying to move to ACTIVE state
[6 Feb 4:19:12][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_4's state: READY ---> ACTIVE (State: READY -> ACTIVE.)
[6 Feb 4:19:12][fw4_0];[vs_0];CLUS-212004-1_01: Remote member 1_04 (state ACTIVE* -> ACTIVE) | Reason: USER DEFINED PNOTE
[6 Feb 4:19:17][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 1 finished pull config
[6 Feb 4:19:17][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 2 finished pull config
[6 Feb 4:19:17][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 3 finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 5 finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 4 finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: all VSs finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: pull config is done
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: fwha_mbs_pull_config_done = 1, removing pnote
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: unregistering pull_config pnote
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-120109-1_01: pull_config PNOTE OFF
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: fwha_mbs_pull_config_done = 1, removing pnote
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: unregistering pull_config pnote
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: DOWN ---> READY (cluster state changed to: READY)
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-112004-1_01: State change: DOWN -> ACTIVE | Reason: USER DEFINED PNOTE
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: READY ---> ACTIVE (cluster state changed to: ACTIVE)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [SMO]: Current blade (1_1) is now SMO Mgmt Master (on 99964543)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 122 -> 128(param_change 1)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_update_member_running_smo_task: Changed fwha_smo_member from member (1_03) to (1_01)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_update_member_running_dr_task: Changed DR manager from member (1_03) to (1_01)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_3's state: ACTIVE ---> DOWN (State: ACTIVE -> DOWN.)
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-211500-1_01: Remote member 1_03 (state ACTIVE -> DOWN) | Reason: VSX PNOTE
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-100303-1_01: Failover member 1_03 | Reason: Available on member 1_03
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 128 -> 122(param_change 1)
[6 Feb 4:19:19][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_2's state: DOWN ---> READY (State: DOWN -> READY.)
[6 Feb 4:19:19][fw4_0];[vs_0];CLUS-216703-1_01: Remote member 1_02 (state DOWN -> ACTIVE*) | Reason: Trying to move to ACTIVE state
[6 Feb 4:19:19][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_2's state: READY ---> ACTIVE (State: READY -> ACTIVE.)
[6 Feb 4:19:19][fw4_0];[vs_0];CLUS-212004-1_01: Remote member 1_02 (state ACTIVE* -> ACTIVE) | Reason: USER DEFINED PNOTE
[6 Feb 4:19:19][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 122 -> 128(param_change 1)
[6 Feb 4:19:58][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_3's state: DOWN ---> READY (State: DOWN -> READY.)
[6 Feb 4:19:58][fw4_0];[vs_0];CLUS-216703-1_01: Remote member 1_03 (state DOWN -> ACTIVE*) | Reason: Trying to move to ACTIVE state
[6 Feb 4:19:58][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_3's state: READY ---> ACTIVE (State: READY -> ACTIVE.)
[6 Feb 4:19:58][fw4_0];[vs_0];CLUS-212004-1_01: Remote member 1_03 (state ACTIVE* -> ACTIVE) | Reason: USER DEFINED PNOTE
[6 Feb 4:19:58][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 128 -> 134(param_change 1)

topic Re: maestro vsx pnotes in Hyperscale Firewall (Maestro)

maestro vsx pnotes

Re: maestro vsx pnotes

Re: maestro vsx pnotes

Re: maestro vsx pnotes

Re: maestro vsx pnotes

Re: maestro vsx pnotes

Re: maestro vsx pnotes