This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
ABOUT CHECKMATES & FAQ
Create a Post
Help
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
wsitu
Explorer
‎2024-02-06 02:34 AM

maestro vsx pnotes

Hi experts,

we have the following configuration and experiencing policy installation failure consistently.  please advise how to proceed with troubleshooting and/or solution.  thank you

dual orchestrator mho-140 with 4 sgm.  configured with 5 virtual systems.  r81.20 jumbo 26 

 

#/var/log/messages:

Feb 6 04:18:32 2024 mho-ch01-01 spike_detective: spike info: type: thread, thread id: 55092, thread name: fw_full, start time: 06/02/24 01:18:25, spike duration (sec): 6, initial cpu usage: 99, average cpu usage: 99, perf taken: 0

Feb 6 04:18:38 2024 mho-ch01-01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fw_full, start time: 06/02/24 01:18:25, spike duration (sec): 12, initial cpu usage: 99, average cpu usage: 97, perf taken: 1

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-111500-1_01: State change: ACTIVE -> DOWN | Reason: VSX PNOTE due to problem in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 kernel:[fw4_0];fwkdrv_process_vsx_global_data_in_kernel: Updating smo task to 1
Feb 6 04:18:38 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:38 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xac1f514 due to failure to read from the socket
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeeb74 due to failure to read from the socket
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xac1f514 due to failure to read from the socket
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:38 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeeb74 due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State change: ACTIVE -> DOWN | Reason: Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 5

Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. (message size is 0)
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeeb74 due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. errno = 104 (Connection reset by peer)
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xabeec7c due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: Failed to receive cluster message header, connection will need to be reestablished. errno = 104 (Connection reset by peer)
Feb 6 04:18:39 2024 mho-ch01-01 routed[84478]: [routed] ERROR: cpcl_recv: deleting peer task 0xac38a24 due to failure to read from the socket
Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:39 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 1

Feb 6 04:18:39 2024 mho-ch01-01 sgm_pmd: SGM has left monitoring task
Feb 6 04:18:40 2024 mho-ch01-01 kernel:[fw4_0];fwkdrv_process_vsx_global_data_in_kernel: Updating smo task to 2
Feb 6 04:18:40 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:18:40 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:18:40 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:42 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:18:42 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:18:42 2024 mho-ch01-01 fwk: CLUS-111700-1_01: State remains: DOWN | Reason: Previous problem resolved, ROUTED PNOTE

Feb 6 04:18:45 2024 mho-ch01-01 fw: SMOApiExecHaDoHookCmd: Creating /opt/CPsuite-R81.20/fw1/CTX/CTX00005/conf/fw_updated
Feb 6 04:18:46 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 1

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 0

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 3

Feb 6 04:18:47 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 3

Feb 6 04:18:48 2024 mho-ch01-01 fwk: CLUS-115700-1_01: State remains: DOWN | Reason: Previous problem resolved, Member state has been changed due to issue in Virtual System 3

Feb 6 04:18:52 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Aligning the local GW LACP bonds to chassis monitor GW LACP bonds
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond4 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond4 First slave System MAC address 00:1c:7f:81:08:01 slaves {'eth2-08': '2', 'eth1-08': '1'}, local GW: VS 0 bond4 First slave System MAC address 00:1c:7f:81:08:01 slaves {'eth2-08': '2', 'eth1-08': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond5 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond5 First slave System MAC address 00:1c:7f:81:09:01 slaves {'eth2-09': '2', 'eth1-09': '1'}, local GW: VS 0 bond5 First slave System MAC address 00:1c:7f:81:09:01 slaves {'eth2-09': '2', 'eth1-09': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond6 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond6 First slave System MAC address 00:1c:7f:81:0a:01 slaves {'eth2-10': '2', 'eth1-10': '1'}, local GW: VS 0 bond6 First slave System MAC address 00:1c:7f:81:0a:01 slaves {'eth2-10': '2', 'eth1-10': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond7 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond7 First slave System MAC address 00:1c:7f:81:0c:01 slaves {'eth2-12': '2', 'eth1-12': '1'}, local GW: VS 0 bond7 First slave System MAC address 00:1c:7f:81:0c:01 slaves {'eth2-12': '2', 'eth1-12': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond1 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond1 First slave System MAC address 00:1c:7f:81:05:01 slaves {'eth1-05': '1', 'eth2-05': '2'}, local GW: VS 0 bond1 First slave System MAC address 00:1c:7f:81:05:01 slaves {'eth1-05': '1', 'eth2-05': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond2 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond2 First slave System MAC address 00:1c:7f:81:06:01 slaves {'eth1-06': '1', 'eth2-06': '2'}, local GW: VS 0 bond2 First slave System MAC address 00:1c:7f:81:06:01 slaves {'eth1-06': '1', 'eth2-06': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond3 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond3 First slave System MAC address 00:1c:7f:81:07:01 slaves {'eth1-07': '1', 'eth2-07': '2'}, local GW: VS 0 bond3 First slave System MAC address 00:1c:7f:81:07:01 slaves {'eth1-07': '1', 'eth2-07': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond8 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond8 First slave System MAC address 00:1c:7f:81:31:01 slaves {'eth1-49': '1', 'eth2-49': '2'}, local GW: VS 0 bond8 First slave System MAC address 00:1c:7f:81:31:01 slaves {'eth1-49': '1', 'eth2-49': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond9 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond9 First slave System MAC address 00:1c:7f:81:35:01 slaves {'eth1-53': '1', 'eth2-53': '2'}, local GW: VS 0 bond9 First slave System MAC address 00:1c:7f:81:35:01 slaves {'eth1-53': '1', 'eth2-53': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond10 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond10 First slave System MAC address 00:1c:7f:81:37:01 slaves {'eth1-55': '1', 'eth2-55': '2'}, local GW: VS 0 bond10 First slave System MAC address 00:1c:7f:81:37:01 slaves {'eth1-55': '1', 'eth2-55': '2'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Skip aligning VS 0 bond11 slaves port numbers since the chassis monitor GW bond slaves and the local GW bond slaves port numbers are identical. Chassis Monitor: VS 0 bond11 First slave System MAC address 00:1c:7f:81:0e:01 slaves {'eth2-14': '2', 'eth1-14': '1'}, local GW: VS 0 bond11 First slave System MAC address 00:1c:7f:81:0e:01 slaves {'eth2-14': '2', 'eth1-14': '1'}
Feb 6 04:18:53 2024 mho-ch01-01 align_the_local_LACP_bonds_to_chassis_monitor_bonds.py: Operation was completed
Feb 6 04:18:53 2024 mho-ch01-01 lbd: align_the_local_LACP_bonds_to_chassis_monitor_bonds result, output: 'Operation was completed
', exit code: 0
Feb 6 04:18:54 2024 mho-ch01-01 lbd: Report LACP_SYNC pnote OK, all LACP bonds were successfully verified
Feb 6 04:19:01 2024 mho-ch01-01 confd: setrlimit(RLIMIT_NOFILE, 512): Operation not permitted
Feb 6 04:19:01 2024 mho-ch01-01 xpand[66780]: Performing database cloning
Feb 6 04:19:01 2024 mho-ch01-01 xpand[66780]: load_config>(/config/db/cloned_db,empty)
Tue Feb 06 01:19:03 PST 2024: [asg_update_diag_config]: merging asg_diag_config from SMO with latest asg_diag_config
Feb 6 04:19:13 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:13 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 0
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:17 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-112004-1_01: State change: DOWN -> ACTIVE | Reason: USER DEFINED PNOTE

Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-115704-1_01: State change: DOWN -> ACTIVE | Reason: Member state has been changed due to issue in Virtual System 0

Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-115704-1_01: State change: DOWN -> ACTIVE | Reason: Member state has been changed due to issue in Virtual System 0

Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-115704-1_01: State change: DOWN -> ACTIVE | Reason: Member state has been changed due to issue in Virtual System 0

Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-115704-1_01: State change: DOWN -> ACTIVE | Reason: Member state has been changed due to issue in Virtual System 0

Feb 6 04:19:18 2024 mho-ch01-01 fwk: CLUS-115704-1_01: State change: DOWN -> ACTIVE | Reason: Member state has been changed due to issue in Virtual System 0

Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 xpand[75311]: Configuration changed from localhost by user admin by the service dbset
Feb 6 04:19:18 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:18 2024 mho-ch01-01 kernel:[fw4_0];fwkdrv_process_vsx_global_data_in_kernel: Updating smo task to 0
Feb 6 04:19:18 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:18 2024 mho-ch01-01 sgm_pmd: SGM became port monitoring task
Feb 6 04:19:18 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:18 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:18 2024 mho-ch01-01 asg_send_alert: [1_1] SGM 1 in Chassis ID 1 state changed to UP
Feb 6 04:19:18 2024 mho-ch01-01 sgm_pmd: Updating Orchestrators matrix...
Feb 6 04:19:18 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:18 2024 mho-ch01-01 asg_send_alert: [1_3] SGM 3 in Chassis ID 1 state changed to DOWN.
Feb 6 04:19:19 2024 mho-ch01-01 lbd: Local member turned to "blade_for_task".
Feb 6 04:19:19 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:19 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0x9
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-Mgmt1 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-10 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-12 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-14 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-49 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-05 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-53 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-55 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-06 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-07 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-08 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 asg_send_alert: [1_2] SGM 2 in Chassis ID 1 state changed to UP
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth1-09 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting eth1-Sync cluster link state to down(current cluster state is up)
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Running: cphaprob link_state set local eth1-Sync -d full -s 10000M down
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-Mgmt1 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-10 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-12 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-14 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-49 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-05 link state to up
Feb 6 04:19:19 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-53 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-55 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-06 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-07 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-08 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting interface eth2-09 link state to up
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Setting eth2-Sync cluster link state to down(current cluster state is up)
Feb 6 04:19:20 2024 mho-ch01-01 sgm_pmd: Running: cphaprob link_state set local eth2-Sync -d full -s 10000M down
Feb 6 04:19:20 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:20 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:21 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:21 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xb
Feb 6 04:19:22 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:23 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:23 2024 mho-ch01-01 sgm_pmd: Running distutil update...
Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: Re-generating /etc/groups
Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: /etc/groups: all 1_01,1_02,1_03,1_04

Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: /etc/groups: chassis1 1_01,1_02,1_03,1_04

Feb 6 04:19:23 2024 mho-ch01-01 sg_xlate: /etc/groups: chassis_active 1_01,1_02
Feb 6 04:19:34 2024 mho-ch01-01 PAM-tacplus[5407]: auth failed: 2
Feb 6 04:19:38 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:38 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:39 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:39 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xb
Feb 6 04:19:40 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:40 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:43 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:43 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:43 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:43 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xb
Feb 6 04:19:44 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:19:45 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.
Feb 6 04:19:45 2024 mho-ch01-01 expert: SSH connection by admin user to Expert Shell with client IP 10.1.1.1 at 01:19 02/06/2024
Feb 6 04:19:58 2024 mho-ch01-01 lbd: lbd_trap_handler: cxl_is_blade_for_task(LOAD_BALANCE) = 1
Feb 6 04:19:58 2024 mho-ch01-01 kernel:ssm_com_should_update_switch: update is 1 due to change in present_members: (old) <=> (new)
Feb 6 04:19:58 2024 mho-ch01-01 asg_send_alert: [1_3] SGM 3 in Chassis ID 1 state changed to UP
Feb 6 04:19:59 2024 mho-ch01-01 lbd: lbd_periodic_handler: lbd_trap_received = 1, lbd_num_of_retry = 0
Feb 6 04:19:59 2024 mho-ch01-01 lbd: lbd_send_bucket_list: members = 0xf, active_members = 0xf
Feb 6 04:20:00 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM1: Rest request sent successfully.
Feb 6 04:20:00 2024 mho-ch01-01 lbd: lbd_rest_api_send:: SSM2: Rest request sent successfully.

-----------------------------------------------------------------------------------------------------------------------------------------------
#fwk.elg

[6 Feb 4:18:38][fw4_0];[vs_0];CLUS-120103-1_01: VSX PNOTE ON
[6 Feb 4:18:38][fw4_0];[vs_0];CLUS-111500-1_01: State change: ACTIVE -> DOWN | Reason: VSX PNOTE due to problem in Virtual System 5
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: ACTIVE ---> DOWN (cluster state changed to: DOWN)
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: [SMO]: Current blade (1_1) is no longer SMO Mgmt Master (on 99964150)
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: fwha_update_member_running_dr_task: Changed DR manager from member (1_01) to (1_02)
[6 Feb 4:18:38][fw4_0];[vs_0];FW-1: fwha_update_member_running_smo_task: Changed fwha_smo_member from member (1_01) to (1_02)
[6 Feb 4:18:39][fw4_0];[vs_0];CLUS-100101-1_01: Failover member 1_01 | Reason: VSX PNOTE
[6 Feb 4:18:40][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_2's state: ACTIVE ---> DOWN (State: ACTIVE -> DOWN.)
[6 Feb 4:18:40][fw4_0];[vs_0];FW-1: fwha_update_member_running_dr_task: Changed DR manager from member (1_02) to (1_03)
[6 Feb 4:18:40][fw4_0];[vs_0];FW-1: fwha_update_member_running_smo_task: Changed fwha_smo_member from member (1_02) to (1_03)
[6 Feb 4:18:40][fw4_0];[vs_0];CLUS-211500-1_01: Remote member 1_02 (state ACTIVE -> DOWN) | Reason: VSX PNOTE
[6 Feb 4:18:40][fw4_0];[vs_0];CLUS-100202-1_01: Failover member 1_02 | Reason: Available on member 1_02
[6 Feb 4:18:42][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_4's state: ACTIVE ---> DOWN (State: ACTIVE -> DOWN.)
[6 Feb 4:18:42][fw4_0];[vs_0];CLUS-211500-1_01: Remote member 1_04 (state ACTIVE -> DOWN) | Reason: VSX PNOTE
[6 Feb 4:18:42][fw4_0];[vs_0];CLUS-100404-1_01: Failover member 1_04 | Reason: Available on member 1_04
[6 Feb 4:18:43][fw4_0];[vs_0];CLUS-211505-1_01: Remote member 1_03 (state ACTIVE -> ACTIVE(!)) | Reason: VSX PNOTE
[6 Feb 4:18:48][fw4_0];[vs_0];CLUS-214904-1_01: Remote member 1_03 (state ACTIVE(!) -> ACTIVE) | Reason: Reason for ACTIVE! alert has been resolved
[6 Feb 4:18:52][fw4_0];[vs_0];CLUS-120103-1_01: VSX PNOTE OFF
[6 Feb 4:18:52][fw4_0];[vs_0];report_lacp_sync_pnote_problem: registering LACP_SYNC pnote with problem
[6 Feb 4:18:52][fw4_0];[vs_0];CLUS-120109-1_01: LACP_SYNC PNOTE ON
[6 Feb 4:18:52][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: READY ---> DOWN (cluster state changed to: DOWN)
[6 Feb 4:18:54][fw4_0];[vs_0];CLUS-120109-1_01: LACP_SYNC PNOTE OFF
[6 Feb 4:18:54][fw4_0];[vs_0];fwha_ch_should_pull_config: registering pull_config pnote with problem
[6 Feb 4:18:54][fw4_0];[vs_0];CLUS-120109-1_01: pull_config PNOTE ON
[6 Feb 4:18:54][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: READY ---> DOWN (cluster state changed to: DOWN)
[6 Feb 4:18:54][fw4_0];[vs_0];fwha_ch_should_pull_config: pulling configuration for local SGM
[6 Feb 4:18:56][fw4_0];[vs_0];fwha_mbs_pull_config_start_notify_cb: pull config was initiated
[6 Feb 4:18:56][fw4_0];[vs_0];fwha_mbs_pull_config_start_notify_cb: waiting for all VSs pull config to be done before going up
[6 Feb 4:19:12][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_4's state: DOWN ---> READY (State: DOWN -> READY.)
[6 Feb 4:19:12][fw4_0];[vs_0];CLUS-216703-1_01: Remote member 1_04 (state DOWN -> ACTIVE*) | Reason: Trying to move to ACTIVE state
[6 Feb 4:19:12][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_4's state: READY ---> ACTIVE (State: READY -> ACTIVE.)
[6 Feb 4:19:12][fw4_0];[vs_0];CLUS-212004-1_01: Remote member 1_04 (state ACTIVE* -> ACTIVE) | Reason: USER DEFINED PNOTE
[6 Feb 4:19:17][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 1 finished pull config
[6 Feb 4:19:17][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 2 finished pull config
[6 Feb 4:19:17][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 3 finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 5 finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: VS 4 finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: all VSs finished pull config
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_done_notify_cb: pull config is done
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: fwha_mbs_pull_config_done = 1, removing pnote
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: unregistering pull_config pnote
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-120109-1_01: pull_config PNOTE OFF
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: fwha_mbs_pull_config_done = 1, removing pnote
[6 Feb 4:19:18][fw4_0];[vs_0];fwha_mbs_pull_config_periodic: unregistering pull_config pnote
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: DOWN ---> READY (cluster state changed to: READY)
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-112004-1_01: State change: DOWN -> ACTIVE | Reason: USER DEFINED PNOTE
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_1's state: READY ---> ACTIVE (cluster state changed to: ACTIVE)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [SMO]: Current blade (1_1) is now SMO Mgmt Master (on 99964543)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 122 -> 128(param_change 1)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_update_member_running_smo_task: Changed fwha_smo_member from member (1_03) to (1_01)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_update_member_running_dr_task: Changed DR manager from member (1_03) to (1_01)
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_3's state: ACTIVE ---> DOWN (State: ACTIVE -> DOWN.)
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-211500-1_01: Remote member 1_03 (state ACTIVE -> DOWN) | Reason: VSX PNOTE
[6 Feb 4:19:18][fw4_0];[vs_0];CLUS-100303-1_01: Failover member 1_03 | Reason: Available on member 1_03
[6 Feb 4:19:18][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 128 -> 122(param_change 1)
[6 Feb 4:19:19][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_2's state: DOWN ---> READY (State: DOWN -> READY.)
[6 Feb 4:19:19][fw4_0];[vs_0];CLUS-216703-1_01: Remote member 1_02 (state DOWN -> ACTIVE*) | Reason: Trying to move to ACTIVE state
[6 Feb 4:19:19][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_2's state: READY ---> ACTIVE (State: READY -> ACTIVE.)
[6 Feb 4:19:19][fw4_0];[vs_0];CLUS-212004-1_01: Remote member 1_02 (state ACTIVE* -> ACTIVE) | Reason: USER DEFINED PNOTE
[6 Feb 4:19:19][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 122 -> 128(param_change 1)
[6 Feb 4:19:58][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_3's state: DOWN ---> READY (State: DOWN -> READY.)
[6 Feb 4:19:58][fw4_0];[vs_0];CLUS-216703-1_01: Remote member 1_03 (state DOWN -> ACTIVE*) | Reason: Trying to move to ACTIVE state
[6 Feb 4:19:58][fw4_0];[vs_0];FW-1: [BLADE]: Blade 1_1 changed Blade 1_3's state: READY ---> ACTIVE (State: READY -> ACTIVE.)
[6 Feb 4:19:58][fw4_0];[vs_0];CLUS-212004-1_01: Remote member 1_03 (state ACTIVE* -> ACTIVE) | Reason: USER DEFINED PNOTE
[6 Feb 4:19:58][fw4_0];[vs_0];FW-1: fwha_assure_system_stability_for_df: Local Chassis grade: 128 -> 134(param_change 1)

0 Kudos
1 Reply
Lari_Luoma
Ambassador Ambassador
Ambassador
‎2024-02-06 11:06 AM

I would recommend opening a TAC case for further troubleshooting and finding a solution.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

View All ≫
 
Upcoming Maestro Events

    CheckMates Events