9800 interfaces links down but switchports up

[genisis__]

Just thought I would post this issue I've just experienced.

9800 appliances with extra interface modules configured as VSX running R82 with JHFA41.

Symptom:

When switchports are enabled which lead to the interfaces on the 9800 it appear that this is up, however on the 9800 side the link status is down.  Visually on the switch where the port should be green, this is orange (Note: Onboard interfaces where fine)

Originally I had only one interface showing this issue and after trying different things I powered off/on the appliance via the LOM.

After this, the port started to work normally again, however, now other ports where showing this issue, additionally it was noted that a number of processes had terminated.

I logged a TAC case with Checkpoint (who are still investigating this).
Eventually via the LOM I did a hard reset, and after this everything was good, including the processes.  At this point its not clear if this is a hardware issue or a potential undiscovered feature (bug!).

I have 4 units in totally and thus far only seen this on one units, which for me would seem more like hardware issue, but I will wait for Checkpoint to conclude there investigation.

[the_rock]

I find usually ifdown ethx; ifup ethx works well.

Andy

[genisis__]

There was more going on here the_rock, but your right normally this would be what I would do.

[the_rock]

Having read you last response, definitely makes more sense now.

Andy

[genisis__]

Also its not obvious but to run hardware diagnostics on a 9000 appliance:
R81 and higher versions support running the Diagnostic Tool in the Expert mode. The applicable command is:
expert# diagMain
(SK97251)

Note:
When exiting the diagMain tool it will reboot the node, so be aware of this!

+--------------------------------------+
| HW Diagnostics |
+--------------------------------------+
| |
| |
| The system will be rebooted now... |
| |
| |
| |
+--------------------------------------+

Use <Up> and <Down> arrow keys to navigate
<ENTER> to select the menu item, <q> to exit current menu
Broadcast message from admin@FW (Sun Oct 12 12:28:24 2025):

The system is going down for reboot NOW!

Message from syslogd@FW at Oct 12 12:28:29 ...
monitord[18996]: monitord got killed

and in order to save logs you do need to connect a USB to the unit which makes sense, but would be nice if you had the option to store on the appliance as well.

[the_rock]

Had no idea, thats super useful, THANK YOU!

[Chris_Atkinson]

Which expansion NIC and SFPs out of interest?

[genisis__]

Hi Chris,

TAC came back with this.

If the system is powered off abruptly or rebooted without a clean shutdown, the file system may be marked as "corrupted."
On the next boot, the operating system will check the file system status:

• If the file system is marked "corrupted," fsck is automatically invoked during boot.

[   29.026418] FAT-fs (md3): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   29.027295] XFS (dm-0): Mounting V5 Filesystem
[   29.056344] XFS (dm-0): Starting recovery (logdev: internal)
[   29.065516] XFS (dm-0): Ending recovery (logdev: internal)

Now the order I did the reboots:
- normal reboot from the cli (Initially did this)
- Poweroff via LOM (Did this when the normal reboot did not fix the issue with the one port that was not coming up)
- Hard reset (Did this after the Poweroff made things worse ie. all the module ports not working), and this fixed everything.

Module part code is : CPAC-8-1/10F-D
SFP modules are 10GB-SR and 1GB copper SFPs