LOM refusing conections after firmware update to 6.15

[Vladimir]

After upgrading HTML5 LOM on new 16200 appliance running R81 stock with option to preserve settings selected, LOM reverted to the factory default IP of 192.168.0.100 and refusing connections.

It is pingable on default IP, but does not allow browser connectivity (including from incognito mode)

Any ideas on how to revive it?

Thank you,

Vladimir

[the_rock]

Just to be sure, below was checked?

 

[Vladimir]

A single checkbox in Maintenance was checked for "Preserve Configuration".

Your screenshot is from HTML5 or the Java-based LOM?

P.S. Even if the config is hosed, it still should allow connectivity to default IP.

[the_rock]

Its from Java based LOM. I agree with you, should allow access to default IP, thats true.

[Bob_Zimmerman]

Try a cold reboot of the LOM card.

service ipmi start
ipmitool bmc reset cold
service ipmi stop

That forces the LOM's OS to come back up from scratch. Might clear out any bad state if the upgrade just caused a warm reset.

[Vladimir]

Thanks- tried that, no dice: getting ERR_CONNECTION_REFUSED.

I can change the IP of the LOM using lomipset and see it taking effect using

service ipmi start

ipmitool lan print 8

but still, cannot connect to it.

Incidentally, show lom ip-address does not show anything, if the gateway is not specified in LOM's configuration.

If gateway is specified, the assigned IP is shown.

show lom version returning Firmware Revision : 6.15

The config looks like:

[Expert@CP1:0]# ipmitool lan print 8
Set in Progress         : Set Complete
Auth Type Support       :
Auth Type Enable        : Callback : MD5
                        : User     : MD5
                        : Operator : MD5
                        : Admin    : MD5
                        : OEM      : MD5
IP Address Source       : Static Address
IP Address              : 10.XX.XX.X
Subnet Mask             : 255.255.255.0
MAC Address             : 00:1c:7f:46:33:97
SNMP Community String   : AMI
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 1.0 seconds
Default Gateway IP      : 0.0.0.0
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : caaaaaaaaaaaXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
[Expert@CP1:0]#

Any suggestions?

[_Val_]

It is very likely that config was not preserved during the upgrade. Try connecting with the initial admin/admin credentials, as described here.

If this does not work, I would suggest a TAC case.

[Lesley · Accepted solution]

My quess would be:

- Are you connecting to the LOM from the same network? I see you do not have a default gateway configured on the LOM 

 ipmitool lan set 8 defgw ipaddr <IP> 

- Try to start services manually:

 /etc/init.d/ipmi start

Or

 service ipmi start

- What about the certificate? Self-signed, maybe expired. Maybe the browser causes some issues. Are you able to telnet towards it on port 443? 

- Also version states 6.15 states: Note: To make sure all changes are applied when upgrading from previous versions to 6.15, do not use the "Preserve configuration" feature. I think you might used this feature. 

- Maybe try to factory reset it with: ipmitool raw 0x2e 0x02

Cannot be sure if this is the correct command for this type of LOM. 

- Perform a real cold start, shutdown appliance and remove ALL power and wait few minutes. Add power and start appliance. Then check again.

This would be last option to be honest. 

[Vladimir]

Thank you for your suggestions.

1. Connecting from connected network

2. With GW configured (using lomipset command) the result is the same

3. The prompt NOT to preserve configuration is counter to the notion of OOB remote management, hence it was ignored and the option to Preserve Configuration was chosen. -Imagine that you must upgrade LOMs around the world- I do not see us being locally present for that.

3. The ipmi tool raw 0x2e 0x02 does not work on either HTML5 LOMs or on this specific LOM.

4. Working on this remotely, so not an option at the moment. May have to resort on remote (local) help to do that if TAC will instruct to do so, when I'll open an SR.

Cheers,

Vladimir

[Lesley]

Regarding point 3, this is not the case since you can set a new IP via de CLI from the firewall. The card you have now got a factory reset, this is why it got the default IP again. If you do not choose the Preserve Configuration option it does the same. The card get's the new firmware and goes back to factory settings. If you assign a new IP you can connect to it and it will start the first time wizard. 

It is unclear for me if the note:

version states 6.15 states: Note: To make sure all changes are applied when upgrading from previous versions to 6.15, do not use the "Preserve configuration" feature. 

Is the cause of the problem, it is there for a reason. I would at least mention it in the TAC case. 

[Lesley]

Just released that it could still be a client server issue (HTTPS). Try different browser, or even old browser like IE.

Make Wireshark capture on the client and see if there is a match between cipher suite. The client sends out a cipher suite and the server. They need to have a match otherwise there is no connection.

Could be that in the software update the cipher suite has been changed and is maybe more strict now on the LOM. 

[Vladimir]

@Lesley The ipmitool raw 0x2e 0x02 ended-up working, when tried again after the reboot.

Thank you.

[eladni]

Hi,

Is there a service request open with TAC?

Please let me now so I can follow-up on this issue.

Thanks

[Vladimir]

No TAC SR opened for this issue yet. Looks like I may have to do that and get back to you with the SR#.

Thank you!

[Vladimir]

Service Request 6-0003455308 

Thank you!

[JozkoMrkvicka]

What was the previous version before upgraded to 6.15 ?

[Vladimir]

It was 6.10.0

LOM Board IAC-AST2500

Hardware revision 1.0

[JozkoMrkvicka]

My experience is that even it is not explicitly mentioned, I always upgrade only 1 version up. So in this case I will go with 3 firmware upgrades:

6.10 -> 6.13 -> 6.14 -> 6.15

And after every firmware upgrade I will wait about 10 minutes between the upgrades, just in case LOM is loaded correctly and fully in operation.