Migrating from 10G to 40G interfaces

[Martijn]

Hello all,

Our customer is using two 10G ports in a bond on his 44K cluster but needs to migrate to a bond of two 40G ports. The 44K is configured as a VSX gateway, so the 10G bond is used in many virtual systems.

What is the best procedure to migrate to a 40G bond with as little impact as possible? I have several options.

1.
Adding the 40G ports to te excisting bond and removing the 10G ports.

I know ports in a bond must be of the same type to avoid problems, but during a maintenance window it is allowed to have some impact. And the time two different type of ports are in the bond is very short.

2.
Creating a new bond and use the 'vsx_util change_interfaces' command to change the interfaces on the virtual system.

But this requires a reboot of the VSX gateway which means a reboot of all SGM's and a total outage. Rebooting the SGM's one by one seems tricky because I do not know what happens when a SGM pulls the configuration from the SMO during boot. 

3.
Create a new bond and change the interfaces per virtual system in SmartConsole.

This is a lot of work because there are many virtual systems and some of them have many VLAN interfaces. And I believe it is best to first remove the old interface, push VSX configuration and policy and then add the same interface on the other bond. Push VSX configuration and policy. And this will cause downtime also.

Anyone has done this before? Tips and tricks are welcome.

Regards,
Martijn

[Bob_Zimmerman]

I would absolutely go with option 1. The only problem I have ever seen with different interface speeds in a bond has been with load distribution (the distribution algorithm doesn't care about interface speed, just about the number of interfaces). I would add the two 40g, remove the two 10g, test, and call it a day.

I don't directly use any interfaces on my firewalls anymore because it's so much easier to shift bonds around.

[Kaspars_Zibarts]

Yapp, I would go with 1. Done it on regular appliances and works like a charm. From memory when we had 41k, it should work with SP too. Especially if you have maintenance window and small "dips" are allowed

[Martijn]

Hi,

Well, we did not had a small dip, but a complete outage because OSPF routes where gone after adding the new interface to the bonding group. The only way we could recover was restarting the OSPF processes on the routers or deleting the OSPF configuration in Gaia and configure OSPF again.

And when we where done with the physical part and removed the old interfaces from the bonding group we had the same outage again.

We had a maintenance window and we could recover quickly. But adding a new interface to a existing and functional bond, causes OSPF routes to be deleted.

Did you see the same in your environment?

Regards,
Martijn

[Kaspars_Zibarts]

We don't use dynamic routing on CP... Past experience wasn't great. Although things might have certainly improved I'm still reluctant to do that 🙂

[andymong]

Sorry..as this is late..but we used option #3 and had no issues with that. Clean Slate with a change of OSPF Area migrating from an old HP to a Cisco ACI backend. 
We did/do however have occasional issues with BGP .