During some changes on switches in the customer’s environment, vCenter lost its network. The customer was unable to access the vCenter vSphere client to make any changes to the network. Management Network only existed in a Distributed Switch, there were also no ephemeral ports in the cluster. vCenter VM had an error saying VM Network doesn’t exist although he was connected to the vDS portgroup Server Network:
It wasn’t possible to connect vCenter back to the Distributed Switch portgroups on the same or different hosts in a cluster and each try ended with the following error message:
“Addition or reconfiguration of network adapters attached to non-ephemeral distributed virtual port groups is not supported.“
The main problem here was that, if vCenter is connected to the Distributed Switch and it loses network access, it is no longer able to connect to a Distributed Port because it does not have access to the ESXis. VMware recommends that you configure Ephemeral Ports for Management Network in your environment to avoid this issue from happening again.
To get the connectivity back I built a temporary Standard Switch where I connected vCenter to recover from the network down scenario. Be careful doing this and be sure that you have at least 2 vmnics used for the Management Network as you are going to remove one vmnic from the vDS Management Portgroup so you can use it for the Standard Switch that you will create temporarily.
Procedure:
Remove a vDS vmnic connected to the Management Network
1. Use the Secure Shell (SSH) protocol to connect to the ESXi host.
2. Identify the Port ID where the vmnic (in my case vmnic1) that you want to remove is connected to the vDS with the following command:
esxcli network vswitch dvs vmware list | egrep "Client: vmnic1" -A3
3. Remove the vmnic with the following command:
esxcfg-vswitch -Q vmnic1 -V 53 vDS_Name(Example using vmnic1, Port ID 53 and vDS Name -DVS)
Create Standard Switch, a Portgroup and add the vmnic to the Standard Switch
1. Create a Standard Switch (in my case – vSwitch0) with the following command:
esxcli network vswitch standard add --vswitch-name=vSwitch0
2. Create a Portgroup (in my case – recoveryPG) with the following command:
esxcli network vswitch standard portgroup add --portgroup-name=recoveryPG --vswitch-name=vSwitch0
3. Add a vmnic to the Standard Switch (in my case – vmnic1) with the following command:
esxcli network vswitch standard uplink add --uplink-name=vmnic1 --vswitch-name=vSwitch0
Recover vCenter’s VM network connectivity
First connect vCenter’s virtual machine to the new Standard Switch Portgroup (recoveryPG in my case). This will help to regain network access to vCenter. This will also allow the ESXis to connect back to vCenter Server too and you will be able to manage your infrastructure again.
1. Login to the ESXi vSphere Client with administrator credentials
2. Go to “Virtual Machines”
3. Check vCenter’s Virtual Machine
4. Click in “Actions” > “Edit Settings”
5. Connect Network Adapter 1 to the newly created Standard Switch Portgroup
6. Click Save
Now vCenter’s network connectivity should be restored and it should be possible to connect to its vSphere Client. If you still can’t, make sure the Standard Switch Portgroup has the correct VLAN and MTU configuration. After reviewing that everything is ok in vCenter Inventory, migrate vCenter back to the Distributed Switch to have the same configuration before the outage.
1. Login to vCenter with Administrator credentials.
2. Go to the vCenter’s virtual machine, right click it and select “Edit Settings”.
3. Connect Network Adapter 1 to the Management Distributed Switch Portgroup
4. Click OK
If everything is ok after a couple of minutes, continue to the next steps.
Migrate vmnic back t the vDS
Put the vmnic back to the vDS by following these steps:
1. Login to vCenter with Administrator credentials.
2. Go to the “Networking” tab.
3. Right click the vDS and select “Add and Manage Hosts”
4. Select “Manage host networking” and click Next
5. Click “Attached hosts…”
6. Check the ESXi with the vmk and vmnic that you want to add back to the vDS and click OK
7. Click Next
8. On the “Manage physical adapters” list, select the vmnic and click “Assign uplink”
9. Select an Uplink with empty “Assigned Adapter” and click OK
10. Click Next
11. Click Next in “Manage VMkernel adapters”
12. Click Next in “Migrate VM networking”
13. Click Finish
Delete the Standard Switch
Now it is possible to delete the temporary Standard Switch created to recover vCenter’s network connectivity.
1. Login to vCenter with Administrator credentials.
2. Go to the “Hosts and Clusters” Tab
3. Select the ESXi used during this process
4. Click in the “Configure” tab
5. Click on “Virtual Switches” under “Networking”
6. Look for the temporary Standard Switch you created and click the ellipsis “…”
7. Click “Remove”
8. Click “Yes” in the warning that pops up
You have now successfully recovered vCenter’s network connectivity and the environment should be up and running.