Remove Iscsi Storage

[vincentc]

Hi

New to ESXi5.0 and Iscsi (upgrade from ESX4 and Fibre), I unpresented some Lun from the SAN before detaching the devices in Vcenter (I do know now!), so a rescan does not remove them anymore.

On the first host I did a rescan, after roughly one hour, the host disconnected from Vcenter. I tried to restart the agent (/sbin/services.sh restart) but it hangs on "Running usbarbitrator restart"

It has been in this situation now for more than 12 hours

I did not rescan the second host, to avoid loosing it from Vcenter too.

All the VM are running OK.

I could take the second host in maintenance mode and reboot it, but I am not sure if the VM will be migrated properly on the first host, if it is not available via Vcenter.

Also  I would not be able to do the same on the first host, as I cannot vmotion the VM, having no control anymore from Vcenter. If I reboot the server, the VM will be probably shutdown and restarted on the other host via DRS but that would cause many VM down time, which I cannot afford at the moment.

I am assuming that the rescan of the device is the main isssue here. Is there a way to unblock the situation without a host reboot (hopefully it will fix it), and if a reboot is the only way, what is the best way to achieve it with no (or minimal) down time for the VM?

Any help would be greatly appreciated

[athlon_crazy]

The problem now I think is your host1 now in hang state and not reachable by host2 and vCenter, no matter how you want to migrate(manual or auto DRS) the VMs either from host 1 to host2 or vice versa it's seem impossible.

BTW, since it hangs at "usbarbitrator", have you try to kill -9 it?

[vincentc]

Yes, not sure why I cannot reconnect to the host from vcenter anymore.

For the usbarbitrator, I did 'chkconfig usbarbitrator off' and was able to restart "/sbin/services.sh restart" successfully.

However it still did not reconnected the host to vcenter automactically, nor when trying manually.

I think the server is in this loop about trying to disconnect the Iscsi devices than nolonger exist and this seems to have a huge impact on the vcenter and management agents.

All the VM on this host are running fine.

If I knew how to 'stop' and 'clear' the 'dear or error' devices, that would probably help.

My last resort is to reboot the host tonigh.

Would you know if there is a command (I have only ssh access to the host) to vmotion properly all the VM, or will the reboot most probably kill the VM to move them to the other hosts with DRS?

[athlon_crazy]

I don't know the command. My suggestion, you can try enter maintenance mode first for host1 before reboot via CLI

#vim-cmd hostsvc/maintenance_mode_enter

and see whether vMotion will happen. If not, soft power off all VMs

#vim-cmd vmsvc/power.shutdown VMID

[vincentc]

The command did not work

I did hard reset the host.

It did not reconnect automatically to vcenter.

After 1/2 hour, I tried to connect manually.

It has been stuck on 89% for nearly 45 minutes now.

I notices that all the VM on that host did and got rebooted, however from Vcenter they have an alarm '"vsphere HA virtual machine failover failed"

When I clear the alarm, the VM shows as powered off in vcenter, even if it is not the case in practice

How do I reconnect the host?

Do I need to remove the VM from inventory and add again whilst it is going, or is there a better way

All you help would be greatly appreciated

[athlon_crazy]

If you can connect to host directly via client or ssh, go verify if the VMs are still inside the host. Then, unregister and register to another host if necessary.

If you still cannot add host to vc, you can try disconnect host, uninstall vpxa agent and reconnect host to vc

Sent from my iPad 2

[vincentc]

About 30 minutes ago, the host reconnected by itself. Obviously I was not patient enough! I did not know it would take that long.

All the VM where OK once the alarm get cleared.

I am glad it is all over.

I just need to make sure I am more cautious next time I unpresent LUN from the san. I have about 50 RDM to vstorate motion to vmdk and then unpresent/remove!

Thanks again for taking the time to give me ideas and directions. Very much appreciated

[VirtuallyMikeB]

Good day,

I know this experience can be frustrating for folks.  For the sake of everyone's time, Jason Bosche threw up a quickie blog post about properly removing datastores from vSphere.  I'll give you the short version: it's not right-click, delete.

http://www.boche.net/blog/index.php/2012/01/18/how-to-properly-remove-vsphere-datastores/

Cheers,

Mike

http://VirtuallyMikeBrown.com

https://twitter.com/#!/VirtuallyMikeB

http://LinkedIn.com/in/michaelbbrown