ESXi host refusing to shutdown/reboot (from VI client or ssh) if not in maintenance mode

[gbras]

Hi,

I'm working on a unlicensed (free) version of ESXi update 4

Using unsuppoprted ssh and J.Holland scripts I'm programming unattended emergency shutdown (with APC ups and apcupsd running from VIMA)

I'm trying the script step by step: after stopping all running vms (with <code>vim-cmd vmsvc/power.shutdown)</code> I needed to gracefully shutdown the Esxi host (a dell PE 2950)

so I issued:

~ # /sbin/shutdown.sh
Running sfcbd-watchdog stop
Stopping sfcbd
Running wsmand stop
Stopping openwsmand
watchdog-openwsmand: Terminating watchdog with PID 1557
Running sfcbd stop
This operation is not supported.
Please use /etc/init.d/sfcbd-watchdog start
Running slpd stop
Stopping slpd
Running hostd stop
watchdog-hostd: Terminating watchdog with PID 1475
Running ntpd stop
Stopping ntpd
config implicitly loaded
Saving current state in /bootbank
Clock updated.
Time: 09:22:18 Date: 05/20/2009 UTC

and then:

~ #/sbin/poweroff
poweroff: No.

Why is the host refusing to shutdown?

Thanks

Guido

[gbras]

Hi,

no one has succeded in gracefully shut down unlicensed esxi U4 with an apc ups and a network card?

Guido

[kjb007]

I've been able to poweroff esxi over ssh. Works just fine. Is this repeatable?

-KjB

VMware vExpert

[gbras]

Yes, it is repeatable.

The steps I take are:

-login via ssh to esxi host
-gracefully shutdown all vms (for each vm: vim-cmd vmsvc/power.shutdown <vmid>)
-issue the /sbin/shutdown.sh command (please find output in my first post)
-issue the /sbin/poweroff command (please findoutput in my first post)-> host does not shut down, message "Poweroff: No." appears

Guido

PS

Are you using v3.5 update 4?

Do you use other commands to shutdown host via ssh?

[kjb007]

I used poweroff instead of shutdown.sh It goes through and brings down services as you would expect, similar to your output, but my server did shutdown.

-KjB

VMware vExpert

[gbras]

No way. host shuts down or reboots only from VI client after being put in maintainance mode.

I did this to troubleshoot the issue:

-login via ssh
-shut down all vms
-issued /sbin/poweroff command: no output, no errors, no shutdown!
-went to the physical console, issued a reboot from text management interface: freezed on "rebooting server"
- had to connect via ssh and force shutdown/reboot with -f

then:

-logged in with VI client
-shut down all vms
-issued a reboot without putting the host in mainteinance mode: it didn't reboot!
-had to login via ssh and force with reboot -f

then again:

-logged in with VI client
-put host in mainteinance mode and issued a reboot: it worked!

I don't understand what could cause the issue, it's not a particularly "exotic" ESXi host.

ESXi version is 3.5.0, 153875

It is connected to an iscsi SAN (a datastore on a MD3000i and one on a Openfiler v2.2) and it has a VIMA vm installed

Any advice?

Thanks

Guido

[kjb007]

I assumed you had no running machines. With no running machines, running poweroff on the console did not require maintenance mode, as there's no running virtual machines. Did you tail /var/log/messages when you ran this?

-KjB

VMware vExpert

[gbras]

I had no running vms.

Before issuing /sbin/shutdown.sh and /sbin/poweroff I issued the "vim-cmd vmsvc/power.shutdown" for every vm (or I shutted down vms in VI client)

then I waited 1 minute for the vms shutdown to complete

This is a tail -f of my /var/log/messages during the execution of /sbin/shutdown.sh (sorry for the long text...)

Thanks for your help!

-

---

May 22 19:17:40 Hostd: Accepted password for user root from 127.0.0.1

May 22 19:17:40 Hostd: Event 39 : User root@127.0.0.1 logged in

May 22 19:17:40 Hostd: Event 40 : User root logged out

May 22 19:18:05 vmkernel: 0:03:07:46.129 cpu6:1693)WARNING: UserThread: 406: Peer table full for sfcbd

May 22 19:18:05 vmkernel: 0:03:07:46.129 cpu6:1693)WARNING: World: vm 44276: 911: init fn user failed with: Out of resources!

May 22 19:18:05 vmkernel: 0:03:07:46.129 cpu6:1693)WARNING: World: vm 44276: 1776: WorldInit failed: trying to cleanup.

May 22 19:18:07 sfcb[1676]: Process "emulex" PID is 44344

May 22 19:18:07 sfcb[1676]: Process "lsi_storage" PID is 44355

May 22 19:18:07 vmkernel: 0:03:07:48.564 cpu6:1693)WARNING: UserThread: 406: Peer table full for sfcbd

May 22 19:18:07 vmkernel: 0:03:07:48.564 cpu6:1693)WARNING: World: vm 44408: 911: init fn user failed with: Out of resources!

May 22 19:18:07 vmkernel: 0:03:07:48.564 cpu6:1693)WARNING: World: vm 44408: 1776: WorldInit failed: trying to cleanup.

May 22 19:18:08 VMware[shutdown]: Stopping VMs

May 22 19:18:09 Hostd: Accepted password for user root from 127.0.0.1

May 22 19:18:09 Hostd: Event 41 : User root@127.0.0.1 logged in

May 22 19:18:09 Hostd: Task Created : haTask--vim.host.AutoStartManager.autoPowerOff-153

May 22 19:18:09 Hostd: Task Created : haTask-16-vim.VirtualMachine.shutdownGuest-154

May 22 19:18:09 Hostd: Invalid transition requested (VM_STATE_OFF -> VM_STATE_SHUTTING_DOWN): Invalid state

May 22 19:18:09 Hostd: Task Completed : haTask-16-vim.VirtualMachine.shutdownGuest-154

May 22 19:18:09 Hostd: Activation : Invoke done on

May 22 19:18:09 Hostd: Throw vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Result:

May 22 19:18:09 Hostd: (vim.fault.InvalidPowerState) { dynamicType = <unset>, requestedState = "poweredOff", existingState = "poweredOff", msg = "" }

May 22 19:18:09 Hostd:

May 22 19:18:09 Hostd: Error in initializing PowerOp for VM 16 : vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Task Created : haTask-32-vim.VirtualMachine.shutdownGuest-155

May 22 19:18:09 Hostd: Invalid transition requested (VM_STATE_OFF -> VM_STATE_SHUTTING_DOWN): Invalid state

May 22 19:18:09 Hostd: Task Completed : haTask-32-vim.VirtualMachine.shutdownGuest-155

May 22 19:18:09 Hostd: Activation : Invoke done on

May 22 19:18:09 Hostd: Throw vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Result:

May 22 19:18:09 Hostd: (vim.fault.InvalidPowerState) { dynamicType = <unset>, requestedState = "poweredOff", existingState = "poweredOff", msg = "" }

May 22 19:18:09 Hostd:

May 22 19:18:09 Hostd: Error in initializing PowerOp for VM 32 : vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Task Created : haTask-96-vim.VirtualMachine.shutdownGuest-156

May 22 19:18:09 Hostd: Invalid transition requested (VM_STATE_OFF -> VM_STATE_SHUTTING_DOWN): Invalid state

May 22 19:18:09 Hostd: Task Completed : haTask-96-vim.VirtualMachine.shutdownGuest-156

May 22 19:18:09 Hostd: Activation : Invoke done on

May 22 19:18:09 Hostd: Throw vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Result:

May 22 19:18:09 Hostd: (vim.fault.InvalidPowerState) { dynamicType = <unset>, requestedState = "poweredOff", existingState = "poweredOff", msg = "" }

May 22 19:18:09 Hostd:

May 22 19:18:09 Hostd: Error in initializing PowerOp for VM 96 : vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Task Created : haTask-64-vim.VirtualMachine.shutdownGuest-157

May 22 19:18:09 Hostd: Invalid transition requested (VM_STATE_OFF -> VM_STATE_SHUTTING_DOWN): Invalid state

May 22 19:18:09 Hostd: Task Completed : haTask-64-vim.VirtualMachine.shutdownGuest-157

May 22 19:18:09 Hostd: Activation : Invoke done on

May 22 19:18:09 Hostd: Throw vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Result:

May 22 19:18:09 Hostd: (vim.fault.InvalidPowerState) { dynamicType = <unset>, requestedState = "poweredOff", existingState = "poweredOff", msg = "" }

May 22 19:18:09 Hostd:

May 22 19:18:09 Hostd: Error in initializing PowerOp for VM 64 : vim.fault.InvalidPowerState

May 22 19:18:09 Hostd: Task Completed : haTask--vim.host.AutoStartManager.autoPowerOff-153

May 22 19:18:09 Hostd: Event 42 : User root logged out

May 22 19:18:10 vmkernel: 0:03:07:50.685 cpu4:1693)WARNING: UserThread: 406: Peer table full for sfcbd

May 22 19:18:10 vmkernel: 0:03:07:50.686 cpu4:1693)WARNING: World: vm 44529: 911: init fn user failed with: Out of resources!

May 22 19:18:10 vmkernel: 0:03:07:50.686 cpu4:1693)WARNING: World: vm 44529: 1776: WorldInit failed: trying to cleanup.

May 22 19:18:12 sfcb[1690]: SLP data collection finished

May 22 19:18:14 Hostd: Accepted password for user root from 127.0.0.1

May 22 19:18:14 Hostd: Event 43 : User root@127.0.0.1 logged in

May 22 19:18:14 Hostd: Event 44 : User root logged out

May 22 19:18:15 Hostd: Accepted password for user root from 127.0.0.1

May 22 19:18:15 Hostd: Event 45 : User root@127.0.0.1 logged in

May 22 19:18:15 Hostd: Event 46 : User root logged out

May 22 19:18:15 root: init Running sfcbd-watchdog stop

May 22 19:18:15 vmkernel: 0:03:07:55.808 cpu2:44792)Config: 491: "StatusText" = "Running sfcbd-watchdog stop", Old value: "Running sfcbd-watchdog start" (Status: 0x0)

May 22 19:19:15 root: sfcbd-watchdog:pid 1582 cleaning up

May 22 19:19:15 root: sfcbd-watchdog:stopping sfcbd

May 22 19:19:15 root: sfcbd Stopping sfcbd

May 22 19:19:15 sfcb[44828]: --- stopped ELXHBA_RegisteredProfile_Provider 44828

May 22 19:19:15 vmkernel: 0:03:08:55.885 cpu7:1068)User: 1973: wantCoreDump : init -enabled : 0

May 22 19:19:15 sfcb[2590]: -#- RawIpmiProvider - 1733 provider exiting due to a SIGSEGV signal

May 22 19:19:15 vmkernel: 0:03:08:55.885 cpu6:2590)User: 1973: wantCoreDump : sfcbd -enabled : 0

May 22 19:19:15 sfcb[44854]: --- stopped VMware_LatchedHealthState 44854

May 22 19:19:15 sfcb[1676]: --- Winding down sfcbd

May 22 19:19:15 sfcb[44867]: --- Stopping adapters

May 22 19:19:15 sfcb[1690]: --- SLP Agent for HTTPS Adapter terminating 1690

May 22 19:19:15 sfcb[44881]: --- stopped ClassProvider 44881

May 22 19:19:15 sfcb[44886]: --- stopped SensorProvider 44886

May 22 19:19:15 sfcb[44898]: --- stopped VICimProvider 44898

May 22 19:19:18 root: sfcbd Stopping sfcbd

May 22 19:19:21 root: sfcbd-watchdog:stopping sfcbd

May 22 19:19:21 root: sfcbd Stopping sfcbd

May 22 19:19:21 root: init Running wsmand stop

May 22 19:19:21 vmkernel: 0:03:09:02.213 cpu7:44934)Config: 491: "StatusText" = "Running wsmand stop", Old value: "Running sfcbd-watchdog stop" (Status: 0x0)

May 22 19:19:21 root: openwsmand Stopping openwsmand

May 22 19:19:21 watchdog-openwsmand: Terminating watchdog with PID 1559

May 22 19:19:21 watchdog-openwsmand: Signal received: exiting the watchdog

May 22 19:19:21 root: init Running sfcbd stop

May 22 19:19:21 vmkernel: 0:03:09:02.430 cpu0:44959)Config: 491: "StatusText" = "Running sfcbd stop", Old value: "Running wsmand stop" (Status: 0x0)

May 22 19:19:21 root: init Running slpd stop

May 22 19:19:21 vmkernel: 0:03:09:02.545 cpu2:44964)Config: 491: "StatusText" = "Running slpd stop", Old value: "Running sfcbd stop" (Status: 0x0)

May 22 19:19:21 root: slpd Stopping slpd

May 22 19:19:21 root: init Running hostd stop

May 22 19:19:22 vmkernel: 0:03:09:02.688 cpu3:44976)Config: 491: "StatusText" = "Running hostd stop", Old value: "Running slpd stop" (Status: 0x0)

May 22 19:19:22 watchdog-hostd: Terminating watchdog with PID 1477

May 22 19:19:22 watchdog-hostd: Signal received: exiting the watchdog

May 22 19:19:22 root: init Running ntpd stop

May 22 19:19:22 vmkernel: 0:03:09:02.844 cpu5:44990)Config: 491: "StatusText" = "Running ntpd stop", Old value: "Running hostd stop" (Status: 0x0)

May 22 19:19:22 root: ntpd Stopping ntpd

May 22 19:19:22 ntpd[1471]: ntpd exiting on signal 1

May 22 19:19:23 vmkernel: 0:03:09:04.017 cpu5:45026)Hardware: 2173: Setting RTC date'n'time to: 5/22/2009, 19:19:22

[kjb007 · Accepted solution]

What is the tail after shutting down the vm's and then running poweroff? No maintenance mode, but no running vm's?

-KjB

VMware vExpert

[gbras]

IT WORKS !!!

You know what it was?

I was not waiting enough time for all the vms to completely shutdown! It takes about 2 minutes on my host.

So after issuing the shutdown command for the last VM,I have to wait 1,5-2 minutes, then the hosts can reboot/shutdown because all VMs are completely powered off!

I can't believe it took me 2 weeks to understand this... if I had only looked /var/log/messages before! lesson learned the hard way

Thanks kjb!

Guido

[kjb007]

Glad it is all figured out.

-KjB

VMware vExpert

[mazday]

I had no running vms.

Before issuing /sbin/shutdown.sh and /sbin/poweroff I issued the "vim-cmd vmsvc/power.shutdown" for every vm (or I shutted down vms in VI client)

then I waited 1 minute for the vms shutdown to complete 2

Hi....

try this. Work fine on my ESXi4.1 host:

 vim-cmd vmsvc/getallvms |grep vmx| awk '{ print $1 }'|while read VMID; do echo "work with VM id=$VMID"; vim-cmd vmsvc/power.shutdown $VMID;sleep 20s; done;sleep 300s; poweroff