VMware Cloud Community
EirikTF
Contributor
Contributor
Jump to solution

VMware ESXi 5.0 Update 1 breaks AutoStartManager (free license ONLY, awaiting patch/fix)

Hi,

Today I updated from build 515841 to 623860, the update reportedly was successful.

However, I've noticed something very annoying - the host no longer automatically starts or shutdowns guests as configured. I've tried to turn the feature on and off, moved guests to "Manual Startup" then back to "Automatic Startup" - with various combinations of these settings, also rebooted host in between changes. But alas, no luck. :smileyplain:

In the "Recent Tasks" list in vSphere Client, the Auto Power On tasks shows up as completed, initiated by root, with "Start Time" and "Completed Time" having the same timestamp. The server log reports the following error (3rd line below):

2012-03-16T07:08:27.296Z [6DE40B90 info 'TaskManager'] Task Created : haTask--vim.host.AutoStartManager.autoPowerOn-97171136
2012-03-16T07:08:27.297Z [6DE40B90 verbose 'Hostsvc::AutoStartManager'] Powering on VM 5 with delay 120
2012-03-16T07:08:27.333Z [6DE40B90 error 'Hostsvc::AutoStartManager'] Error in initializing PowerOp for VM 5 : vim.fault.RestrictedVersion
2012-03-16T07:08:27.334Z [6DE40B90 info 'TaskManager'] Task Completed : haTask--vim.host.AutoStartManager.autoPowerOn-97171136 Status success

RestrictedVersion? Did something happen to licensing features for free ESXi? :smileyconfused:

Anyone else with the same problem/a solution?

Thanks in advance!

Reply
0 Kudos
78 Replies
beamerblvd
Contributor
Contributor
Jump to solution

Thanks, Robin. I haven't tried it yet, but I'm certain this will be very useful for me. Just waiting for the weekend to apply it.

However, I'm curious: Do you happen to know what the equivalent change would be to make sure all VMs shut down gracefully, too? Right now it's just "pulling the cord" because auto-shutdown is broken by this change as well.

Reply
0 Kudos
fribergr
Contributor
Contributor
Jump to solution

Well, I haven't tried this myself.. But it might be possible to change the file /etc/inittab

Add a row like:

::shutdown:vim-cmd vmsvc/power.shutdown 1; sleep 15; vim-cmd vmsvc/power.shutdown 2; sleep 90

Before the rest of the ::shutdown rows. For me this would be:
::shutdown:vim-cmd vmsvc/power.shutdown 1; sleep 15; vim-cmd vmsvc/power.shutdown 2; sleep 90
::shutdown:/usr/lib/vmware/vmksummary/log-bootstop.sh stop
::shutdown:/sbin/shutdown.sh
at the end of inittab. This MIGHT work, I no longer have a test environment to try this at thou Smiley Happy
To explain, the /etc/inittab defines commands that should run when the system is in a specific state. When the system enters shutdown state, we can call vim-cmd vmsvc/power.shutdown to gracefully shutdown our VMs that are still alive. By sleeping 90 seconds in the end, I would like to think that the VMs gets powered off before the ESXi-host.
This is as I said, not tested. It might work, it might not.
I do not know if the inittab runs sequential or parallel, so a worst case scenario would be that the VMs are getting a gracefull shutdown and the ESXi hosts reboot in the middle of the shutdown.
Also, if you are running Windows VMs, and they have a lot of security patches or such, the shutdown process of the guest might be alot longer than 90 seconds.
Try this and report back Smiley Happy
Good night!
Regards
/Robin
Reply
0 Kudos
indexs
Contributor
Contributor
Jump to solution

Hello!

I’m have also run in the same problem and have searched for a solution and found this:

This was made for VMware ESXi 4.1, but this might work with ESXi 5, I have not try this.

http://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A%2F%2Fwww.vmgu.ru%2Farticles%2Fvmwa...

But I will try this evening and get back with the results, if someone will attempt this, please report the results

p.s. Don’t try the first method (Способ №1: Как делать не надо:)) - google translate didn’t the right translation.

Reply
0 Kudos
isanaud78
Contributor
Contributor
Jump to solution

I've added a kline in /etc/rc/local to start my guest at hosts start up.

But I can't suspend my guests at host shutdown (with APC)

I've modified /etc/inittab with ::shutdown:/bin/vim-cmd vmsvc/power.suspend 1;sleep 90, It seems not work :smileyconfused:

Reply
0 Kudos
thoso
Contributor
Contributor
Jump to solution

OK!

VmWare released 5.0.0. update 1 build 623860 official for free hypervisor.

But autostart isn't fixed !! Ok, it's the same iso for the sphere Esxi!

So i installed the latest patch ( Build 702118 ) but the problem isn't fixed !!!

Vmware - what's now!?

We are waiting ....

Reply
0 Kudos
johnnyt29
Contributor
Contributor
Jump to solution

am looking at re-installing ESXi 5.0 (free) released Dec 2011 to fix this problem, then applying patches one at a time (from http://www.vmware.com/patchmgr/findPatch.portal) up to but NOT including the one that breaks auto start.

There is a patch that's part of update 1 I absolutely need (KB 2010817). Can one "cherry pick" the patches (from update 1 and later) in a such as way as to avoid the code that breaks auto start but get the other stuff? I don't see a way to do that at the "Download Patches" portal.

Reply
0 Kudos
Dave_Mishchenko
Immortal
Immortal
Jump to solution

Patches are cummulative, so you would need to select something from before the Update 01 patch release.

Reply
0 Kudos
peetz
Leadership
Leadership
Jump to solution

Well, you could just extract the driver's vib from the U1 patch bundle and install it.

See my post here: http://communities.vmware.com/message/2055466#2055466

- Andreas

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
Reply
0 Kudos
gwiesenekker
Contributor
Contributor
Jump to solution

I have created a couple of scripts as a temporary workaround for this issue.

vmsuspend.sh will suspend all running VMs:

#!/bin/ash

for Vmid in $(vim-cmd vmsvc/getallvms | tail -n +2 | cut -f1 -d' ')
do
    if vim-cmd vmsvc/power.getstate $Vmid | grep -q "Powered on"
    then
        vim-cmd vmsvc/power.suspend $Vmid
    fi
done

and vmresume.sh will power on all suspended VMs:

#!/bin/ash

for Vmid in $(vim-cmd vmsvc/getallvms | tail -n +2 | cut -f1 -d' ')
do   
    if vim-cmd vmsvc/power.getstate $Vmid | grep -q "Suspended"
    then
        vim-cmd vmsvc/power.on $Vmid
    fi
done

I execute these scripts using ssh to the ESXi servers. As ESXi seems to remove new files from the / filesystem on reboot I store these files in /vmfs/volumes/datastore1.

I can call vmresume.sh from /etc/rc.local to resume VMs on reboot, but I have not been able to call vmsuspend.sh from for example /sbin/shutdown.sh to suspend VMs on shutdown, as ESXi seems to restore the original file on reboot. Is there a way to prevent this from happening?

Regards,

Gijsbert

Reply
0 Kudos
mello73
Contributor
Contributor
Jump to solution

Hi, same problem here, so i created a script to autostart VM that read config file /etc/vmware/hostd/vmAutoStart.xml and start only VM that are configured to be started with the appropriate delay and options. Add the following line at the bottom of /etc/rc.local; for the stop command to work add "::shutdown:/etc/rc.local stop" as the first ::shutdown line in /etc/inittab

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#######AUTO START VMS##############
CFGFILE=`cat /etc/vmware/hostd/vmAutoStart.xml`
SLEEPSTART=`echo "$CFGFILE"|grep startDelay|tail -n 1|sed 's/[^>]*>\(-\?[^<]*\).*/\1/'`
SLEEPSTOP=`echo "$CFGFILE"|grep stopDelay|tail -n 1|sed 's/[^>]*>\(-\?[^<]*\).*/\1/'`
STOPACT=`echo "$CFGFILE"|grep stopAction|tail -n 1|sed 's/[^>]*>\(-\?[^<]*\).*/\1/'`
WAIT=`echo "$CFGFILE"|grep waitForHeartbeat|tail -n 1|sed 's/[^>]*>\(-\?[^<]*\).*/\1/'`

GetStopState(){
    case "$1" in
        PowerOff)
            echo off
            ;;
        GuestShutdown)
            echo shutdown
            ;;
        Suspend)
            echo suspend
            ;;
        SystemDefault)
            echo `GetStopState $STOPACT`
    esac
}

greppa(){
  echo "$CFGFILE"|grep startAction -C 4|grep $1 -C 4|grep $2|sed 's/[^>]*>\(-\?[^<]*\).*/\1/' 
}

OTHERVMS=`greppa None moid`
NOTHERVMS=`greppa None moid|wc -l`
VMS=`greppa PowerOn moid`
NVMS=`greppa PowerOn moid|wc -l`
SLEEPSTART_VMS=`greppa PowerOn startDelay`
SLEEPSTOP_VMS=`greppa PowerOn stopDelay`
STOPACT_VMS=`greppa PowerOn stopAction`
N=0

if [ "$1" = "stop" ];then
  for VM in `echo "$OTHERVMS"`;do
    ID=`echo $OTHERVMS|cut -d" " -f$(($NOTHERVMS - $N))`
    vim-cmd vmsvc/power.getstate $ID|tail -n 1|grep -q "Powered on" || continue
    sleep $SLEEPSTOP
    vim-cmd vmsvc/power.off $ID
    N=$(($N+1))
  done
  N=0
  for VM in `echo "$VMS"`;do
    ID=`echo $VMS|cut -d" " -f$(($NVMS - $N))`
    vim-cmd vmsvc/power.getstate $ID|tail -n 1|grep -q "Powered on" || continue
    SLEEPTIME=`echo $SLEEPSTOP_VMS|cut -d" " -f$(($NVMS - $N))`
    [ $SLEEPTIME != "-1" ] || SLEEPTIME=$SLEEPSTOP
    STOPMETHOD=`echo $STOPACT_VMS|cut -d" " -f$(($NVMS - $N))`
    STOPMETHOD=`GetStopState $STOPMETHOD`
    sleep $SLEEPTIME
    vim-cmd vmsvc/power.${STOPMETHOD} $ID
    N=$(($N+1))
  done
  exit 0
fi

for VM in `echo "$VMS"`;do
  ID=$VM
  vim-cmd vmsvc/power.getstate $ID|tail -n 1|grep -q "Powered on" && continue
  SLEEPTIME=`echo $SLEEPSTART_VMS|cut -d" " -f$((N+1))`
  [ $SLEEPTIME != "-1" ] || SLEEPTIME=$SLEEPSTART
  sleep $SLEEPTIME
  vim-cmd vmsvc/power.on $ID
  N=$(($N+1))
done

exit 0

Reply
0 Kudos
gwiesenekker
Contributor
Contributor
Jump to solution

A nice trick to call /etc/rc.local with a 'stop' argument to work around the /sbin/shutdown.sh issue!

Thanks,

Gijsbert

Reply
0 Kudos
johnnyt29
Contributor
Contributor
Jump to solution

great script, Mello73. thanks for posting.

doesn't work for shutdown for me though - vm's go down hard.

I did add "::shutdown:/etc/rc.local stop" as the first ::shutdown line in /etc/inittab

both filles attached here in case I'm just not seeing a mistake somewhere

Reply
0 Kudos
mello73
Contributor
Contributor
Jump to solution

sorry i didn't test it on shutdown (no time to stop servers), i hoped inittab script would have been executed sequentially, but if it didn't work them maybe they are run in parallel so stop script have no time to be executed before shutdown.

To debug the problem you could paste the relevant section in a file and put some echo before shutdown command, then launch it and sees if it outputs the correct command (especially the sleep one).Ex:

echo sleep $SLEEPSTOP

echo vim-cmd vmsvc/power.off $ID

....

....

echo sleep $SLEEPTIME
echo vim-cmd vmsvc/power.${STOPMETHOD} $ID

If i have time i will investigate more.

Bye, Mello.

Reply
0 Kudos
johnnyt29
Contributor
Contributor
Jump to solution

Am not really competent enough yet to troubleshoot this. This is my first time with ESXi (a home lab). For example I don't even now where to look for the output of echo commands...

Is the bug maybe something that affects running any/all scripts at shutdown? I mean the /sbin/shutdown.sh does have the following command, which works fine when I'm in evaluation mode but doesn't when I put in my "free" license.

#Power off autostart configured VMs
/sbin/vmware-autostart.sh stop

I didn't try your script in eval mode but I may since I will have to go back to eval mode to get autostop working again. I don't want my vm's going down hard when power fails and my UPS calls for a host shutdown. Also if it does affect all shutdown scripts, it makes wonder what else isn't shutting down properly (am guessing at least all the things called by shutdown.sh)  So on top of crashing my vm's I'm worried it could cause data corruption of a  host file that one day prevents the system from restarting/running properly.

Reply
0 Kudos
gwiesenekker
Contributor
Contributor
Jump to solution

Meanwhile I found why a modified /sbin/shutdown.sh gets replaced on reboot:

http://blog.andrecardoso.eu/2011/09/gracefully-turn-off-an-esxi-host-and-vms/

I will try to include a modifed /sbin/shutdown.sh in /bootbank tomorrow.

Regards,

Gijsbert

Reply
0 Kudos
mello73
Contributor
Contributor
Jump to solution

What i mean is to ssh to esx server, create a file with the relevant section (the one i posted before to be added to /etc/rc.local/ and modify the lines in the way i posted before (adding echo before sleep and shutdown command). Then make the file executable (chmod +x "nome file"), and run it Es "/tmp/test stop". This way you could see if the proper sleep and shutdown commadn that the script launch are correct (its correctly parsig the xml file).

Tomorrow i will try to reboot my server and test shutdown procedure.

Bye, Mello.

Reply
0 Kudos
gwiesenekker
Contributor
Contributor
Jump to solution

I have it working now by embedding a modified /sbin/shutdown.sh in /bootbank. The procedure is as follows:

The vmsuspend.sh and vmresume.sh scripts mentioned in my previous post reside in the directory /vmfs/volumes/datastore1.

Add the line '/vmfs/volumes/datastore1/vmresume.sh' as the last line to /etc/rc.local

For some reason you cannot edit and save /sbin/shutdown.sh even if you give it write permissions, but after some trial and error I found out that making a copy of /sbin/shutdown.sh, vi the copy and copying the modified copy back achieves the desired result, so

cd /sbin

cp shutdown.sh shutdown1.sh

chmod +w shutdown1.sh

Add the line '/vmfs/volumes/datastore1/vmsuspend.sh' to shutdown1.sh just before the line '#Power off autostart configured VMs'.

cp shutdown1.sh shutdown.sh

chmod -w shutdown.sh

rm shutdown1.sh

cd /bootbank

tar -zcf shutdown.tgz /sbin/shutdown.sh

Add ' --- shutdown.tgz' to the end of the 'modules=' line in boot.cfg.

Regards,

Gijsbert

Reply
0 Kudos
gernst
Contributor
Contributor
Jump to solution

Sorry, but this breaks my ESXi server. It hangs with a fatal error during boot.

But I found a really easy working solution:

Based on the script of mello73 I created I file file called vmware-autostart.sh which is originally

the one who starts and stops the VMs (now broken, and I am not shure this is intentionally).

This file should be reside in the /vmfs/volumes/datastore1 to make shure it is not cleared

during a boot process.

In the file /etc/rc.local insert the following line at the end or use my rc.local

# Bugfix for VMware 5.0.0U1 Auto Start and Stop all VMs
# Replace VMware autostart shell script to start and stop VMs by command
cp /vmfs/volumes/datastore1/vmware-autostart.sh /sbin/vmware-autostart.sh

Please make shure both files are got the correct access rigths.

As you could see, the /sbin/vmware-autostart.sh are overwritten during init,

end later on executed by the init process and during shutdown.

I hope this helps everybody, please report if there is any error.

Ernst

Reply
0 Kudos
mello73
Contributor
Contributor
Jump to solution

I am sorry that my script breaked your boot, i tested it and during boot process had no problem at all and machine got booted in the right order and timing.

I still had problem during shutdown, esx seems to totally ignore the line in inittab; i am still investigating, unfortunately i don't have much time to test because people are always working on the Virtual Machine and i cannot reboot the server.

Maybe i will try gwiesenekker solution for shutdown, but onestly, for the moment, shutdown procedure is less important for me.

Bye Mello.

Reply
0 Kudos
gernst
Contributor
Contributor
Jump to solution

Thats a mistake. Nothing is wrong with your script.

The last post from gwiesenekker about changing the boot.cfg in /bootbank breaks my boot.

Your script is very good, only the shutdown and some small other things are not working.

So I have modified your script (is now in file vmware-autostart.sh).

You do not need to change anything in inittab !!!

Just use my procedure as described in my previus post. Thats all !!!

Ernst

Reply
0 Kudos