scripted build problem, when removing all hba drivers

[Yattong]

Hi,

I have a scripted build that works fine when I zone it out so that it is unable to see the SAN.

When I dont unzone it, I get a red screen once the build finishes. ok. rpoblem is the SAN.

So, Ive done my researching around already and Ive done what rftm-ed says and taken out all the drivers for the san in the boot up images.

This works brilliantly as it ignores all the san luns and builds nicely. Only problem is, half the post script commands are not working. Most noteably all the vimsh commands which use to work. e.g. vimsh -n -e "/hostsvc/vmotion/vnic_set portgroup2" which sets the nic to vmotion enabled.

thanks for any help.

[azn2kew]

Can you post your script for us to check and specify which commands in VIMSH doesn't work?

[Yattong]

The thing that is mind boggling is why it seems to not want to work now... when it did work before when I unzone the server. Only other difference is taking out the hba drivers.

Whilst this is still a fresh question, when I change the installation method to ftp instead of CDrom, the server finishes building. But the local vmfs partition is corrupt. As in, it does not appear in the local storage, but when I try to add storage, I can see it in the pop up window that it exists. Via putty, it doesnt exist in the /vmfs/volumes. Im unable to delete it and re-create it. If i take out the line which creates the local vmfs, I'm still unable to create it after the build is finished.

All the lines starting with vimsh do not seem to work btw...

(except the enter maintenance mode one which worked. As it allowed me to patch the server automatically)

aargghh... why is every # turned into a 1. indented??

1. Auto-Generated Scripted Install Configuration file.

2. This file is used for VMware ESX Server Scripted Install Deployment

1. Installation Method

cdrom

1. root Password

rootpw --iscrypted encrptedpassword

1. Authconfig

auth --enableshadow --enablemd5

1. BootLoader ( The user has to use grub by default )

bootloader --location=mbr

1. Timezone

timezone Europe/London

1. X windowing System

skipx

1. Install or Upgrade

install

1. Text Mode

text

1. Network install type

network --device eth4 --bootproto static --ip x.x.x.x --netmask 255.255.255.0 --gateway x.x.x.x --nameserver x.x.x.x --hostname esx012.test.co.uk --addvmportgroup=0 --vlanid=0

1. Language

lang en_US

1. Langauge Support

langsupport --default en_US

1. Keyboard

keyboard uk

1. Mouse

mouse none

1. Reboot after install ?

reboot

1. Firewall settings

firewall --disabled

1. Clear Partitions

clearpart --all --initlabel --drives=cciss/c0d0

1. Partitioning

part /boot --fstype ext3 --size 256 --ondisk cciss/c0d0

part / --fstype ext3 --size 10245 --ondisk cciss/c0d0

part swap --size 1647 --ondisk cciss/c0d0

part /var/log --fstype ext3 --size 4096 --ondisk cciss/c0d0

part /tmp --fstype ext3 --size 4096 --ondisk cciss/c0d0

part /opt --fstype ext3 --size 4096 --ondisk cciss/c0d0

part None --fstype vmfs3 --size 10000 --grow --ondisk cciss/c0d0

part None --fstype vmkcore --size 100 --ondisk cciss/c0d0

1. VMware Specific Commands

vmaccepteula

vmlicense --mode=server --server=27000@x.x.x.x --edition=esxFull --features=backup

%packages

@base

%vmlicense_text

%post

#1st post installation script. This should have constants.

cat > /tmp/post.sh <<\EOF1

#

#

1. *constants*

#

#

host="012"

ip="x.x.x.x"

vip="x.x.x.x"

1. Restarting the shell

service sshd restart

sleep 3

1. Change Service Console

esxcfg-vswif -i $ip -n subnetmask -b x.x.x.x vswif0

esxcfg-vswif -E vswif0

1. Add to host file

echo -e "$ip\t\tesx$host.test.co.uk" >> /etc/hosts

1. Change Service Console nic too 100 FULL

esxcfg-nics -s 100 -d full vmnic0

esxcfg-vswitch -L vmnic3 vSwitch0

esxcfg-nics -s 100 -d full vmnic3

service mgmt-vmware restart

sleep 20

vimsh -n -e "hostsvc/net/vswitch_setpolicy --nicorderpolicy-active=vmnic0 --nicorderpolicy-standby=vmnic3 vSwitch0"

1. Create a vSwitch1 with a port group of VM Kernel and Vmotion using vmnic1

esxcfg-vswitch -a vSwitch1

esxcfg-vswitch -A VMkernel vSwitch1

esxcfg-vswitch -L vmnic5 vSwitch1

esxcfg-nics -s 1000 -d full vmnic5

esxcfg-vmknic -a VMkernel -i $vip -n 255.255.255.0

1. BUT this does - By using vimsh together with the portgroup internal ID

2. A restart of the management service and a sleep is required...

service mgmt-vmware restart

sleep 20

vimsh -n -e "/hostsvc/vmotion/vnic_set portgroup2"

1. Create a vSwitch2 with a port group of VLAN 945/6/7 using vmnic2 and vmnic4

esxcfg-vswitch -a vSwitch2

esxcfg-vswitch -A "VLAN 451" vSwitch2

esxcfg-vswitch -p "VLAN 451" vSwitch2 -v 451

esxcfg-vswitch -A "VLAN 461" vSwitch2

esxcfg-vswitch -p "VLAN 461" vSwitch2 -v 461

esxcfg-vswitch -A "VLAN 471" vSwitch2

esxcfg-vswitch -p "VLAN 471" vSwitch2 -v 471

esxcfg-vswitch -L vmnic2 vSwitch2

esxcfg-nics -s 1000 -d full vmnic2

esxcfg-vswitch -L vmnic4 vSwitch2

esxcfg-nics -s 1000 -d full vmnic4

service mgmt-vmware restart

sleep 20

vimsh -n -e "hostsvc/net/vswitch_setpolicy --nicorderpolicy-active=vmnic2 --nicorderpolicy-standby=vmnic4 vSwitch2"

vimsh -n -e "hostsvc/net/vswitch_setpolicy --securepolicy-macchange=false vSwitch2"

vimsh -n -e "hostsvc/net/vswitch_setpolicy --securepolicy-forgedxmit=false vSwitch2"

vimsh -n -e "hostsvc/net/vswitch_setpolicy --shapingpolicy-enabled=true --shapingpolicy-average-bandwidth=104859998 --shapingpolicy-peak-bandwidth=104859998 --shapingpolicy-burst-size=104859998 vSwitch2"

1. Add 2nd DNS settings and Search domain

rm -f /etc/resolv.conf

cat > /etc/resolv.conf <<\DNS

nameserver x.x.x.x

nameserver x.x.x.x

search test.co.uk donttest.co.uk

DNS

1. Add user dpt

useradd -G users -c username -p 'encrpytedpassword' username

1. This section configures SC Memory to 800

mv -f /etc/vmware/esx.conf /tmp/esx.conf.bak

sed -e 's/boot\/memSize = \"272\"/boot\/memSize = \"800\"/g' /tmp/esx.conf.bak >> /etc/vmware/esx.conf

mv -f /boot/grub/grub.conf /tmp/grub.conf.bak

sed -e 's/uppermem 277504/uppermem 818176/g' -e 's/mem=272M/mem=800M/g' /tmp/grub.conf.bak >> /boot/grub/grub.conf

sleep 3

1. Reset rc.local to original

mv -f /etc/rc.d/rc.local.sav2 /etc/rc.d/rc.local

reboot

exit

EOF1

#2nd post installation script generic configs

cat > /tmp/post2.sh <<\EOF2

1. Allow firewall holes outbound for ftp transfer

esxcfg-firewall -o 21,tcp,out,ftp

esxcfg-firewall -e ftpClient

esxcfg-firewall --allowOutgoing

sleep 5

service mgmt-vmware restart

sleep 20

1. dowload files for HP Insight Manager.

ftp -n x.x.x.x <<SCRIPT

user username password

cd esx/hp

binary

prompt

lcd /tmp

mget hpmgmt*

sleep 10

1. download updates patch script

cd ..

lcd /var

mget esx-autopatch-http.pl

sleep 5

quit

SCRIPT

esxcfg-firewall --blockOutgoing

cd /

tar -zvxf /tmp/hpmgmt.gz

sleep 20

cd /hpmgmt/*

./install*.sh --silent --inputfile /tmp/hpmgmt.conf

sleep 10

1. Update ESX server

cd /var

chmod +x esx-autopatch-http.pl

vimsh -n -e /hostsvc/maintenance_mode_enter

sleep 5

./esx-autopatch-http.pl

sleep 20

vimsh -n -e /hostsvc/maintenance_mode_exit

1. Set NTP service

esxcfg-firewall -o 123,udp,out,ntp

esxcfg-firewall -l

sleep 5

service ntpd stop

ntpdate timeserver.co.uk

service ntpd start

hwclock --systohc

sleep 5

1. Reset rc.local to original

mv -f /etc/rc.d/rc.local.sav3 /etc/rc.d/rc.local

echo vmkload_mod -u vmfs2 >> /etc/rc.d/rc.local

reboot

exit

EOF2

1. save a copy of rc.local

cp /etc/rc.d/rc.local /etc/rc.d/rc.local.sav

cp /etc/rc.d/rc.local /etc/rc.d/rc.local.sav2

cp /etc/rc.d/rc.local /etc/rc.d/rc.local.sav3

1. make post.sh executable

chmod +x /tmp/post.sh

chmod +x /tmp/post2.sh

1. add post.sh to rc.local before 1st reboot

cat >> /etc/rc.d/rc.local <<\EOF3

cd /tmp/

/tmp/post.sh

EOF3

1. add post2.sh to rc.local before 2nd reboot

cat >> /etc/rc.d/rc.local.sav2 <<\EOF4

cd /tmp/

/tmp/post2.sh

EOF4

1. add post3.sh to rc.local before 3rd reboot

cat >> /etc/rc.d/rc.local.sav3 <<\EOF5

cd /tmp/

/tmp/post3.sh

EOF5

reboot

[dominic7]

Which version of ESX? ESX 3.5.0 inttroduced new syntax to enable vmotion via vimsh:

1. The way to enable VMotion on a port group has been altered in this

release. The 'new' way to enable VMotion on a VMKernel port group (

assuming the port group that you want to enable VMotion on is the first

VMKernel port group added ) is as follows:

vimsh -n -e "/hostsvc/vmotion/vnic_set vmk0"

[Yattong]

esx 3.0.2 dominic.

Thanks for the tip on 3.5 though, Ive got another project starting on esx 3.5.

[Schorschi]

VMware told me NEVER use vimsh. I went back to using the 'enable' trick and esxcfg-vmknic to add a kernel NIC port to the VMotion vswitch. To be honest, I don't get how VMware can say never use vimsh... it is in the distribution, but since the old way seems to work, we are staying with it. Also, with ESX 3.5, don't forget to use the new -N and -M options with esxcfg-vswitch to explicitly delete/create the uplinks, when we skipped this step we kept getting VMNICs in stand-by mode in bonds.

[dwight]

The installer runs under Linux not VMware ESX. So ESX commands placed in the post section may or may not work. Move your commands to a seperate script that runs on the first boot of the ESX server. An easy way to do this is to name the script as something like /etc/rc3.d/S18firstrun, make sure it is executable and make sure it removes itself from the /etc/rc3.d directory or sets a flag to indicate it should not run again when it completes.

RHCE, VCP

Blog: http://computing.dwighthubbard.info

[Yattong]

I think 'vimsh' is the only way to enable the nic/port vmkernel for vmotion. Note that once this is enabled, the licensing is also enabled automatically.

esxcfg-vmnic will only add the nic to the vmkernel port.

It is already added to a script (post.sh) which runs after esx is built. There are 2 scripts post.sh and post2.sh.

These run via being added to the rc.local which is the last script run after boot up of esx.

Thanks for the help guys, any other ideas?

[Yattong]

I am going to try and put all the vimsh commands in a 3rd post3.sh script and run it on 3rd reboot after installation.

Let you all know how it goes.

[Yattong]

Didnt work.

[Yattong]

I have an update, if anyone can help.

Ive put a sleep after the vimsh command and now I can see the error, which is...

Failed to connect: 514 Error connecting to hostd-vmdb service instance.

So, what service do I need to start?

[snapper]

Have you tried replacing the offending vimsh commands with:

vmware-vim-cmd hostsvc/ .. &lt; same commands &gt;

This command appears to work a little better than the vimsh stuff, and has a proper help and lists the objects that can be managed (yay).

Cheers,

SP

[Yattong]

Ive fixed it.

Needed a longer sleep time after the restart of the mgmt-vmware service. 20 seconds just seems to be not long enough.

I'll give the other command a go and see if it works better, but at last... I have a script that works in most scenarios.

Thanks

[mphodge]

What's the syntax for esxcfg-vswitch with the -N and -M options?

When I add a new vmnic to a vSwitch with esxcfg-vswitch -L, it comes up in stand by mode!

I then have to modify the esx.conf file to get the vmnics active...

[Schorschi]

esxcfg-vswitch --help or just esxcfg-vswitch

# esxcfg-vswitch

esxcfg-vswitch [vswitch[:ports]]

-a|--add Add a new virtual switch.

-d|--delete Delete the virtual switch.

-l|--list List all the virtual switches.

-L|--link=pnic Set pnic as an uplink for the vswitch.

-U|--unlink=pnic Remove pnic from the uplinks for the vswitch.

-M|--add-pg-uplink Add an uplink to the list of uplinks for a portgroup

-N|--del-pg-uplink Delete an uplink from the list of uplinks for a portgroup

-p|--pg=portgroup Specify a portgroup for operation

Use ALL to set VLAN IDs on all portgroups

-v|--vlan=id Set vlan id for portgroup specified by -p

0 would disable the vlan

-c|--check Check to see if a virtual switch exists.

Program outputs a 1 if it exists, 0 otherwise.

-A|--add-pg=name Add a new portgroup to the virtual switch.

-D|--del-pg=name Delete the portgroup from the virtual switch.

-C|--check-pg=name Check to see if a portgroup exists. Program

outputs a 1 if it exists, 0 otherwise.

-B|--set-cdp Set the CDP status for a given virtual switch.

To set pass one of "down", "listen", "advertise", "both".

-b|--get-cdp Print the current CDP setting for this switch.

-m|--mtu=MTU Set MTU for the vswitch. This affects all the nics attached on the vswitch.

-r|--restore Restore all virtual switches from the configuration file

(FOR INTERNAL USE ONLY).

-h|--help Show this message.

#

[mphodge]

erm.. thanks I figured out the list of options... I was kinda hoping for an example ?:|

[demz]

I didn't fix the problem, even with a sleep 40, how much seconds did you put in your sleep Yattong ?

However, I'll start a new thread on that because I getting crazy with my vimsh commands in my ks.cfg !

[Yattong]

Hey Demz

You may need to put a sleep in after the vimsh command, say 60 secs or a pause so that you can see what the error is showing.

You can also post the vimsh cmds that arent working and I'll test them when I can on my own kit. No guarentee's though, bit busy at the moment... too much work!

Good Luck

~y

[demz]

Thanks Yattong, but as I said in the thread http://communities.vmware.com/message/996038, I fixed the issue !

I've got still problem with the security policies like vimsh -n -e "/hostsvc/net/portgroup_set --nicteaming-rollingorder=true vSwitch0 'Service Console'" but I think I'll fix that too until tonight.

Thanks btw