hostasaurus's Posts

Get ready for all kinds of other host profile failures after you get your upgrade done.  There's numerous bugs that will basically leave you stuck with all of your servers out of profile complian... See more...
Get ready for all kinds of other host profile failures after you get your upgrade done.  There's numerous bugs that will basically leave you stuck with all of your servers out of profile compliance, making it nearly useless from a keeping servers in compliance perspective.
I ended up having to do it using Host Profiles.  I created a profile off one of the 4.1 servers, edited as needed to set the various interface names and to prompt me for ip information, then appl... See more...
I ended up having to do it using Host Profiles.  I created a profile off one of the 4.1 servers, edited as needed to set the various interface names and to prompt me for ip information, then applied it back to the 5.1 server, it asked what ip's should be used for the various vnic's and then applied.  It destroyed the config that was there and replaced with the correct dVS setup.
Hi all, I'm working through an upgrade of a 4.1 Enterprise Plus cluster to 5.1.  Thus far, I've gotten vCenter updated to 5.1 build 947673 and I took one host out of the cluster, wiped it, and di... See more...
Hi all, I'm working through an upgrade of a 4.1 Enterprise Plus cluster to 5.1.  Thus far, I've gotten vCenter updated to 5.1 build 947673 and I took one host out of the cluster, wiped it, and did a fresh install of ESXi 5.1, then patched up to current build 914609.  I added it back in as a new host to vCenter.  I updated a quantity of v4 cpu licenses to v5 and applied the new key.  All licenses are enterprise plus. I went to Home -> Inventory -> Networking and attempted to add the new host into one of the distributed virtual switches but it does not show in the list of available hosts, nor does it show in the list of incompatible hosts; both are empty.  It's ports don't show up as available to add to a port group. Any ideas?  Can you not mix 4.1 and 5.1 hosts in a distributed switch?  Hopefully that's not hte issue or that will make upgrading difficult. Thanks!
I wasted a bunch of time today figuring my way around numerous issues so I figured I'd post about it in hopes Google picks it up and others find the thread so they don't waste as much time. I ... See more...
I wasted a bunch of time today figuring my way around numerous issues so I figured I'd post about it in hopes Google picks it up and others find the thread so they don't waste as much time. I have a CentOS 5.5 x86_64 (aka RedHat Enterprise 5) system that had been set up with vCLI to communicate with a vCenter 4.1 server and numerous ESXi 4.1 hosts, mostly just for command line resetting of vm's, etc.  We upgraded vCenter to 5.1 and will soon be upgrading the hosts so the old vCLI commands stopped functioning. I downloaded the VMware-vSphere-CLI-5.1.0-780721.x86_64.tar.gz file from the vCLI 5.1 download page in my portal.  It extracts to vmware-vsphere-cli-distrib.  Vmware's documentation would have you believe installing everything is as simple as running vmware-vsphere-cli-distrib/vmware-isntall.pl and it takes care of the rest for you.  What it actually did is correctly recognize that I had a version 4.1 install of vCLI and told me it would need to be removed before the 5.1 install goes on and would I like it to do that for me?  Sure.  It spits out a license agreement and then acts as if everything went fine when all it apparently did was uninstall my 4.1 commands and did nothing else. So, here's the process of getting things working again that I had to took; some steps may be unecessary and/or redundant, but no way for me to know which ones were after the fact, just that it works now: You're probably going to be hand installing some CPAN perl libraries, so if you mount your system's /tmp directory with the 'noexec' flag, you'll want to temporarily remount it with noexec removed because the perl CPAN installer compiles things in /tmp Use yum to install libxml2-devel, perl-Crypt-SSLeay, expat-devel and perl-Archive-Zip; i.e.: yum install libxml2-devel perl-Crypt-SSLeay expat-devel perl-Archive-Zip These are all standard RHEL packages.  The installer stupidly needs these for the perl library install portion to complete but doesn't bother to tell you this, doesn't attempt to install them and just fails silently. For safe measure, I removed /etc/vmware-vcli/ to clear out any config related to the old install.  Normally this is done with the vmware-uninstall-vSphere-CLI.pl script before installing the new software, but I didn't know that until the new software had already removed most of my 4.1 install. Things still weren't working so I tried to run the perl module installer part manually thinking maybe that's why nothing else (the binaries) was installing either.  That gave me a bunch of missing perl libraries (some of which I included in the above yum installer steps if there is a pre-packaged RHEL version).  The rest we'll need to install manually: Run this:  perl -MCPAN -e shell That will give you an interactive perl CPAN install shell.  From there, you can type: install Class::Inspector install ExtUtils-MakeMaker install SOAP::Lite install Task::Weaken install Test::Simple install UUID install XML::LibXML install XML::NamespaceSupport install XML::Parser install XML::SAX install XML::SAX-Base Some of these are pre-reqs for one or more of each other, so as you go through the list, you may find it asks you to install one or more of them to complete installing one of the others, so then when you go to install the one in question, it will tell you its already up to date; no harm.  Now, some of these will fail to install for whatever reason, and when that occurs, you'll need to exit the interactive shell (type ctrl-D) and go to /root/.cpan/build/ and then into the directory of the module file in question.  For example, UUID will fail, so you'll end up going to /root/.cpan/build/UUID-0.05/ and in there you'll want to type:  perl Makefile.PL followed by make followed by 'make install'  Several of the modules need to be installed manually in that manner.  Each time you encounter one that requires the manual install, just complete it and then go back into the CPAN shell and install the next module. Once all those happy pre-reqs have been installed, go back to your vmware-vsphere-cli-distrib directory and run:  perl Makefile.PL followed by make if it was successful, followed by 'make install' if make was successful.  At that point, it should finally install all of the relevant vmware perl libraries in the appropriate places on the system. Now you can finally run the original vmware-install.pl file from the same vmware-vsphere-cli-distrib directory and it will install the binaries correctly. To finish, if you have written your own scripts that depend on the old 4.1 sample scripts, you'll want to: mkdir /usr/local/lib/vmware-vcli/ mv -f vmware-vsphere-cli-distrib/apps /usr/local/lib/vmware-vcli/
Adding to the previous poster's suggestions; are you using the pvscsi disk adapter?  I found that that made a huge difference for us compared to the sas emulated controller. I'm curious about ... See more...
Adding to the previous poster's suggestions; are you using the pvscsi disk adapter?  I found that that made a huge difference for us compared to the sas emulated controller. I'm curious about how the SAN is connected too, how many spindles, what kind of hardware, etc..  We have an EMC CX4 connected to our ESXi hosts via 10 gig iSCSI and I can easily achieve 250 MB/sec write speeds and 500 MB/sec read speeds at the guest level, however, we're not doing raw devices because when I tested that, it offered what I think was actually lower performance, so unless I need to use a raw disk for something like Oracle, didn't seem to give me any real benefit.
I haven't made any progress on this with vmware support and have not been able to get the pvscsi kernel library from version 5 (of the OS) to work in version 6 since that's still what I think the... See more...
I haven't made any progress on this with vmware support and have not been able to get the pvscsi kernel library from version 5 (of the OS) to work in version 6 since that's still what I think the issue is.  Support did mention there being a known performance issue with RHEL 6.1 and pvscsi, which alarms me even more since I'm already having horrible performance on CentOS 6.0 as they do not have RHEL 6.1 rebuilt yet, so now it sounds like things could get even worse.  They said the solution to that issue is to run the older kernel until it gets figured out but could not give me any specifics.  They did point me to the ESXi v5 tools install pdf: http://packages.vmware.com/tools/docs/manuals/osp-esxi-50-install-guide.pdf which has a useful article on how to override kernel modules with ones from the tools package instead.  I've been trying to get that to work using the pvscsi.ko from the centos 5 tools kernel mod package but have not had any success getting a bootable system yet, or one that uses the other module file... So no ideas at this point other than going back to CentOS 5 until someone else figures it out.
I'd probably just do raid 50 across the whole thing with two hot spares; so you'd get the effective usable capacity of four drives, which is what you would have gotten anyway with a two disk raid... See more...
I'd probably just do raid 50 across the whole thing with two hot spares; so you'd get the effective usable capacity of four drives, which is what you would have gotten anyway with a two disk raid 1, four disk raid 5 and two hot spares if your controller won't let you use one hot spare for both arrays, but you get much better performance in that configuration and more flexibility on the vmware side. If space is critical, AND you can pool the hot spare for both arrays, then I'd go with your original idea. I'd still do vmfs on both though rather than the raw device; gives you more flexibility and I think you'll find the performance will be the same either way. Regarding performance though; I've found ESXi 4.1 with direct attached storage performs pretty horribly at the guest level compared to the native OS running on the same exact server.  For example, before going with a SAN, and vmware for that matter, I tested our normal redhat enterprise 5 configuration's native performance on a Dell server we were considering for our cluster with a six drive raid 5 configuration.  Then installed ESXi and deployed the same RHEL 5 'server' as a guest and benchmarked it using both the SAS and paravirtual scsi controllers in the guest (pvscsi only on the data partition since you can't boot rhel 5 off it) and we were lucky if we saw half the performance.  I nearly abandoned the idea of virtualization because I thought that was the issue, then someone suggested trying it on the SAN we were building for something else and instead of lower performance, we got dramatically better performance in the guest than we could have achieved using direct attached storage and the native OS thanks to the SAN's performance. Just food for thought.
I also do exactly what jfield was saying, and ran into the exact same issue. We have ESXi 4.1 and use 1 MB block size on our VMFS volumes because most of our guests have no need to go beyond 1... See more...
I also do exactly what jfield was saying, and ran into the exact same issue. We have ESXi 4.1 and use 1 MB block size on our VMFS volumes because most of our guests have no need to go beyond 100 GB, and even the unusual ones don't go beyond a few hundred gigs.  They also don't grow in size very quickly.  In any case, sometimes customers will upload huge amounts of data and then delete them, so I found a bunch of our guests wasting a lot of room.  I used dd to fill the guest disk with zeroes, migrated from thin to thick and back, one LUN to another and back on our EMC, no change.  I created a new LUN and attached our ESXi hosts to it, formatted it as a 2 MB VMFS.  Did the exact same migration, thick to thin, and it recovered the space.  Now we keep the 2 MB LUN around just for moving guests to and from when we need to shrink them. So yeah, like jfield said, if you're using 8 MB out of necessity for vmdk's larger than the lower sizes support, you're screwed if you need to recover that space without a long outage and a lot of pain.  Or get a more advanced storage system that can dedupe as an ugly workaround so you can thin provision below the vmware level.
Hi all, we have an vsphere ESXi 4.1 cluster talking 10 gig iscsi to an EMC CX4.  For the past year we've been deploying servers based on centos 5 x86_64 where I'd do the install of boot and root ... See more...
Hi all, we have an vsphere ESXi 4.1 cluster talking 10 gig iscsi to an EMC CX4.  For the past year we've been deploying servers based on centos 5 x86_64 where I'd do the install of boot and root using the emulated 'lsi logic sas' controller and ext3 filesystem.  I'd also add a paravirtual controller and second hard disk in vsphere, after the OS is installed I'd install the vmware tools and ext4 drivers then format the second drive ext4 and mount it under /var for data. With this configuration, I have been able to consistently achieve about 260 MB/sec write speeds, 500 MB/sec read speeds and anywhere from 7000 to 11,000 IOPS depending on the number of drives in the LUN on the EMC. So, out comes RHEL/CentOS v6 x86_64 and low and behold, pvscsi is built in; that's great news.  I've deployed a few of those where now all I have to do is go back to provisioning one hard drive to the guest because the kernel has the pvscsi loaded at boot.  I'm still doing a /var partition in linux for the data but now all the partitions are ext4 and pvscsi instead of just /var. I never bothered to do any performance testing when we began to deploy these new guests because I didn't notice anything odd and figured it wouldn't perform any differently than v5.  Well, we started to see issues when one of the guests got loaded up with I/O and not an amount that would have ever caused issues before.  I started doing some testing and with the exact same guest configuration (memory and vCPU) on the same ESXi host with its data on the same LUN on the EMC, I can't do better than about 70 to 80 MB/sec write, 250 MB/sec read, 800 IOPS.  So my write has dropped by nearly 75%, read by 50% and IOPS are down tenfold. I can't find anything to explain this other than the pvscsi module installed in CentOS 5 by vmware tools reports itself as 1.0.1.1 and the kernel module included in CentOS/RHEL 6 reports itself as 1.0.1.0-k.  I can't find much of any reference online to the versions or their differences so I have no idea if that could be the problem but I can't come up with any other ideas. Anyone else seen this issue?  Any ideas for me to try? Thanks!
Matt Liebowitz wrote: The shrink option isn't available in thin provisioned disks.  One thing I have seen recently with 4.1 is that svMotion won't reclaim space on a thin provisioned disk (ev... See more...
Matt Liebowitz wrote: The shrink option isn't available in thin provisioned disks.  One thing I have seen recently with 4.1 is that svMotion won't reclaim space on a thin provisioned disk (even after sdelete) if you are moving between datastores of the same block size.  If you can, try doing the svMotion between datastores of different block sizes and see if it resizes properly. That was it.  I deleted one of my LUNs and recreated it as a 2 MB block size instead of 1 MB like all the rest, migrated to it and now the vm dropped down to the same size as what's really in use.  Thanks!
I have an empty LUN I could re-format to a new block size.  Before I do that, if I change it back to thick again, will the shrink option show up in the tools?
I have a Win2k8 R2 guest on esxi 4.1 enterprise plus with thin provisioned vmdk.  The customer let some log files get out of hand to the tune of about 80 GB of wasted space.  I cleaned it up and ... See more...
I have a Win2k8 R2 guest on esxi 4.1 enterprise plus with thin provisioned vmdk.  The customer let some log files get out of hand to the tune of about 80 GB of wasted space.  I cleaned it up and then ran the sdelete utility with the -c option to zero the free space.  I then live migrated to a new data store converting to thick, and back again converting to thin.  The end result was only reducing the size by about 4 GB.  For some reason the other 76 GB of wasted space has not been recovered.  I tried with the guest on and off. Any other tool I can try or am I missing a critical step?
Yep, the default x86_64 kernel is smp.  When the bootup fails and I go into it in single user mode, both cpu's are visible.
Check the switch(s) the new storage connects to for any errors.  I was having the same slow rescans with our EMC storage and it turns out it was just a dirty fiber connector on one of the four te... See more...
Check the switch(s) the new storage connects to for any errors.  I was having the same slow rescans with our EMC storage and it turns out it was just a dirty fiber connector on one of the four ten gig paths to the storage; it was producing CRC errors that the switch was recording.  The percentage was pretty small but steady.  Pulled it, cleaned it, problem went away.
Some additional info.  If I create a machine with two vCPU's, do the same install, tools, reboot, switch to pvscsi/vmxnet3, reboot, we're good.  Shut down, reconfigure for one vcpu, boot up, work... See more...
Some additional info.  If I create a machine with two vCPU's, do the same install, tools, reboot, switch to pvscsi/vmxnet3, reboot, we're good.  Shut down, reconfigure for one vcpu, boot up, working fine.  Shutdown again, configure back to two vCPU's, we're still good.  So whatever the issue is, it can at least be avoided for future guests by doing the initial installs with multiple cpu's and then taking one away if the guest is not one that should have more than one cpu initially.
Hello all, I've got a centos 5.5 x86_64 guest with 12 GB memory and one vCPU.  It has its primary partitions like / and /usr on ext3 the virtual LSI Logic SAS adapter, then after initial install,... See more...
Hello all, I've got a centos 5.5 x86_64 guest with 12 GB memory and one vCPU.  It has its primary partitions like / and /usr on ext3 the virtual LSI Logic SAS adapter, then after initial install, I install vmware tools and move /var and /tmp to pvscsi adapter along with the network card being replaced with vmxnet3.  This is on vsphere esxi 4.1 enterprise plus. I needed to increase processing power for this guest so I increased cpu count from 1 to 2 and now the machine won't make it through boot; it starts the boot process fine, but when it gets to the part of mounting the pvscsi partitions it fails as if they're not there.  I enter single user mode and they'll mount fine.  The vmxnet3 eth0 won't work either, but if I down it and re-up it, it will start working.  It's almost as if by adding the second cpu, the vmware tools and associated drivers are now loading out of order.  So I switched it back to single cpu, it boots fine, I downloaded all the vmware tools rpm's manually, switch back to 2 cpu's, enter single user, manually mount /var and /tmp, remove the vmware tools, add them back in from the rpm's but that didn't fix the issue.  I was hoping maybe by reinstalling when two cpu's were present, whatever is causing them to fail would be wiped out and replaced. The generic error I get at boot is the "no such file or directory while trying to open /dev/sdb1: the superblock could not be read or does not describe a correct ext3 filesystem...."  That's the first of the pvscsi partitions.  Swithc it back to one vCPU, no problems. Any ideas?