VMware Communities : Document List - General

List of Procedures undergoing Research and Development - Update 20091121

Sat, 21 Nov 2009 20:34:57 GMT

Here is the list of Procedures, with 3 letter identifier, being researched and developed in my 3BX blog. The status of each procedure is noted: P-Plans for future; O-Open; C-Completed; S-stopped until a question answered or some other obstacle removed; D-Deprecated

******************
Open - 1
******************
O - WIP
Wiki Platform
Inception 2009 / Opened 20091121 / Procedure first complete X / Procedure count X
Notes:

******************
Completed - 4
******************
O - MAP (4)
Mathematica Platform
Inception 20091109 / Opened 20091109 / Procedure first complete 20091112 / Procedure count 1

C - BES (3)
Backup Virtual Machine to External Storage
Opened 20091108 / Procedure first complete 20091109 / Procedure count 1

C - GUI (2)
Graphical Network Simulator on Ubuntu server platform installation
Opened 20091013 / Procedure first complete 20091013 / Procedure count 1

C - UBP (1)
Ubuntu platform build for graphical network simulator GNS
Opened 20091012 / Procedure first complete 20091013 / Procedure count 1
Notes:

******************
Stopped - 0
******************

******************
Planned - 3
******************
P - RES
Restore Virtual Machine from External Storage backup
Inception 20091109 / Opened X / Procedure first complete X / Procedure count X
Notes:
This is the necessary corollary procedure to BES. The procedure will be tested on the Wiki platform created at the completion of the WIP procedure.

P - JOS
Build a VMware platform for Olive to simulate the Juniper Operating System
Inception 20091004 / Opened X / Procedure first complete X / Procedure count X

P - NIT
Reconfigure system to allow Network Interface Trunking
Inception 20091004 / Opened X / Procedure first complete X / Procedure count X

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

******************
Deprecated - 1
******************
D - SES
Snapshot external storage - How do I store snapshots on external storage?
Opened 20091013 / Procedure first complete X / Procedure count 0
Notes: This was deprecated in favor of procedure BES

Dell PowerEdge 1855 Blade

Sat, 21 Nov 2009 16:09:07 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	Dell
Product	PowerEdge 1855
Product Version	Blade
Tell Us About the Configuration You Tested
VMware Infrastructure	ESXi 4.0.1, ESXi 4.0, ESX 4.0.1, ESX 4.0, ESXi 3.5u4, ESX 3.5u4
Configuration Tested	Old blade with: 2 old Xeon (no VT support, but 64 bit support) 4 GB RAM
Integration Method
Tell Us About the Support Provided
Submitter	AndreTheGiant
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues	Not in vSphere HCL, but it works
Additional Comments

Community-Supported Hardware/Software List

HP HP Proliant ML 570 G3 G3

Sat, 21 Nov 2009 16:01:23 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	HP
Product	HP Proliant ML 570 G3
Product Version	G3
Tell Us About the Configuration You Tested
VMware Infrastructure	ESXi 3.5u4, ESXi 3.5u3, ESXi 3.5u2, ESX 3.5u4, ESX 3.5u3, ESX 3.5u2
Configuration Tested	2 old Xeon MP 12 GB RAM
Integration Method
Tell Us About the Support Provided
Submitter	AndreTheGiant
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues
Additional Comments	See also: ESX4 / ESXi 4 on Proliant 570 G4

Community-Supported Hardware/Software List

Shuttle SX58H7 -

esloof@ntpro.nl — Fri, 20 Nov 2009 19:56:09 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	Shuttle
Product	SX58H7
Product Version	-
Tell Us About the Configuration You Tested
VMware Infrastructure	ESX 4.0
Configuration Tested	Yes
Integration Method	• Processor Intel® Core i7 975 Extreme 1366 • DDR3 1333MHz 4x4GB • Storage Transcend 32GB TS32GSSD25S-M SATA • SSD Bracket Scythe Twin Mounter (2x 2,5" in one 3,5") • Network Intel® PRO/1000 PT Quad Port Server • Graphic Point VGA 8400GS 512MB PCIe
Tell Us About the Support Provided
Submitter	esloof
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues
Additional Comments	http://www.ntpro.nl/blog/archives/1331-A-True-Nehalem-Whitebox-for-ESX4.html

Community-Supported Hardware/Software List

Asus Bare bone V3-P5G45

esloof@ntpro.nl — Fri, 20 Nov 2009 19:53:46 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	Asus
Product	Bare bone
Product Version	V3-P5G45
Tell Us About the Configuration You Tested
VMware Infrastructure	ESXi 4.0
Configuration Tested	Yes
Integration Method	•Processor Intel(R) Core(TM) Quad CPU Q8200 @ 2.33 GHz •Memory GeiL 8 GB DDR2-667 Kit •Storage Samsung HD501LJ 500GB Internal SATA 16MB 7200RPM Hard Drive •Network Intel® PRO/1000 PT Dual Port Server Adapter •DVD-ROM Samsung SA-D163
Tell Us About the Support Provided
Submitter	esloof
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues	Onboard NIC not spported
Additional Comments	http://www.ntpro.nl/blog/archives/930-New-ESX-WhiteBox-Asus-V3-P5G45.html

Community-Supported Hardware/Software List

Asus Bare bone V2-P5G33

esloof@ntpro.nl — Fri, 20 Nov 2009 19:52:03 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	Asus
Product	Bare bone
Product Version	V2-P5G33
Tell Us About the Configuration You Tested
VMware Infrastructure	ESX 4.0
Configuration Tested	Yes
Integration Method	•Processor Intel(R) Core(TM) Quad CPU Q9400 @ 2.66 GHz •Memory OCZ 8 GB DDR2-800 Kit •Storage Samsung HD501LJ 500GB Internal SATA 16MB 7200RPM Hard Drive •Network Intel® PRO/1000 PT Dual Port Server Adapter •DVD-ROM Samsung SH-D163
Tell Us About the Support Provided
Submitter	esloof
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues	NIC not supported
Additional Comments	http://www.ntpro.nl/blog/archives/1067-A-VMware-High-Availability-Cluster-at-Home.html

Community-Supported Hardware/Software List

HP BL460c G1

Wed, 18 Nov 2009 06:15:23 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	HP
Product	BL460c
Product Version	G1
Tell Us About the Configuration You Tested
VMware Infrastructure	ESXi 4.0, ESX 4.0, ESXi 3.5u4, ESXi 3.5u3, ESXi 3.5u2, ESXi 3.5u1, ESXi 3.5, ESX 3.5u4, ESX 3.5u3, ESX 3.5u2, ESX 3.5u1, ESX 3.5
Configuration Tested
Integration Method
Tell Us About the Support Provided
Submitter	Anton V Zhbankov
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues
Additional Comments

Community-Supported Hardware/Software List

Areca ARC 1110

Wed, 18 Nov 2009 01:43:33 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	Areca
Product	ARC
Product Version	1110
Tell Us About the Configuration You Tested
VMware Infrastructure	ESXi 4.0.1, ESXi 4.0
Configuration Tested	Areca ARC-1110 4-Port PCI-X to DATA RAID Controller (4x SATA300 RAID, XOR, 128MB, PCI-X) Asus NCCH-DL,82875P (LAN, SATA-RAID,PCI-X) 4x Samsung Spinpoint F1 DT HD753LJ, 750GB SATA-300, 32MB, 7200rpm
Integration Method	More config info: 2x Intel Xeon 2.8GHz (604, Single, 2.8GHz, 1MB, 4x200MHz FSB, Boxed) 4x 512MB Kingston Memory (PC-3200)
Tell Us About the Support Provided
Submitter	j15
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues	With firmware v1.44, the device wasn't recognized; requires firmware v1.46
Additional Comments

Community-Supported Hardware/Software List

Dell Optiplex 755

Wed, 18 Nov 2009 01:19:41 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Tell Us About the Product

Vendor	Dell
Product	Optiplex
Product Version	755
Tell Us About the Configuration You Tested
VMware Infrastructure	ESXi 4.0.1, ESXi 4.0
Configuration Tested	Intel Core2Duo E6850 @ 3.00GHz 4GB RAM
Integration Method
Tell Us About the Support Provided
Submitter	j15
Vendor Representative?	no
Configuration Supported?	no
Provider Name
Provider Contact
Provider URL
Support End Date
Known Issues	None
Additional Comments	works well

Community-Supported Hardware/Software List

Test

Wed, 11 Nov 2009 04:43:13 GMT

Test

List of Procedures undergoing Research and Development

Tue, 10 Nov 2009 17:29:12 GMT

******************
Open - 1
******************
O - MAP
Mathematica Platform
nception 20091109 / Opened 20091109 / Procedure first complete X / Procedure count X

******************
Completed - 3
******************
C - BES (3)
Backup Virtual Machine to External Storage
Opened 20091108 / Procedure first complete 20091109 / Procedure count 1

C - GUI (2)
Graphical Network Simulator on Ubuntu server platform installation
Opened 20091013 / Procedure first complete 20091013 / Procedure count 1

C - UBP (1)
Ubuntu platform build for graphical network simulator GNS
Opened 20091012 / Procedure first complete 20091013 / Procedure count 1
Notes:

******************
Stopped - 0
******************

******************
Planned - 2
******************
P - RES
Restore Virtual Machine from External Storage backup
Inception 20091109 / Opened X / Procedure first complete X / Procedure count X

P - JOS
Build a VMware platform for Olive to simulate the Juniper Operating System
Inception 20091004 / Opened X / Procedure first complete X / Procedure count X

P - NIT
Reconfigure system to allow Network Interface Trunking
Inception 20091004 / Opened X / Procedure first complete X / Procedure count X

VI 3.5 vs vSphere 4.0

Thu, 05 Nov 2009 05:44:54 GMT

VI 3.5 and vSphere 4.0

What’s New with VMware vSphere 4 - http://www.vmware.com/products/vsphere/upgrade-center/
What's New in VMware vSphere 4.0 - http://www.vmware.com/support/vsphere4/doc/vsp_40_new_feat.html
VMware vSphere Editions Comparison - http://www.vmware.com/products/vsphere/buy/editions_comparison.html

Pro and cons

VI 3.5 pro

Does not require a "new" x64 CPU
Support old hardware - See HCL (http://www.vmware.com/resources/compatibility)
Less resources for ESX (1 GB vs 2 GB requirement)
Less resources for vCenter Server (3 GB vs 3 GB requirement)
Free ESXi can be managed by vCenter Server
VI 3.5 has the disk hot add feature also in free version (on vSphere you need at least the Advanced version)
Lab Manager 3 is supported only on VI 3.5 (but now there is also LM 4)
SRM 1 is supported only on VI 3.5 (but now there is also SRM 4)
View 3 is supported only on VI 3.5 (but now there is also View 4)
3th part tools may work only on VI 3.5 (but most of them now works also on vSphere)
It can use a 2 TB disk (without the 512B limitation)... ok it's bug... it's not a feature - Re: Disks bigger than 2TB - 512B are not supported.

vSphere pro

Faster on CPU, Mem, I/O - See also http://www.vmware.com/files/pdf/VMware-vSphere-Performance-WP-EN.pdf
VMware Fault Tolerance (only with Advanced, Enterprise and Enterprise Plus) - See also VMware FT
Thin provisioning with GUI (in ESX 3.5 was possible only with CLI)
GUI Storage VMotion (but there was also 3th part GUI for VI 3.5)
Host profiles (only with Enterprise Plus)
vDistributed Network Switches and Cisco Nexus 1000v integration (only with Enterprise Plus)
vShield Zone (only with Advanced, Enterprise and Enterprise Plus)
VMware Data Recovery (only with Essential Plus, Advanced, Enterprise and Enterprise Plus)
Hot pluggable (add/remove devices while VM's online)
Double resources support (ram, memory, etc.)
Support cloud solution & x64 bit architecture & seperate SC VM (vmdk)

Compatibility between products

You need last version of vSphere Client and vCenter Server to manage old and new products.
See also:
http://www.vmware.com/pdf/vsphere4/r40/vsp_compatibility_matrix.pdf - VMware Infrastructure Compatibility Matrixes

Downgrade to VI 3.5

New customers will have only the vSphere licenses.
To downgrade them to the VI 3.5 version see:
http://kb.vmware.com/kb/1010533 - Downgrading a vSphere 4.0 license to a ESX 3.5 license

To evaluete VI 3.5 try to evaluete View 3, you will have also VI 3.5 evaluation licenses.

VMware Communities FAQs

Mon, 02 Nov 2009 22:50:01 GMT

ESXi vs Full ESX

Sat, 31 Oct 2009 10:07:34 GMT

ESX vs ESXi

ESX and ESXi official comparison:
http://kb.vmware.com/kb/1006543 - VMware ESX and ESXi 3.5 Comparison
http://kb.vmware.com/kb/1015000 - VMware ESX and ESXi 4.0 Comparison

Probably the big difference is that ESXi has a POSIX Management Appliance that runs within the vmkernel and ESX has a GNU/Linux Management Appliance that runs within a VM.

ESX vs. ESXi which is better?
http://communities.vmware.com/blogs/vmmeup/2009/04/07/esx-vs-esxi-which-is-better
Dilemma: buy vSphere with ESXi4 or with ESX4?

ESXi vs. ESX: A comparison of features
http://blogs.vmware.com/esxi/2009/06/esxi-vs-esx-a-comparison-of-features.html

ESXi limitations

No official interactive console (there is only a "unsupported" hidden console: http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-articles/general/how-to-access-the-vmware-esxi-hidden-console.html)
No official supported SSH access (there is only a "unsupported" access: http://www.yellow-bricks.com/2008/08/10/howto-esxi-and-ssh/)
No support for some 3th part backup programs (see also Backup solutions for VMware ESXi)
No support for some 3th part programs
No hot-add feature for virtual disk in ESXi 4.0, but only on ESXi 3.5 (or also on ESXi Advanced with hot-add license)
No SNMP "get" polling"
No virtual Serial or Parallel port
No USB disk in the "console"
No support for mount ext2/ext3/ntfs/cifs filesystem in the console
No fully functional esxtop
No fully functional esxcfg-mpath
No vscsiStats tool
No PXE install or scriptable installation (but seems that there is a solution: http://www.mikedipetrillo.com/mikedvirtualization/2008/11/howto-pxe-boot-esxi.html)
Normally with a standalone ESXi host you'll get 1 hour of performance data, the graph should be set to reset every 20 seconds (you can also extend the performance data to 36 hours will a simple hack - http://www.vm-help.com/esx/esx3i/extending_performance_data.php)

Free ESXi limitations

ESXi 4.0 free can not be connected to a vCenter Center (it required a VC agent license)
VCB does not work (it require a VCB license)
Lot of 3th part backup programs does not work with free ESXi (see also: Backup solutions for VMware ESXi)
RCLI and VMware Infrastructure toolkit are limited to read-only access for ESXi free
The free version of VMware ESXi does not support SNMP
The free version of VMware ESXi does not support Active Directory integration at this time
Free ESXi EULA has some interesting restrictions including enforced read only mode for v4 and later versions of v3.5.

See also: What's the difference between free ESXi and licensed ESXi?
http://vmetc.com/2008/08/10/whats-the-difference-between-free-esxi-and-licensed-esxi/
VMware ESX/ESXi EULA - http://www.vmware.com/download/eula/esx_esxi_eula.html

ESXi advantages

Small footprint
Can be installed on a flash card or USB key
Does not use a vmdk for console filesystems (as ESX 4.0 does)

Sources

Serious ESX vs ESXi in the Enterprise
Compare ESXi with ESX?
Re: list of features that you cannot use with ESX4i
Looking for more good comparisons of ESX 4 and 4i
Re: ESXi Free edition - limitations

Serenity Systems International eComStation 2.0 Silver Release

Thu, 29 Oct 2009 08:14:19 GMT

This document is a detail page for the Community-Supported Hardware/Software List.

Product Information

Vendor	Serenity Systems International
Product	eComStation
Product Version	2.0 Silver Release
Configuration Tested
VMWare Infrastructure	ESXi 4.0.1
Configuration Tested	eComStation 2.0 Desktop/server OS - SMP, ACPI, 32Bit
Integration Method	Image created by installing to a new virtual machine
Support Information
Submitter	edurrant1
Vendor Representative?	no
Configuration Supported?	no
Support Provider
Support Provider Contact Info
Support Provider URL
Support End Date
Known Issues	No OS/2 additions available, so mouse integration does not work. Neither does sound or USB however in a server environment that's not so relevant.
Additional Comments	For a full "blow by blow" instruction along with pictures (still being sorted) please refer to: http://ewiki.ecomstation.nl/eCSunderVMWareESXiHowto

Community-Supported Hardware/Software List

VMware VMDirectPath I/O

Thu, 29 Oct 2009 06:10:43 GMT

Introduction

VMDirectPath I/O allows a guest operating system on a virtual machine to directly access physical PCI and PCIe devices connected to a host. Each virtual machine can be connected to up to two PCI devices. PCI devices connected to a host can be marked as available for passthrough from the Hardware Advanced Settings in the Configuration tab for the host.
VMDirectPath allows guest operating systems to directly access an I/O device, bypassing the virtualization layer. This direct path, or passthrough can improve performance for VMware ESXTM systems that utilize high‐speed I/O devices, such as 10 Gigabit Ethernet.

VMware unveils VMDirectPath technology, Intel to support it with Nehalem:
http://www.dabcc.com/article.aspx?id=8474
http://www.virtualization.info/2008/08/vmware-unveils-vmdirectpath-technology.html

Intel releases new CPUs and NICs with nested page tables and I/O virtualization:
http://www.virtualization.info/2009/03/intel-releases-new-cpus-and-nics-with.html

Configuration

Configuring VMDirectPath I/O pass-through devices on an ESX host - http://kb.vmware.com/kb/1010789
Configuration Examples and Troubleshooting for VMDirectPath - http://www.vmware.com/pdf/vsp_4_vmdirectpath_host.pdf

Configuration Examples and Troubleshooting for VMDirectPath
{youtube}http://www.youtube.com/watch?v=WhMkmTqBbUA{youtube}

Configuring VMDirectPath IO with Cisco UCS and vSphere
{youtube}http://www.youtube.com/watch?v=jmQ5Ej8r-aA{youtube}

Requirements

vSphere (well… common)
Intel Virtualization Technology for Directed I/O (VT-d)
AMD IP Virtualization Technology (IOMMU)
Devices must be connected to the host and marked available for pass through
VMs require hardware version 7

VMDirectPath I/O is experimentally supported for the following Storage and Network I/O devices:

QLogic QLA25xx 8 Gb Fibre Channel adapters
Emulex LPe12000 8 Gb Fibre Channel adapters
LSI 3442e-R and 3801e (1068 chip based) 3 Gb SAS adapters
Intel 82598 10 Gigabit Ethernet controller
Broadcom 57710 and 57711 10 Gigabit Ethernet controllers

VMware regularly adds support for new hardware. Check on HLC: http://www.vmware.com/resources/compatibility/search.php

Limitations

When using VMDirectPath you lose following features:

VMotion
Storage VMotion
FT
Device hot add
Suspend and resume
Record and replay

USB virtualization

If your server has support for device pass-through you may be able to provide USB access to Virtual Machines:
Enable USB Support for ESXi with VMDirectPath - http://vm-help.com/esx40i/VMDirectPath/USB_Setup.php
VMware ESXi4 and VMDirectPath - http://www.petri.co.il/vmware-esxi4-vmdirectpath.htm

Note that you give (if it works...) the entire USB chipset... so all the USB devices to a single VM!

Sources

Introduction from SimonLong Blog
http://www.simonlong.co.uk/blog/2009/08/03/vmware-vmdirectpath-io/

VMDirectPath? Paravirtual SCSI? – vSphere VM Options and You!
http://professionalvmware.com/2009/08/vmdirectpath-paravirtual-scsi-vsphere-vm-options-and-you/

Client Hypervisor

Thu, 29 Oct 2009 05:17:30 GMT

Introduction

Client Hypervisors: Why Would I Need One of Those? - http://www.virtual-strategy.com/Features/AppSense-20090721.html
Endpoint virtualization: http://www.neocleus.com/OurVirtualizationApproach.html

Existing solutions

There are several Client Hypervisors out there at the moment, like:

NXtop - http://www.virtualcomputer.com/Products/nxtop
Neucleus - http://www.neocleus.com/ (is easier to deploy as it self virtualises itself)

VMware and Citrix have on in the pipeline:
http://www.vmware.com/company/news/releases/cvp-intel-vmworld.html
http://www.virtualization.info/2009/01/citrix-and-intel-to-jointly-develop.html

VMware Client Hypervisor (CVP)

http://www.vmware.com/company/news/releases/cvp-intel-vmworld.html
http://www.brianmadden.com/blogs/brianmadden/archive/2008/10/13/a-deeper-look-at-vmware-s-upcoming-bare-metal-client-hypervisor.aspx
http://vinf.net/2009/02/24/vmware-client-hypervisor-cvp-grid-application-thoughts/
Seems that CVP will be not release with the new version of View 4.0: CVP release with View 4??

Source

Re: How do I run many operating systems at same time without host OS?

How work without vCenter Server

Sun, 25 Oct 2009 09:58:31 GMT

For who does not have vCenter Server here are some features that can be realized (of course with a lots of limitations) also without this product.

Note also that now there is a new free management tool specific for ESXi that can do a lot of interesting task:
VMware Go - http://www.vmware.com/go/vmware-go/
VMware to launch Go: a free web management service for ESXi - http://www.virtualization.info/2009/08/vmware-to-launch-go-free-web-management.html
VMware Free Service Simplifies ESXi Hypervisor Use - http://www.eweek.com/c/a/Virtualization/VMware-Free-Service-Simplifies-ESXi-Hypervisor-Use-341468/

Host and VM Management

The client is still the same (vSphere Client or VI Client), just point directly to the ESX/ESXi name or IP.
You will loose all vCenter specific menu (migrate, template, cluster, converter, maps, alarms, ...)

VMware HA

VMware HA require vCenter Server only during installation, that the HA agents will work on the ESX/ESXi. Also you must be licensed for HA.

Without vCenter Server and with only two hosts (but is possible to do also with more) you have to use shell script and VMware CLI.
See:
http://vmetc.com/2009/03/25/script-for-vmware-ha-feature-without-virtualcenter/

Also you can try Starwind: in this whitepaper (http://www.starwindsoftware.com/vmware-availability-guide) you can read how to implement HA using SAN.

VM template

Re: Creating templates without VirtualCenter Server?
http://www.dabcc.com/article.aspx?id=8774

VM Host Migration (with shared storage)

If you have vCenter, you could shutdown the VM and do a cold migration. Or use VMotion on a live VM but only if you have VMotion license and a shared storage.

If you do not have vCenter Server, you can simply power off the VM, go on destination host, browse the datastore with the required VM, right click on the vmx file, add to the inventory and then power-on the VM.
There could be a question about keep or modify the UUID of the VM (on vSphere the question is about you have moved or you have copied the VM). Choose "keep" (or "moved").

This is very useful also in case where exist a vCenter Server, it's a VM, but there isn't a VMotion license (like in vSphere Essential bundle).
How to migrate vCenter Server VM

VM Storage Migration

If you have vCenter Server, you could shutdown the VM and do a cold migration. Or if the ESX host in which the guest is on local storage, can see the new shared storage, you can do an SVMotion and then you wouldn't have downtime.

If you do not have vCenter Server, you can copy the VM folder (with VM powered off) using cp commnad from ESX CLI.
Note that your datastore are under /vmfs/volumes.
After the copy you add add the "new" VM to the inventory, test it, and if all works delete the old one.

Upgrade a major release of ESX/ESXi

With vCenter Server is very simple: just use VUM.
Without vCenter Server there are different solutions.

ESXi 3.5 -> ESXi 4.0

http://www.vm-help.com/esx40i/ESXi40_upgrade_without_virtualcenter.php
You can do it with "host upgrade utility" which is included to vSphere Client bundle.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011542 - Take a look at this VMware kb here on a method to obtain the Host Update Utility since you don't have vCenter.
You then download the media and add the media to the host update utility...

ESX 3.5 -> ESX 4.0

http://kb.vmware.com/kb/1009440 - Performing an offline upgrade from ESX 3.x to ESX 4.0 on a local ESX host
http://www.vmware.com/pdf/vsphere4/r40/vsp_40_upgrade_guide.pdf

Backup

VCB and most of the beackup programs can works both with vCenter Server or directly with ESX/ESXi.
Just point to the ESX name or IP.
See also: VCB backup without vCenter vSphere 4.0 Standard

SRM without SRM

Automating Disaster Recovery without SRM

Understanding Desktop Virtualization and Its Challenges

Tue, 20 Oct 2009 05:58:44 GMT

vSphere Client on the Linux Desktop

Fri, 16 Oct 2009 09:48:43 GMT

A lot of linux users are looking for a way to run .Net applications on their desktop, too. Since Wine does not provide the ability to do so and running an entire VM to use the complete Windows Desktop is an annoying option, these users are looking for a better solution. For Example the VMware vSphere Client, which requires .Net Framework and does not come as a Linux Version, forces Users and Admins to step back to Windows. But using Seamless RDP by Cendio, there is a way around this.

This Howto applies to Windows VMs running in background in any
virtualization product or physical systems alike. Also, this is of
course not limited to the VMware vSphere Client and can be used to run
any Windows application.

On the Windows System (e.g. a VM or the VMware vCenter Server) download the GPL licensed SeamlessRDP Tool and extract it into a folder, e.g. C:\RDP

Also, install the application you wish to use on your Linux Desktop

Enable the Remote Desktop in System Settings - Remote tab - "Allow users to connect remotely to this computer.". You might want to create an extra user for the remote access and enable it to connect via RDP.

Within the SemalessRDP folder, create a .bat file that runs your needed application, followed by the logoff command to make sure the session gets closed after usage. E.g: vsphere.bat:

"C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe"
logoff

Install rdesktop on your Linux system, using the appropriate way for your distribution or sources from www.rdesktop.org. Make sure you are using at least rdesktop version 1.5.

On your Linux Desktop, create a shourtcut for your Application, e.g. vSphere Client and call the following command from this shourtcut:

rdesktop -A -s "C:\RDP\seamlessrdpshell.exe batchfile" IP Address -u User -pPassword

You can of course create an extra user for the remote access and need
to insert the appropriate IP Address and Batch file to call. In my
vSphere example this looks like:

rdesktop -A -s "C:\RDP\seamlessrdpshell.exe C:\RDP\sphere.bat" 192.168.1.23 -u remote -p remoteP@55

On running the shourtcut from your Linux desktop, rdesktop will initialize a seamless session to the Windows system, start the batchfile to open the vSphere Client, and present only the application Window to our Linux Desktop.

So far this is probably the best solution we have to bring the VMware vSphere Client to the Linux Desktop. Enjoy.

This article can be found in English and an additional German Version on my Blog: http://thepickyblog.blogspot.com/

IOmega ix4-200d IOmeter results (100MB/Full Network)

Thu, 15 Oct 2009 21:07:59 GMT

Suitable server for ESXi

Thu, 15 Oct 2009 13:21:06 GMT

Hi guys

A simple request possibly

I need a server for development and pilot testing. Ideally it needs to be as small as possible but the HAL must be able to support ESXi 3.5. Sizewise id prefer something around the dimensions of any conventional small form factor PC that is scalable. Anyone managed to find something similar along those lines?

Advanced Networking Performance Options

Wed, 14 Oct 2009 16:58:33 GMT

Some of the advanced networking options available in vSphere 4.0 are reviewed in this paper. Many of these options control trade-offs between latency, throughput, CPU utilization, and reliability (e.g., dropped packets). It is not possible to optimize all of these at the same time, so option defaults are chosen to be suitable for the vast majority of applications. These options are provided to meet the stricter requirements of other applications. Advanced options often have subtle side effects, or merely move an issue from one area to another. Therefore it is recommended that VMware Support be engaged before changing such options, especially for production machines.

There are over 100 options that can be set under Configuration → Advanced Settings → Net. Of these, the ones listed below are most likely to be useful for tuning networking performance. Many of the others are for internal testing or enable unreliable features.

All of the options listed here take integer values. For the “Boolean” ones only the default value is shown: 0 for “false”, and 1 for “true”. Other parameters are shown with their default, minimum, and maximum values.

Parameter Name	(Default, Minimum, Maximum)	Description
MaxPortRxQueueLen	(80, 1, 500)	Maximum length of the Rx queue for virtual ports whose clients support queueing. Possibly should be increased if Rx packet drops are seen in the port connected to a VM. Relevant only for e1000 vNICs used with Fault Tolerance (FT) and VLANs.
MaxNetifTxQueueLen	(500, 1, 1000)	Maximum length of the Tx queue for the physical NICs. Increase if Tx packet drops are seen in uplink port to the pNIC.
GuestTxCopyBreak	(64, 60, 4294967295)	Packet header transmits smaller than this in bytes will be copied rather than mapped. More security and functionality than performance implications.
VmxnetTxCopySize	(256, 0, 4294967295)	Transmits smaller than this in bytes will be copied rather than mapped. Copying costs CPU but puts lets pressure on the Tx queue and doesn’t require completion.
VmxnetWinUDPTxFullCopy	(1)	Enable full copy of Windows vmxnet UDP Tx packets. Might disable to save CPU, especially for jumbo frames, at the cost of risking more packet drops.
NetTxDontClusterSize	(0, 0, 8192)	Tx packet size (in bytes) smaller than this are transmitted immediately (coalescing options are over-ruled for these packets). Used to ensure good latency for small packets.
CoalesceTxTimeout	(4000, 1, 4294967295)	The coalesce timeout in micro-seconds, or effectively the maximum latency without transmitting. Smaller values can reduce the packet latency at the cost of CPU. Risky to go below 1000.
CoalesceDefaultOn	(1)	Enable dynamic coalescing. Disable to test if issues are related to coalescing.
CoalesceHandlerPcpu	(1, 0, 128)	pCPU that coalesce timeout handler runs on. May be important to set this if VM CPU pinning is used.
CoalesceTxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Tx packets to coalesce. Reduce if Tx coalescing appears to be too aggressive.
CoalesceRxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Rx packets to coalesce. Reduce if Rx coalescing appears to be too aggressive.
vmxnetThroughputWeight	(0, 0, 255)	How far to favor Tx throughput for vmxnet 2 & 3. “0” is dynamic, otherwise this is a weight where a lower value favors latency and a higher value favors throughput.
TcpipHeapSize	(24, 24, 120)	Initial size of the TCP/IP module heap in megabytes. May need to increase if there are many vmkernel connections (NFS, iSCSI, etc.).
TcpipDefLROMaxLength	(16000, 1, 65535)	Maximum length for the LRO aggregated packet for vmkernel connections. Increasing this reduces the number of acknowledgments, which improves efficiency but may increase latency.
E1000TxZeroCopy	(0)	If disabled copy UDP or non-TSO Tx packets for e1000.
E1000TxTsoZeroCopy	(1)	If enabled do not copy TSO Tx packets for e1000.
E1000IntrCoalesce	(1)	Enable interrupt coalescing for e1000. Disabling can improve latency at the expense of CPU.
MaxPktRxListQueue	(3500, 0, 200000)	Maximum number of packets queued in vmkernel. Increasing this can reduce the number of dropped packets but at the cost of increased vmkernel memory and queuing latency.
Vmxnet3RSSHashCache	(1)	Enable RSS hash cache for vmxnet3 in Windows guests.
VmklnxLROEnabled	(0)	Enable large packets for recent Linux guests with vmxnet 2 & 3. Most likely to benefit hosts with small number of VMs with few sessions each, where each session has a heavy Rx load (more than 1 MB/sec). This is an experimental feature and has not been tested extensively.
VmklnxLROMaxAggr	(6, 0, 24)	Maximum aggregation count in number of packets for vmklinux LRO.

rename_vm.sh

Wed, 07 Oct 2009 14:00:46 GMT

clone_vm2.sh

Wed, 07 Oct 2009 13:57:43 GMT

force external load

Sun, 04 Oct 2009 13:50:45 GMT

afdsaff

aaaa

Sun, 04 Oct 2009 13:48:54 GMT

VAM Doc

Mon, 14 Sep 2009 12:52:51 GMT

This About Business Marketing.

Community Member Showcase (September 2009): Stephen Beaver - sbeaver

Fri, 11 Sep 2009 01:35:56 GMT

Stephen Beaver (*sbeaver*) is currently Virtualization Evangelist at Tripwire, and a VMware Certified Professional (VCP). Stephen is a co-author of two books on virtualization, Essential VMware ESX Server, and Scripting VMware Power Tools: Automating Virtual Infrastructure Administration. In addition, he is Technical Editor of VMware ESX Server: Advanced Technical Design Guide, and a contributing author to How to cheat at configuring VMware ESX Server. A respected pundit on virtualization technology, Stephen is a frequently-requested speaker at venues such as VMworld, the VMware Virtualization Forum, and the VMware Healthcare Forum.

Stephen is an active community expert in VMware's weekly online show, Community Round Table Panel, and is a moderator and one of the most active participants of the VMware Community forums. In fact, Stephen was recently named a 2009 VMware vExpert, an award given to individuals who have significantly contributed to the community of VMware users and helped spread the word about virtualization over the past year.

Prior to joining Tripwire, Stephen was a systems engineer with one of the largest private hospitals in the United States, Florida Hospital in Orlando, Florida, where he was responsible for the entire virtualization life cycle - from strategic planning to design and test, integration and deployment, to operations management. Prior to Florida Hospital, Stephen served as a Senior Engineer at the law firm Greenberg Traurig where he designed and deployed the firm's virtual infrastructure worldwide. He has also held posts at Lockheed Martin, the State of Nebraska, and The World Bank.

Bloggers & Twitterers at VMworld 2009

Tue, 25 Aug 2009 01:59:07 GMT

Dear Bloggers & Twitterers at VMworld 2009

Please add yourself if you're coming to VMworld! Personal feeds only, please. Feeds for corporate entities (ie, partners & exhibitors) should get emailed to jtroyer@vmware.com and we'll do those in a separate document.

Log in and click on "Edit Document" in the Actions box on the right
Click on the table where you'd like your name. (Right now it's alphabetical by last name, but I'll clean up any messes by importing this back into a spreadsheet, so feel free just to add yourself to the end.
1. If you are using the Rich Text editor tab, click on one of the buttons "Add Row Before" or "Add Row After" and do your thing.
2. If you are using the Plain Text editor tab, enter your data separated by | characters.
Click "Publish"

And voila. If you do this by Friday I'll try to get you a snazzy blogger badge at the show. See here for more information about blogging at VMworld If you are on Twitter, here's more about our VMworld Tweetup on Monday @ 8pm during the Welcome Reception.

Update: if you are a blogger on this list, and you would like to attend a press/analyst event with Paul Maritz about the Cloud at 10:45am Tuesday, email me jtroyer@vmware.com.

First Name	Last Name	Site	URL	Twitter
Meghan	Stabler	Condensing Substance on Cloud Computing\	http://communities.bmc.com/communities/blogs/meghan_stabler	MeghanatBMC
Nick	Allen	Wikibon	http://wikibon.org/	stor2
Wil van	Antwerpen	VI-Toolkit.com	http://www.vi-toolkit.com/ http://www.planetvm.net\	wilva
Steve	Beaver	Virtual Black Hole	http://www.thevirtualblackhole.com/author/sbeaver	sbeaver
Randy	Bias	Cloudscaling	http://cloudscaling.com/	randybias
Jason	Boche	VMware Virtualization Evangelist	http://boche.net/blog/	jasonboche
Rich	Brambley	VM /ETC	http://vmetc.com/	rbrambley
Guy	Brunsdon	virtual networking	http://blogs.vmware.com/networking	gbrunsdon
Daniel	Brusilovsky	Teens in Tech	http://teensintech.com/	danielbru
Thomas	Bryant	Work in Progress	http://blog.thb3.com/	kix1979
Cody	Bunch	Professional VMware	http://professionalvmware.com/	cody_bunch
Brian	Butler	PowerWF Blog	http://blog.powerwf.com/	Brian_Butler
Kevin	Camasi	NA	NA	kcamasi
Shawn	Cannon	NA	NA	rolltidega
Sean	Clark	Sean Clark	http://www.vmworldunderground.com/	vseanclark
Theron	Conrey	Conrey.org	http://www.conrey.org/	theronconrey
Dave	Convery	DailyHypervisor	http://www.dailyhypervisor.com/	dconvery
David	Davis	VMware Videos	http://vmwarevideos.com/	davidmdavis
Nava	Davuluri	Sample Code	http://communities.vmware.com/community/developer/codecentral\	navadavuluri
Frank	Denneman	Frank Denneman	http://frankdenneman.wordpress.com/	frankdenneman
Hannah	Drake	SearchVMware.com	TechTarget VMworld 2009 conference coverage\ Virtualization Pro\ SearchVMware.com\	SearchVMwareTT
Jae	Ellers	Mister VM	http://blog.mr-vm.com/	jellers
Duncan	Epping	Yellow Bricks	http://yellow-bricks.com/	DuncanYB
Jen	Erickson	TechJen	http://techjen.wordpress.com/	techjen
Jason	Fiset	NA	NA	jfiset
Stephen	Foskett	Stephen Foskett, Pack Rat & GestaltIT	http://blog.fosketts.net/ http://gestaltit.com/	sfoskett
David	Foyer	Wikibon	http://wikibon.org/	dfloyer
Kareen	Frascaria-Fossard	Virtuanews	http://www.zdnet.fr/blogs/virtuanews/
Jay	Fry	Data Center Dialog	http://datacenterdialog.blogspot.com/	jayfry3
Corey	Gilmore	coreygilmore.com	http://coreygilmore.com/	coreygilmore
Dave	Graham	flickerdown.com	http://www.flickerdown.com/	davegraham
Eric	Gray	VCritical	http://vcritical.com/	eric_gray
Forbes	Guthrie	vReference	http://www.vReference.com\
Edward	Haletky	Blue Gears	http://www.astroarch.com/blog/	Texiwill
Rod	Haywood	Musings of Rodos	http://rodos.haywood.org/	rodos
Chris	Hoff	Rational Survivability	http://rationalsecurity.typepad.com/blog/virtualization/index.html	Beaker
Charles	Hood	Charles Hood Dot Net	http://charleshood.net/	charleshood
Tom	Howarth	PlanetVM	http://planetvm.net/	tom_howarth
Sven	Huisman	virtualfuture.info	http://virtualfuture.info/	svenh
Johan	van Zanten	virtualfuture.info	http://virtualfuture.info/	jvzanten
Jim	Jones	VM Land	http://twitter.com/vmland	vmland
Michael	Keen	DABCC.com	http://www.dabcc.com/michaelkeen	michael_keen
Steve	Kemp	NA	NA	steverkemp
Brian	Knudtson	knudt blog	http://www.knudt.net/vblog/	bknudtson
Gabe	Knuth	Gabe Knuth	http://www.brianmadden.com/blogs/gabeknuth/	GabeKnuth
Jaime	Lamar	Modmaven's Weblog	http://modmaven.wordpress.com/
Jason	Langone	ThinkVirt	http://www.thinkvirt.com/	langonej
Kirk	Larsen	Security BoF	http://www.vmworld.com/blogs/security	kirkdude
Greg	Lato	latoga labs	http://www.latogalabs.com/	latoga
Mike	Laverick	RTFM Education	http://rtfm-ed.co.uk/	Mike_Laverick
David	Lawrence	The VMguy	http://vmguy.com/	thevmguy
Scott	Lowe	blog.scottlowe.org	http://blog.scottlowe.org/	scott_lowe
Brian	Madden	Brian Madden	http://www.brianmadden.com/blogs/brianmadden/	brianmadden
Andi	Mann	Enterprise Management Associates	NA	AndiMann
David	Marshall	VMBlog.com	http://www.vmblog.com/	vmblogcom
Andres	Martinez	Virtualizemos.com (Blog in Spanish)	http://blog.virtualizemos.com/	andresmartinez_
Jase	McCarty	Jase's Place	http://jasemccarty.com/blog/	jasemccarty
Jose Luis	Medina	Be virtual, my friend	http://bevirtual.blogspot.com/
Daniel	Morris	NA	NA	DanMoz
Bill	Mottram	Wikibon	http://wikibon.org/blog	Veridictus
Vlad	Nagornyi	NA	NA	vnagornyi
Kenon	Owens	DanteDog's Virtualization Blog	http://dantedog29.blogspot.com/	MS_Int_Virt
Jon	Owings	2vcps and a Truck	http://www.2vcps.com/	2vcps
Devang	Panchigar	StorageNerve	http://storagenerve.com/	storagenerve
Steve	Pantol	Steve's House of Virtualization and Waffles	http://www.virtualizationandwaffles.com/	StevePantol
Alessandro	Perilli	virtualization.info	http://virtualization.info/	govirtual
Tom	Phillips	NA	NA	tomaste
Joep	Piscaer	Virtual Lifestyle	http://www.virtuallifestyle.nl/	jpiscaer
Bob	Plankers	The Lone Sysadmin	http://lonesysadmin.net/category/virtualization/
Kristie	Popp	Data Center Management Blog \| HP Reality Check: Server Insights	http://www.communities.hp.com/online/blogs/reality-check-server-insights/default.aspx	kpopp
Dominic	Rivera	vmprofessional	http://vmprofessional.com/	n/a
Maish	Saidel-Keesing	Technodrone	http://technodrone.blogspot.com/	maishsk
Ed	Saipetch	Breathing Data	http://breathingdata.com/	edsai
Chad	Sakac	Virtual Geek	http://virtualgeek.typepad.com/virtual_geek/	sakacc
Rick	Scherer	VMware Tips	http://vmwaretips.com/	rick_vmwaretips
Simon	Seagrave	TechHead.co.uk	http://www.techhead.co.uk/	Kiwi_Si
Carter	Shanklin	The PowerCLI Blog	http://blogs.vmware.com/vipowershell	cshanklin
Andrew	Sharrock	NA	NA	AndrewSharrock
Eric	Siebert	vSphere Land	http://vsphere-land.com/	ericsiebert
Jason	Silva	SilvaECS	http://silvaecs.com/	silvaecs
Glenn	Sizemore	Get-Admin	http://get-admin.com/blog/	glnsize
Andrew	Storrs	Storrs.ca	http://www.storrs.ca/	astorrs
David	Strebel	Holy VM!	http://www.holy-vm.com/	holyvm
Andrew	Sullivan	Get-Admin	http://get-admin.com/blog/	acsulli
Various	Tech Target	Virtualization Pro	http://itknowledgeexchange.techtarget.com/virtualization-pro/
Vijay	Tewari	Microsoft	http://blogs.technet.com/virtualization	vtango
Joshua	Townsend	Vmtoday	http://vmtoday.com/	joshuatownsend
Lars	Troen	Virtual Thoughts	http://communities.vmware.com/blogs/larstr\	larstr
John	Troyer	VMTN Blog	http://blogs.vmware.com/vmtn/	jtroyer
Rick	Vanover	Everyday Virtualization	http://virtualizationreview.com/blogs/everyday-virtualization/list/blog-list.aspx	RickVanover
David	Vellante	Wikibon	http://wikibon.org/blog	dvellante
Lode	Vermeiren	Virtualization.com	http://www.virtualization.com/	lode
Ben	Vierck	PowerCLI Workflow Screencasts (PowerWF)	http://www.youtube.com/user/powerworkflow	xcud
	Virgil	Endless Consulting	http://www.endless.net.au/	virgilwashere
Nicholas	Weaver	Nickapedia.com	http://www.nickapedia.com/	lynxbat
Jay	Weinshenker	Livejournal	http://jweinshe.livejournal.com/	jweinshe
Chris	Wolf	Chris Wolf	http://chriswolf.com/	cswolf
Edwin	Yuen	Virtualization Planet	http://blogs.technet.com/virtplanet	edwinyuen
Gabrie van	Zanten	Gabe's Virtual World	http://www.gabesvirtualworld.com/	gabvirtualworld
Calvin	Zito	Around the Storage Block (HP)	http://www.hp.com/storage/blog	HPstorageGuy
Michael	Moeschler	Voce Nation	http://vocecommunications.com/blog/	moesch
Jay	Rogers	Virtual-Jay	http://virtual-jay.blogspot.com/
Ian	Koenig	Its Just Another Layer	http://www.itsjustanotherlayer.com/	iankoenig
William	Lam	VMware ESX and ESXi Scripts & Resources	http://engineering.ucsb.edu/~duonglt/vmware/	lamw
Tom	Ralph	Halfbytetechnologies \| four bits in an eight bit world	http://www.halfbytetechnologies.com/	tomralph
Nate	D'Amico	SiliconANGLE	http://siliconangle.com/ver2/vmworld09	kaiyzen
Pablo	Roesch	VMware Developer Community	http://developer.vmware.com	heyitspablo

HowTo P2V SCO UnixWare 7.1

Mon, 24 Aug 2009 14:35:34 GMT

Following a little HowTo for P2V a SCO UnixWare 7.1.3 Server

Requirements

Sco UnixWare Boot CD same version as installed (CD1)
HBA Buslogic Driver (on CD1 - search for "BLC" folder)
NIC AMD PcNet Driver for SCO UnixWare (http://www.amd.com/us-en/assets/content_type/utilities/unixware_71.exe)

Create a floppy image file with drivers then copy to your datastore.

1) P2V your UnixWare server with "VMWare Converter Boot CD" or any other way to create a virtual machine. Pay attention at number of CPU

2) Boot your Virtual Machine mounting UnixWare CD1 and floppy images. Use evaluation license, installBLC Drivers, go ahead till "NODE NAME" installation screenshot.

3) press (CTRL+ALT+SPACE) + ESC to access to UnixWare Console

4) Mount CDROM
#mount -F cdfs -r /dev/cdrom1 /cd-rom

5) Load File System Modules
#modreg 4 vxfs
#modadmin -l /cd-rom/.extra.d/etc/conf/mod.d/vxfs

#modreg 4 bfs
#modadm -l /cd-rom/.extra.d/etc/conf/mod.d/bfs

6) Mount partitions
#sh_mount -F vxfs /dev/dsk/c0b0t0d0S1 /mnt

WARNING!
if mount generate an error means that partition is unclean. FSCK with following command, then retry.
#/cd-rom/.extra.d./etc/fs/vxfs/fsck /dev/dsk/c0b0t0d0S1

7) Mount BootFS
#sh_mount -F bfs /dev/dsk/c0b0t0d0S1 /mnt/stand

8) CHROOT /mnt
#chroot /mnt /bin/sh

9) Install BLC Driver for BusLogic
#pkgadd -d /dev dsk/f0 blc

10) After Driver Installation mark BusLogic as Boot Device
#resmgr -m blc -i0 -p BOOTHBA -v0

11) Update config
#/etc/conf/bin/idconfupdate -f

12) Static Link BLC drivers
#vi /etc/conf/sdevice.d/bls
add "$static" after "$version 2"

13) Kernel Relink
#/etc/conf/bin/idbuild -FB#

On Unisys systems I found psm_share a propietary application without sources, disable it changing "Y" value to "N" in /etc/conf/sdevice.d/psm_share
If there are more than one disk, control path in /etc/loadmods.
Idbuild put compiled files in default folder, if necessary manually copy "unix" file in /stand.

You can use "AMD PCNet Driver" to install VMWare NIC

Hope to be helpful.

Massimo "Break" Biagi

vmware + truecrypt for fully encrypted windows OS?

Fri, 21 Aug 2009 20:27:42 GMT

started from thread on truecrypt forums: http://forums.truecrypt.org/viewtopic.php?p=22979

my question from there repeated for convenience:

if I do the following, will any traces be left on my machine at all?

create a truecrypt volume. mount it, and create a vmware partition for a new operating system on the truecrypt volume. if I boot up the vmware operating system (which only has access to the truecrypt section of the hard drive), would any traces of my activity be left behind? what about with the swap/page file in the native OS?

there responses seem to indicate "there's no way that'd work." is that accurate? can I configure vmware to do what I want? if so, how do I go about it?

Encrypting the VM volume is not really the answer here.
What you need to do is seal up the OS (inbound and outbound) within itself.
This will give you the self protecting layer you require.

Encrypting the vm volume will only stop other volumes/ hosts accessing it, not control the effects you are talking about in your thread.

Good luck.

Hello!
Could you please be more specific on how this should/could be done?

P.S.: Merry Christmas!

VMware products can create files outside the VM's directory, but you can play with the tmpDirectory field to control that.

However, the larger problem of the swap/pagefile containing traces of your activity remains intact. This is an operating systems shortcoming, not a VMware one.

I haven't tried, but if you gave the VM 100% reservation of the memory would it not create the swap file? Thus securing the VM from the host perspective?

when you give th VM 100% reservation a zero byte sized swap file is created

There are other traces left besides the swapfile
There are traces in the different logs (vmkernel,hostd,...).
Question is how easily these traces can be used.

But those traces contain no data of what is in the VM. So even a zero byte swapfile is useless, because it has no data in it. I would think if you secure the guest os, turn of things like TPS and have no swap it would be considered secure. The only thing on top of that you could add would be maybe encryption of the VMDKs, but that would add a significant amount of overhead to the virtualization layer.

But those traces contain no data of what is in the VM.

only VMware knows, I guess

So even a zero byte swapfile is useless, because
it has no data in it.agree

Yes, it will leave traces see my post http://www.vmware.com/community/thread.jspa?threadID=70884&tstart=0

then any dissent forensic will get you.

Encryption of the VMDKs, suspend file, and configuration files has been available in VMware ACE, since Dec 2004. You might want to look into that if you're on the desktop, not the server.

I know what you want to do but who exactly do you want to protect your data from? If it's law enforcement having a VM inside of an encrypted drive might be sufficient. If you're trying to protect things from the government then all bets are off.

If you want to hide porn from your parents it's easy. (Unless one of your parents happens to work for the FBI forensics team or NSA data recovery.)

If you want to hide things from the law, it gets harder as they have access to the FBI's forensic services. The best you can do is get everything encrypted and then run ENCASE on your own drives to see if you can find anything.

If you want to hide things from criminals then consider worst case they have a black hat with the skills of the FBI's forensic lab.

If you want to hide things from the NSA what the h@#$ are you doing in the first place that would draw their attention and you should seal your computers into an EM shielded room (TEMPEST) with no outside connections (Sniffing and intrusion) and have a block of thermite setting on top of your drives with a panic button on you at all times (NSA Data recovery that can get data off your drive no matter how many times you overwrite it though it will drop the classification level of the drive by one step. Per their own directive for destruction of classified material get an NSA approved degausser, yank your drive and toss it in. Note that those degaussers that don't require you to take the platters out cost in the range of $30k+) Even with all of that I can't guarantee they won't get your data.

If you want ideas look at the Common Criteria approved products list for data encryption that's approved per NSTISSP No. 11 for use on classified data. I'd advise looking for EAL 4+ products.

This document was generated from the following thread: vmware + truecrypt for fully encrypted windows OS?

Check your VM’s alignment for Better Performance

Fri, 21 Aug 2009 18:15:42 GMT

The Partition alignment process is not a new concept and has existed in the industry for years for Windows and Linux system way before the introduction of server virtualization.

The problem is that by default, some Operating Systems, including versions of Windows (Windows 2000, Windows 2003) and Linux align the first sector on a full disk track boundary. But now windows 2008 comes with this issue fixed J

When setup routine queries the CHS (Cylinder / Head / Sector) information of the disk from the BIOS, the CHS data doesn't actually corresponded to anything physical. At large LUN sizes, the S number (sectors per track) is always reported as 63, so partitioning is set at 63 sectors starting offset.

The problem is that 63 sectors, corresponds to 32256 bytes or 31.5KB, which does not align with storage array's logical block size. Physical disk blocks always correspond to 512 bytes in size, but for efficiency and performance reasons, disk arrays use a logical block size unit that is a multiple of the physical block size.

In a properly setup environment (physical or virtual) the host should align its partitions in such a way that writes/reads are aligned to the array's logical block size. That would mean that you'd have a 1:1 ratio of logical to physical reads/writes. In any other case, you'd be forced to read/write potentially up to twice as many blocks as the application actually requested, since any File system block actually occupies at least two partial physical blocks.

The end result of this from a performance as well as a sizing perspective can be significant since you can be doing up to 2x the number of IOs and thus you may require up to 2x more spindles to support the number of IOPs. Furthermore, in an undersized environment where this is happening you will see a sharp increase in latency and when folks complain about performance, it's latency they typically complain about.

mbralign - This is a tool the provides you with the ability to align a GuestOS's partitions in 4 steps:

1) Shut down the VM

2) Run mbralign

3) Power on the VM

4) Remove the backup files

Prior to going thru the process, the tool, makes a backup of the .vmdk and *-flat.vmdk files and creates a new file .vmdk and properly aligned *-flat.vmdk and copies the data from the old source vmdk to the new destination vmdk. After the process is done you have to manually clean up the two backup files created initially.

What type of performance can be expected while running this tool?

There are many factors which gets involved you are running this tool.

The size if the file
Load on ESX host (because you run this tool through console on the ESX where VM is registered)
The Load on Controller
The IP network Traffic till the time vmdk is fully aligned ( I am talking about network traffic because I am taking about NFS which is using TCP/IP protocol )

Please let me know if anyone else is seeing performance issue specially on using NFS protocol.

VMKillies

vmware

Thu, 13 Aug 2009 18:20:07 GMT

Hot add cpu to supported Linux guestOSes using vSphere ESX(i) 4.0

Sun, 09 Aug 2009 20:21:42 GMT

Summary

This document will demonstrate how to hot add CPU to a live running Linux VM (Ubuntu 9.04 x86-64) using vSphere ESX(i) 4.0, this feature is only supported on certain guestOSes, check vSphere documentation for more information.

Resources

http://vmlover.blogspot.com/2009/05/hot-add-cpu-vsphere.html
http://communities.vmware.com/message/1223006#1223006

Requirements

ESX(i) 4.0
Ubuntu running kernel 2.6.14+ 64bit

Instructions

1. Hot add CPU requires a VM to be running hardware version 7, you'll need to first right click on the VM and upgrade to hardware 7 before powering on the VM if it's not already. You will also need to enable hot add cpu which is a new configuration item once you've upgraded to HW7 under OPTIONS->Memory/CPU Hotplug and just select 'enable'

2. Power on Linux VM

3. Let's verify the default cpu available within the VM (this can be done using /proc/cpuinfo,top,etc)

root@ubuntu-test:~# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz
stepping        : 5
cpu MHz         : 2000.072
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable pni ssse3 cx16 sse4_1 sse4_2 hypervisor lahf_lm
bogomips        : 4000.14
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Default installation of our Ubuntu Server is 1 vCPU, we'll increase this to 4 vCPU

4. You will now use the vSphere Client and edit the VM's cpu and change it from 1vCPU to 4vCPU

5. Download and copy online_hotplug_cpu.sh to the VM

root@ubuntu-test:~# chmod +x online_hotplug_cpu.sh

7. Execute the script which will find any new cpu cores allocated to VM and online the core

root@ubuntu-test:~# ./online_hotplug_cpu.sh
Found cpu: "/sys/devices/system/cpu/cpu0" ...
        cpu0 already configured prior to hot-add
Found cpu: "/sys/devices/system/cpu/cpu1" ...
        cpu1 is new cpu, onlining cpu ...
Found cpu: "/sys/devices/system/cpu/cpu2" ...
        cpu2 is new cpu, onlining cpu ...
Found cpu: "/sys/devices/system/cpu/cpu3" ...
        cpu3 is new cpu, onlining cpu ...

As you can see from the script output cpu[0] was already allocated and cpu1-3 was the newly added cpu which needed to be brought online before the OS can use it.

8. Let's verify the new cpu that has been allocated while the VM was running

root@ubuntu-test:~# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz
stepping        : 5
cpu MHz         : 2000.072
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable pni ssse3 cx16 sse4_1 sse4_2 hypervisor lahf_lm
bogomips        : 4000.14
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz
stepping        : 5
cpu MHz         : 2000.072
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable pni ssse3 cx16 sse4_1 sse4_2 hypervisor lahf_lm
bogomips        : 3999.24
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz
stepping        : 5
cpu MHz         : 2000.072
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable pni ssse3 cx16 sse4_1 sse4_2 hypervisor lahf_lm
bogomips        : 4004.59
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz
stepping        : 5
cpu MHz         : 2000.072
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable pni ssse3 cx16 sse4_1 sse4_2 hypervisor lahf_lm
bogomips        : 4002.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Microsoft Windows Server Clustering 101

constantin.videnski@starwindsoftware.com — Wed, 05 Aug 2009 08:45:46 GMT

Configuration advice please (Single ESX Server, 2 VMs)

Wed, 05 Aug 2009 08:27:16 GMT

All,

we have been provided with a single HP DL380 with 4x146GB HDDs and have been asked to provision 2 VMs on this Host. In itself this is not a problem as the host has sufficient RAM and CPU power, the question is how to best use the HDDs. The 2 VMs are to be SQL2005 Servers, so I am not sure as to whether to create 1 large Raid5 with all disk, or create 2 Mirrors ??

All advise welcomed

TIA

Nels

503 service unavailable

Wed, 29 Jul 2009 10:00:19 GMT

Hi,

I had installed Vmware 2.0.1 on my system. it is successfully installed. but when i start VMware i am getting 503 service unavailable.

please help out me on this error.

Thanks,

Anjan.

inaccessibleDatastore error

Mon, 27 Jul 2009 10:48:38 GMT

while i am creating a new virtual machine, i am getting one error message,

The server response included one or more errors. If these problems persist, please contact your system administrator.

Details: InaccessibleDatastore: Datastore 'standard /' is not accessible.

Memory Performance Chart Metrics in the vSphere Client

Thu, 23 Jul 2009 22:05:34 GMT

The vSphere Client exposes several memory performance statistics for users to identify VM memory usage.

Some of the important memory performance metrics follow. Each metric name appears under the Measurement column of the Performance Chart Legend, as shown in the following screenshot:

Active: The amount of guest physical memory that is being used by the VM. Active memory may be different from what is seen inside the guest operating system. This is because the guest operating system generally has a more precise view about what memory is “active” than the hypervisor because it knows when applications allocate or deallocate memory. In addition, the sampling technique used by ESX often takes time to converge, so the memory usage measured in the guest operating system may be more accurate when the workload memory usage is fluctuating.
Shared: The amount of guest physical memory shared through transparent page sharing. This includes the memory shared with other VMs and the memory shared within the VM.
Consumed: The amount of host physical memory allocated to the VM, accounting for saving from memory sharing with other VMs. When multiple VMs share a host memory region, each VM is accounted to consume the shared memory proportionally based on the total references to that host memory. For example, if a VM has 100MB host memory equally shared with the other three VMs, the Consumed memory only accounts for 25MB. If the 100MB memory is only shared within the VM, the Consumed memory accounts for 100MB.

Note that for a host that is not memory overcommitted, the Consumed memory represents a “high water mark” of the memory usage by the VM. It is possible that in the past, the VM was actively using a large amount of host physical memory but currently it is not. Because host memory is not overcommitted, the Consumed memory will not be shrunk through ballooning or swapping. Hence, the Consumed memory could be much higher than the Active memory when host memory is not overcommitted.
Granted: The amount of guest physical memory currently backed by the host physical memory. Due to memory sharing, the Granted memory is greater than or equal to the Consumed memory. For instance, assuming a guest allocates 100MB memory while the whole memory are zeroes, once all the zeroed pages are shared, the VM’s Granted memory is 100MB but the VM’s Consumed memory is only 4k.
Overhead: The extra host physical memory used by the ESX to run a VM. The Overhead memory has two components: 1) System wide overhead from VMkernel; 2) Additional overhead for each VM, including the space reserved for the VM frame buffer and various virtualization data structures. Since the Overhead memory always resides in host memory, ESX must reserve memory for it. Thus a VM’s memory reservation has two individual components: user-specified memory reservation and overhead memory reservation. For example, if the user specifies a 1GB reservation and the Overhead memory for the VM is 100MB, the VM’s memory reservation when powered on would be 1.1GB.
Balloon: The amount of guest physical memory that is currently reclaimed through the balloon driver.
Swapped: The amount of guest physical memory swapped out to the VM’s swap device by ESX.
Swapped in rate: The rate at which the host physical memory is being swapped in from the host swap device.
Swapped out rate: The rate at which the host physical memory is being swapped out to the host swap device.

Embedding YouTube videos made easy

Fri, 17 Jul 2009 21:32:27 GMT

Hello Community Members,

Did you know that you can easily embed a YouTube video (with link to the source) in messages/threads, documents, and blog posts, on VMware Communities? Simply place the tag "youtube" within braces before and after the YouTube video link, as illustrated below:

{youtube}your_YouTube_video_link{youtube}

Therefore,

{youtube}http://www.youtube.com/watch?v=F7O4uMRADB8{youtube}

would give you:

{youtube}http://www.youtube.com/watch?v=F7O4uMRADB8{youtube}

Example within a blog: Embedding YouTube videos made easy

Example within a discussion thread: @ Developer Day Sessions: Introduction to vSphere Web Services SDK 4.0 and NewFeatures

Badsah Mukherji, VMware Web Communities Team

Performance Troubleshooting for VMware vSphere 4 and ESX 4.0

Mon, 13 Jul 2009 14:03:44 GMT

Performance problems can arise in any computing environment. Complex application behaviors, changing demands, and shared infrastructure can lead to problems arising in previously stable environments. Troubleshooting performance problems requires an understanding of the interactions between the software and hardware components of a computing environment. Moving to a virtualized computing environment adds new software layers and new types of interactions that must be considered when troubleshooting performance problems.

The attached document is the first installment in a guide covering performance troubleshooting in a vSphere environment. It uses a guided approach to lead the reader through the observable manifestations of complex hardware/software interactions in order to identify specific performance problems. For each problem covered, it includes a discussion of the possible root-causes and solutions. Topics covered include performance problems arising from issues in the CPU, memory, storage, and network subsystems, as well as in the VM and ESX host configuration. Guidance is given on relevant performance metrics to observe using the vSphere Client and esxtop in order to isolate specific performance issues.

This first installment of Performance Troubleshooting for VMware vSphere 4 covers performance troubleshooting on a single VMware ESX 4.0 host. It focuses on the most common performance problems which affect an ESX host. Future updates will add more detailed performance information, including troubleshooting information for more advanced problems and multi-host vSphere deployments.

This is a living document. Reader comments, questions, and suggestions are encouraged.

Vista Sidebar VIClient Launcher Gadget

Mon, 22 Jun 2009 19:26:21 GMT

Storage Workload Characterization and Consolidation in Virtualized Enviornments

Wed, 03 Jun 2009 00:21:48 GMT

vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server

Mon, 01 Jun 2009 22:53:39 GMT

Using vscsiStats for Storage Performance Analysis

Fri, 29 May 2009 22:28:27 GMT

Introduction

esxtop is a great tool for performance analysis of all types. However, with only latency and throughput statistics, esxtop will not provide the full picture of the storage profile. Furthermore, esxtop only provides latency numbers for Fibre Channel and iSCSI storage. Latency analysis of NFS traffic is not possible with esxtop.

Since ESX 3.5, VMware has provided a tool specifically for profiling storage: vscsiStats. vscsiStats collects and reports counters on storage activity. Its data is collected at the virtual SCSI device level in the kernel. This means that results are reported per VMDK (or RDM) irrespective of the underlying storage protocol. The following data are reported in histogram form:

IO size
Seek distance
Outstanding IOs
Latency (in microseconds)
More!

Running vscsiStats

vscsiStats collection and analysis requires two steps:

Start statistics collection.
View accrued statistics.

Documentation on command-line parameters are available when running '/usr/lib/vmware/bin/vscsiStats -h'.

Starting and Stopping vscsiStats Collection

The tool is started with the following command:

/usr/lib/vmware/bin/vscsiStats -s -w <world_group_id>

This command starts the process that will accrue statistics. The world group ID must be set to a running virtual machine. The running VMs' IDs can be obtained by running '/usr/lib/vmware/bin/vscsiStats -l'.

After about 30 minutes vscsiStats will stop running. If the analysis is needed for a longer period, the start command should be repeated above in this window. That will defer the timeout and termination by another 30 minutes.

Since results are accrued and reported out in summary, the histograms will include data since collection was started. To reset all counters to zero, run '/usr/lib/vmware/bin/vscsiStats -r'.

Viewing Statistics

Counters are displayed by using the following command:

/usr/lib/vmware/bin/vscsiStats -p <histo_type> [-c]

The histogram type is used to specify either all of the statistics or one group of them. Options include all, ioLength, seekDistance, outstandingIOs, latency, interarrival.

Results can be produced in a more compact comma-delimited list by adding the optional "-c" above.

Using vscsiStats Results

Use Case 1: Identifying Sequential IO

Storage arrays can process sequential IO much faster than random IO. You can therefore improve the performance of a sequential workload by placing it on a dedicated LUN to allow the array to optimize access. vscsiStats can help you identify your sequential workloads even if you don't understand anything about the application in the VM.

Take the following graph as example, which I generated by running '/usr/lib/vmware/bin/vscsiStats -p seekDistance':

This graph shows that most of the commands are being issued a great distance from the previous command. It looks like all of the commands were 50,000 or more logical blocks away from the previous command. When I looked at the raw data, I saw that over 99% of the commands were more than 128 blocks away from the previous command. That's random access if I've ever seen it. Here's the opposite example:

In this case the logical block number (LBN) of each command is most frequently exactly one larger than the previous command. That's the signature of a heavily sequential workload. It shouldn't surprise you to learn that both of these profiles were generated by Iometer using random and sequential writes, respectively.

Use Case 2: Optimizing for IO Sizes

The IO size is an important characteristic of storage profiles. A variety of best practices have been provided by storage vendors to enable customers to tune their storage to a particular IO size. As an example, it may make sense to optimize an array's stripe size to its average IO size. vscsiStats can provide a histogram of IO sizes to help this process. The following graph was generated by '/usr/lib/vmware/bin/vscsiStats -p ioLength':

From these results I can see that about a quarter of the commands came in IOs smaller than 4k. About half of the commands were sized to 4k commands. The minute number of remaining IOs were larger than 4k. This signature is common of a VMDK formatted to 4k blocks and supporting OS and application execution. The storage array should be optimized for 4k blocks if this disk's performance is a priority.

Use Case 3: Storage Latency Analysis (Including NFS!)

esxtop is a terrific tool for latency-based storage analysis. Fibre Channel and iSCSI HBAs have device and kernel latencies in esxtop's storage panel. Software iSCSI initiators will show up as vmhba32 (ESX 3.5 and earlier) and vmhba33 (ESX 4.0 and later.) But esxtop does not provide latency statistics for NFS stores.

Because vscsiStats collects its results where the guest interacts with the hypervisor, it is unaware of the storage implementation. Latency statistics can be collected for all storage configurations with this tool.

The above graph shows that the server in my office with a single direct-attached SCSI disk is performing as I would expect. About half of all the operations are completing in under 5 ms. The other half take 5-15 ms to complete. A few commands took longer than 15 ms, but the number is so small that it doesn't concern me. Similar results can be seen with NFS arrays.

vscsiStats on ESXi

vscsiStats can be installed on ESXi hosts after putting the host into tech support mode. More information on this process is availalble on Scott's blog on the subject on vPivot.

Additional Resources

My colleagues Ajay Gulati, Chethan Kumar, and Irfan Ahmad presented at VPACT 09 Storage Workload Characterization and Consolidation in Virtualized Enviornments. This paper serves as an excellent example of vscsiStats in action.

I learned vscsiStats by reviewing Irfan's VMworld 2007 presentation (vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server) and playing with the tool. Check out his presentation if you'd like more detail.

Clustering Software in a VMware environment

Sat, 23 May 2009 15:42:13 GMT

It is possible to combine virtualization technology with "traditional" cluster technology to increase the availability of some service.
As is physical case, guest clustering usually require shared storage.

Guest Cluster type

Guest Cluster can be created in 3 different schema:

Cluster in a box: two VM inside the same ESX (can also work without shared storage)
Cluster across box: two VM in different ESX
Cluster physical-virtual: one physical node and one VM

A different approach could be use a software iSCSI initiator inside the guests: in this case the cluster's nodes are "normal" VM with only their own local disks. The shared disks are visible only inside the VM so from VMware side you do not have any limitation, from guest side the OS and the vendor must only support a configuration with software iSCSI.

MSCS vs VMware HA

Note that application/guest clustering is different from VMware HA (and also from VMware FT):
Microsoft Cluster Server (MCS) versus VMWare HA
Pros and Cons to have MS Cluster?
To use MSCS or not?
VMware HA Vs Microsoft Cluster Server: http://it20.info/blogs/main/archive/2008/03/26/102.aspx

Windows 2003 MSCS

Actually the only supported cluster environment in ESX 3.x is MSCS (Microsoft Cluster Service) 2003 in a 2 node configuration using FC SAN (for cluster across boxes and cluster physical-virtual):
http://www.vmware.com/pdf/GuestOS_guide.pdf
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_mscs.pdf

This does not mean that is not possible to have a working Guest Cluster with iSCSI storage (and it can work fine also with iSCSI RDM disks, but is not officially supported by VMware...).

To configure VMware for a cluster in a box there is a good step-by-step guide:
http://exchangeexchange.com/blogs/bkeane/archive/2007/07/30/mscs-clustering-in-vmware.aspx

To configure the Microsoft part, just use the Technet Guide:
For 2003: Microsoft Windows Server 2003 Server Clusters Quick Start Guide
To setup a MSCS Cluster, could also be usefull this document:
http://www.boche.net/blog/index.php/2009/04/01/setup-for-microsoft-cluster-service/

Windows 2008 Fail-over Cluster

Windows 2008 fail-over cluster will work only in ESX4 (due to lack of SCSI3 reservation in ESX 3.x).
For more info of MSCS configuration in vSphere:
http://www.vmware.com/pdf/vsphere4/r40/vsp_40_mscs.pdf
http://vmguy.com/wordpress/?p=1019

To configure the Microsoft part, just use the Technet Guide:
For 2008: Failover Cluster Step-by-Step Guide: Configuring a Two-Node File Server Failover Cluster

Other informations about Microsoft cluster

For a complete list of Microsoft Cluster Service (MSCS) support on ESX Server 3:
Microsoft Cluster Service (MSCS) support on ESX - http://kb.vmware.com/kb/1004617

For specific Microsoft cluster solutions:
SQL 2008: SQL Server Clustering Guide
Exchange 2007: Implementing Exchange 2007 on VMware

To using VMware Converter to convert Microsoft Cluster (MSCS) nodes to virtual machines there is a specific KB:
http://kb.vmware.com/kb/1002661

Other type of Guest Cluster

Other cluster technology could work, but usually are Linux based and Linux is not Windows, and Windows is not Linux .

For Veritas Cluster Server (VCS) and VMware:
http://symantec.dciginc.com/2008/10/veritas-cluster-server-vcs-and.html

For RedHat Cluster and VMware:
http://sources.redhat.com/cluster/wiki/VMware_FencingConfig

Guest Cluster limitations

Note the application cluster configuration will have several limitations and restrictions (http://communities.vmware.com/message/1099039#1099039):

no shapshot on the VM
no vMotion
no VCB

Meet the Engineer Series: VMware Performance Advancements

Thu, 21 May 2009 23:41:52 GMT

Real people, real faces, discussing VMware vSphere topics...

In this video, VMware's Chief Performance Architect discusses why you should seriously consider virtualizing all of your applications on VMware:

{youtube}http://www.youtube.com/watch?v=I-W0ZWm5Jf4{youtube}

VMFS Recover or Undelete

Tue, 19 May 2009 07:16:25 GMT

As you know, when you remove a VM or a vmdk disk from disk (and not only from inventory or VM) you loose all your data.
Is possible to roll back? Yes... just restore from a backup.

But if you do not have a VM backup? Or a recent one?

The solution could be the VMDK Recovery Tool.

It requires proactive steps to protect the VMFS datastore: only if the block-list (a plain text file that keeps a record of all VMDKs of the virtual machines) is generated automatically it will work in a production environment.
Also if VMs are moved, cloned etc. or reconfigured it must be refreshed or the tool could do more damage than good.

Some limitations of the VMDK Recovery Tool are that is an experimental software (means that the tool is not supported and might not work as expected in all customer environments), it works only in ESX 3.x (is not available for ESXi and for ESX 4.0 ) and you cannot restore from multiple block list files.

To configure this tools there is also an HowTo: Protecting ESX VMFS Stores with Automation

What can I do if I delete the data and I never configure the VMDK Recovery Tool?

In this case you can use some tools that make a search at physical block-level: for example http://www.ufsexplorer.com/
UFS Explorer is designed to make no matter where your data reside. It supports fast and safe access to set of storages, including SCSI/SATA/PATA hard disks, USB mass storage devices, RAID arrays, VMWare, Microsoft Virtual PC, Parallels Workstation, Bochs virtual disks and RAW disk or partition image files.
UFS Explorer did not find any file system or folder structure or file names so all the deteced file are RANDOM#.JPG or RANDOM#.TXT. The data that is stored inside files like for images EXIF and word files the Author are intact so I was able to indentify my data by a combination of looking at file sizes + adding additional fields to the display in windows.
Note that the advanced data scan took about 1 hour per GB.

What can I do if I loose the VMFS partition?

If you only loose the partition, is possible recreate with fdisk.
See: Re: Removed VMFS3 volume from iSCSI target - am I screwed?

For more details on VMFS there is the great Understanding VMFS Volumes

ESX Monitor Modes

Tue, 28 Apr 2009 20:40:05 GMT

VMware has supported Intel and AMD's virtualization assist since 2006. Long before then we were using an all-software approach that we call binary translation (BT). With the benefit of years of development and optimization, BT outperformed the early versions of hardware assist. But as hardware assist evolved the use of these new features became more attractive.

Because our support for hardware assist is rich and BT is heavily optimized, the monitor can benefit from using either technology in different situations. The following tables detail the defaults in ESX 4.0, which can be changed through VM settings if desired.

Monitor Defaults with Intel Processors

VM Configuration	Core-i7 (Nehalem)	45nm Core2 with VT-x	65nm Core2 with VT-x and FlexPriority	65nm Core2 with VT-x and No FlexPriority	P4 with VT-x	EM64T without VT-x	No EM64T
FT enabled	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	Not runnable	Not runnable	Not runnable
64-bit guests	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	Not runnable	Not runnable
VMI enabled	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT
OpenServer, UnixWare, OS/2	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	BT + SPT	BT + SPT
32-bit Linux and 32-bit FreeBSD	VT-x + EPT	VT-x + SPT	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
32-bit Windows XP, Windows Vista, Windows Server 2003, Windows Server 2008	VT-x + EPT	VT-x + SPT	VT-x + SPT	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
Windows 2000, Windows NT, DOS, Windows 95, Windows 98, Netware, 32-bit Solaris	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
All other 32-bit guests	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	BT + SPT	BT + SPT

(*) When we use BT on an Intel system with VT-x capability, we dynamically switch to VT-x if the guest enters long mode.

Monitor Defaults with AMD Processors

Configuration	Barcelona, Phenom, and Newer	AMD64 pre-Barcelona	No AMD64
FT enabled	AMD-V + SPT	Not runnable	Not runnable
64-bit guests	AMD-V + RVI	BT + SPT	Not runnable
VMI enabled	BT + SPT	BT + SPT	BT + SPT
OpenServer, UnixWare, OS/2	AMD-V + RVI	BT + SPT	BT + SPT
32-bit Linux and 32-bit FreeBSD	AMD-V + RVI	BT + SPT	BT + SPT
32-bit Windows XP, Windows Vista, Windows Server 2003, Windows Server 2008	AMD-V + RVI	BT + SPT	BT + SPT
Windows 2000, Windows NT, DOS, Windows 95, Windows 98, Netware, 32-bit Solaris	BT + SPT	BT + SPT	BT + SPT
All other 32-bit guests	AMD-V + RVI	BT + SPT	BT + SPT

Legend

VT-x: Intel's virtualization hardware assist.
EPT: Extended Page Tables. Intel's on-board, virtualization-aware memory management unit (MMU).
EM64T: Intel's 64-bit extensions to the x86 architecture.
SPT: Shadow page tables. ESX's software memory management unit (i.e., not EPT or RVI.)
BT: Binary translation. ESX's software virtualization capability (i.e., not VT or AMD-V)
AMD-V: AMD's virtualization hardware assist.
RVI: Rapid Virtualization indexing. AMD's on-board, virtualization-aware memory management unit (MMU).

ThinApp Resolution Paths

Tue, 28 Apr 2009 01:48:23 GMT

System Management Resolution Paths

Tue, 28 Apr 2009 01:47:26 GMT

OS Resolution Paths

Tue, 28 Apr 2009 01:46:37 GMT

Storage Resolution Paths

Tue, 28 Apr 2009 01:45:52 GMT

View Manager Resolution Paths

Tue, 28 Apr 2009 01:45:06 GMT

Network Resolution Paths

Tue, 28 Apr 2009 01:44:06 GMT

Licensing Resolution Paths

Tue, 28 Apr 2009 01:43:23 GMT

VMware Lab Manager Resolution Paths

Tue, 28 Apr 2009 01:42:43 GMT

Install Resolution Paths

Tue, 28 Apr 2009 01:41:54 GMT

Hosted Resolution Paths

Tue, 28 Apr 2009 01:41:09 GMT

Fault Resolution Paths

Tue, 28 Apr 2009 01:37:58 GMT

VMware Converter Resolution Path

Tue, 28 Apr 2009 01:35:30 GMT

vGhetto Script Repository

Fri, 24 Apr 2009 17:17:22 GMT

Here is a nice consolidation of all the scripts we've written for easier access and also check out our site at: http://engineering.ucsb.edu/~duonglt/vmware/

We'll be updating this page as new scripts are released and if there are any script/tool(s) that might be of use to the general community that does not already exists, please post in this thread and I'll take a look as time permits.

Creation Date	Script Name	Category	Supported Platform	Executed on
11/11/09	configureAdvHAClusterOptions.pl	Cluster Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
11/10/09	vmdkManagement.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
11/02/09	findVMsWithRDMs.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
10/28/09	listHAClusterNodes.pl	Host Management	vMA 4.0	vMA 4.0
10/26/09	vmISOManagement.pl	VM Management	VMware vMA 4.0	VMware vMA 4.0
10/26/09	upgradeVMVirtualHardware.pl	VM Management	VMware vMA 4.0	VMware vMA 4.0
10/21/09	rdmManagmement.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
10/14/09	Getting Started with the vMA (tips/tricks)	vMA/VIMA	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
10/11/09	getVMDiskInfo.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
10/08/09	useVIFastpassOnvMAToRunPerlScriptWithoutClearTextPassword.pl	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
10/07/09	setUserDatastorePermission.pl	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
10/03/09	pluginExtensionManager.pl	Plugin Managment	vSphere SDK for Perl	VMware VIMA 1.0 & vMA 4.0
10/02/09	Getting Started with the vSphere SDK for Perl	vSphere SDK for Perl	vSphere SDK for Perl	VMware VIMA 1.0 & vMA 4.0
09/28/09	addVMAnnotation.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
09/26/09	getVMThinProvisionedSavings.pl	VM Management	VMware vMA 4.0	VMware & vMA 4.0
09/25/09	generateVMRemoteConsoleURL.pl	VM Management	VMware vMA 4.0	VMware & vMA 4.0
09/24/09	getVMsPerDatastore.pl	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
09/23/09	listVMsInResourcePool.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
09/21/09	queryVMsCreated.pl	VM Management	VMware vMA 4.0	VMware & vMA 4.0
09/21/09	listDatastoreFiles.pl	Host Management	vMA 4.0	vMA 4.0
09/14/09	getAllVMsDiskFormatType.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
09/12/09	executeCommandUsingFastPass.pl	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
09/10/09	getDatastoreUUID.pl	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
09/09/09	Instructions on how to install vCLI 32bit/64bit on Ubuntu 9.04	Host/VM Management	Ubuntu 9.04	Ubuntu 9.04
09/09/09	getWWPN.pl	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/30/09	getVMsResourcePools.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/28/09	factoryResetESXi\	Host Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/23/09	ghettoVCB-restore.sh\	Backup/Recovery	ESX(i) 3.5 and 4.0+	ESX Service Console and/or ESXi Busybox Console
08/19/09	renameLocalStorageDatastore.pl	Host Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
08/18/09	removeHostFromvCenterByForce.pl\	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/18/09	getNumofvCPUInCluster.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/18/09	addvSwitchWSecurityPolicy.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/17/09	verifyVpxaAndHostConf.sh\	Host Management	ESX(i) 3.5 & 4.0	ESX Service Console and/or ESXi Busybox Console
08/16/09	getVMToolsStatus.pl	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/14/09	getVMonShareableAndNonSharDatastore.pl\	VM Management	vMA 4.0	vMA 4.0
08/13/09	addVMAdvParamOption\	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/13/09	addVMCustomField\	VM Management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/08/09	takeVMScreenshot.pl\	VM Management	ESX(i) 4.x	vMA 4.0
08/09/09	online_hotplug_cpu.sh\	Linux guestOS Management	Linux Supported guestOS	Linux Supported guestOS
08/09/09	online_hotplug_memory.sh\	vMA and Linux guestOS Managment	vMA 4.0 or Linux supported guestOS	vMA 4.0 or Linux supported guestOS
08/08/09	How to increase/resize vMA Disks	vMA/VIMA management	VMware VIMA 1.0 & vMA 4.0	VMware VIMA 1.0 & vMA 4.0
08/08/09	getAllVMGuestHostname.pl	VM Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
08/07/09	getAllVMMacs.pl	VM Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
07/31/09	whichClusterIsMyVMin.pl\	VM Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
07/01/09	poweroff_suspend_vm.pl	VM Managerment	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/30/09	ftCLI.pl\	VM Deployment/Managerment	ESX(i) 4.x and vCenter 4.0	VMware vMA 4.0
06/26/09	queryDeletedVMs.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/18/09	queryVMCustomField.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/17/09	getVMDiskCID.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/17/09	rescanAllHBAInCluster.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/14/09	removeOrphansFromvCenter.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/06/09	updateVMPortgroup.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
05/29/09	listVMByFolder.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
05/28/09	getAllPowedOffVMs.pl	VM Deployment/Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
06/13/09	queryDiskParition.pl	Host Management	ESX(i) 3.5 + 4.x	VMware VIMA 1.0 & vMA 4.0
02/21/09	ghettoVCBg2\	Backup/Recovery	ESX(i) 3.5u2+ & 4.x+	VMware VIMA 1.0 & vMA 4.0
05/12/09	checkLicenseServer.pl\	Host Management	ESX(i) 3.5+ & vCenter 2.5+	VI Perl Toolkit & VMware VIMA 1.0
03/16/09	vGhettoLinkedClone.pl\	VM Deployment/Management	ESX(i) 4.x+ & vCenter 4.x+	vSphere SDK for Perl 4.0 & VMware vMA 4.0
03/05/09	cdp.pl\	Reporting/VM Management	ESX(i) 3.5+ & vCenter 2.5+	VI Perl Toolkit & VMware VIMA 1.0/vMA 4.0
02/14/09	vmwarevSphereHealthCheck.pl\	Reporting/VM Management	ESX(i) 4.x+ & vCenter 4.x+	vSphere SDK for Perl 4.0 & VMware vMA 4.0
02/13/09	ghettoShutdown.pl\	VM Management	ESX(i) 3.5u2+	VMware VIMA 1.0
01/26/09	vmwareHealthCheck.pl\	Reporting/VM Management	ESX(i) 3.5+ & vCenter 2.5+	VI Perl Toolkit & VMware VIMA 1.0
01/13/09	ghettoQuickMigrate.sh\	VM Management	ESX(i) 3.5u2+	VI Perl Toolkit & VMware VIMA 1.0
01/05/09	ghettoClone.sh\	VM Deployment/Management	ESX(i) 3.5u2+	VI Perl Toolkit & VMware VIMA 1.0
11/03/08	rdm.sh\	VM Management	ESX(i) 3.5+	ESX Service Console and/or ESXi Busybox Console
11/17/08	ghettoVCB.sh\	Backup/Recovery	ESX(i) 3.5 and 4.0+	ESX Service Console and/or ESXi Busybox Console
09/30/08	ghetto-esxi-linked-clones.sh\	VM Deployment/Management	ESXi 3.5+	ESXi Busybox Console
09/20/08	npiv_wwn_discovery-esx.sh\	VM Management	ESX(i) 3.5+	ESX Service Console
09/20/08	npiv_wwn_discovery-esxi.sh\	VM Management	ESX(i) 3.5+	ESXi Busybox Console
07/05/08	my-vmware-cmd.sh\	VM Management	ESX(i) 3.5+	ESX Service Console/ESXi Busybox Console/VMware VIMA/UNIX or Linux Console
06/26/08	ghetto-esx-linked-clones.sh\	VM Deployment/Management	ESX 3.x+	ESX Service Console

FYI - "v" in vGhetto is not vSphere ... it's very

Hope you guys/gals enjoy

=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

Capacity Planner Resolution Paths

Thu, 23 Apr 2009 21:36:00 GMT

VMware vSphere Health Check Report v1.2

Wed, 22 Apr 2009 06:24:36 GMT

Description:

This script generates a health check report similiar to that of vmwareHealthCheckScript but for the new vSphere release of VMware ESX(i) 4.x and VMware vCenter 4.x and it's managed entities.

The script reports on the following:

New New licensing format/summary
New EVC Enabled information
New Cluster VM monitoring
New Cluster Host monitoring
New # of VMotions within a cluster
New Datastore uncommitted info
New CPU power management info
New VM info (FT, Record/Replay, Clean Poweroff)
New Host IPv6, FT, SSL Thumbprint
New Host Profiles
New vApp information
New Distributed vSwitch information
vCenter Build/Release
Active Sessions
ESX/ESXi Build/Release
Cluster(s) Name/Statistics (Hosts,CPU and MEM availabity, HA,DRS and DPM enabled, Resource Pools, Health)
ESX/ESXi Hardware configuration (NICs/HBAs)
ESX/ESXi Hardware Health Sensor via CIM
ESX/ESXi State
ESX/ESXi Configurations (for detailed information, use detail-hosts option)
ESX/ESXi Multipathing Info (only available in host or detail-hosts option)
ESX/ESXi Datastore summary
ESX/ESXi LUN summary
ESX/ESXi Portgroup summary
ESX/ESXi Hostd logs
CDP Summary
Recent Tasks
Virtual Machine summary
VM Storage summary
VM Network summary
VM w/Snapshots
VM w/Snapshot delta age
VM w/RDMs
VM w/NPIV enabled
VM w/connected CD-ROMs
VM w/connected Floppys

For more details, please take a look at the sample reports located here (standard) and here (detailed)

Requirements:

vSphere SDK for Perl (formally VI Perl Toolkit) or VMware vMA 4.0 (formally VIMA)
VMware vCenter 4.x+ and/or ESX(i) 4.x+

Usage

[vi-admin@rafaeli ~]$ ./vmwarevSphereHealthCheck.pl
Required command option 'type' not specified.

Synopsis: ./vmwarevSphereHealthCheck.pl OPTIONS


Command-specific options:
   --cluster
      The name of a vCenter cluster
   --datacenter
      The name of a vCenter datacenter
   --logcount
      The number of lines to output from hostd logs
   --report
      The name of the report to output. Please add ".html" extension
   --type (required)
      Type: [vcenter|datacenter|cluster|host|detail-hosts]


Common VI options:
   --config (variable VI_CONFIG)
      Location of the VI Perl configuration file
   --credstore (variable VI_CREDSTORE)
      Name of the credential store file defaults to <HOME>/.vmware/credstore/vicredentials.xml on Linux and <APPDATA>/VMware/credstore/vicredentials.xml on Windows
   --encoding (variable VI_ENCODING, default 'utf8')
      Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
   --help
      Display usage information for the script
   --passthroughauth (variable VI_PASSTHROUGHAUTH)
      Attempt to use pass-through authentication
   --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
      Pass-through authentication negotiation package
   --password (variable VI_PASSWORD)
      Password
   --portnumber (variable VI_PORTNUMBER)
      Port used to connect to server
   --protocol (variable VI_PROTOCOL, default 'https')
      Protocol used to connect to server
   --savesessionfile (variable VI_SAVESESSIONFILE)
      File to save session ID/cookie to utilize
   --server (variable VI_SERVER, default 'localhost')
      VI server to connect to. Required if url is not present
   --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
      Service path used to connect to server
   --sessionfile (variable VI_SESSIONFILE)
      File containing session ID/cookie to utilize
   --url (variable VI_URL)
      VI SDK URL to connect to. Required if server is not present
   --username (variable VI_USERNAME)
      Username
   --verbose (variable VI_VERBOSE)
      Display additional debugging information
   --version
      Display version information for the script

Supported Use Cases:

1) This will collect information about all cluster(s)/host(s) providing vCenter Server:

./vmwarevSphereHealthCheck.pl --server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type vcenter

2) This will collect information about a specific datacenter provided with vCenter Server:

./vmwarevSphereHealthCheck.pl --server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type datacenter --datacenter DATACENTER_NAME

3) This will collect information about a specific cluster provided with vCenter Server:

./vmwarevSphereHealthCheck.pl --server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type cluster --cluster CLUSTER_NAME

4) This will collect information about a specific host provided with ESX/ESXi Server:

./vmwarevSphereHealthCheck.pl --server ESX_ESXi_SERVER --username ESX_ESXi_USERNAME --password ESX_ESXi_PASSWORD --type host

5) This will collect detail information about all ESX/ESXi hosts under vCenter (for datacenter/cluster specific, please look at option 1-3).

This option is meant to provide a report similiar to that of esxhealthscript within the capablities of the VI API utilizing the VI Perl Toolkit. For a high level view of your VMware environment, please use the other options, as this pertains to detail configurations of your hosts.

./vmwarevSphereHealthCheck.pl -server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type detail-hosts

To specify the number of the latest lines in the hostd logs, use --logcount [number_of_lines]
(e.g.)

./vmwarevSphereHealthCheck.pl -server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type detail-hosts --logcount 20

Custom Configurations

Edit the following at the top of the script to configure the warning levels for the various resource consumptions (disk,memory,cpu):

####################################
# resource consumption warnings
####################################
# yellow < 30 %
my $yellow_warn = 30;

# orange < 15 %
my $orange_warn = 15;

# red < 10%
my $red_warn = 10;

Edit the following at the top of the script to configure the number of days aged for a snapshot delta file:

######################################
# vm snapshot age warnings
######################################
# yellow < 15 days
my $snap_yellow_warn = 15;

# orange < 30 days
my $snap_orange_warn = 30;

# red < 60 days+
my $snap_red_warn = 60;

Changes

##########################################################################
05-24-2009 - v1.2
Fixes:
-Minor fix in nDVS that was causing script to halt

##########################################################################

05-04-2009 - v1.1
Enhancements:
-Display additional VM Storage details (committed,uncommitted and unshared)

TO-DO/WIP

-None atm

Known Issues

-None atm

Storage Performance: VMFS and Protocols

Fri, 13 Mar 2009 00:12:37 GMT

Introduction

VMware's customers are always asking us about the storage stack. Without exception, the two most common questions about our storage system performance are:

Which storage protocol performs best?
Does VMFS scale to meet the demands of many servers and VMs?

This document will contain a few of the points needed to help understand this issue.

Storage Protocols

VMware published a paper comparing storage protocols in 2008. This paper detailed the two key characteristics of ESX's storage stack:

The hypervisor is easily able to drive the storage connection to link speed.
Configurations where protocol management happens in the HBA (Fibre Channel and HW iSCSI) are more CPU efficient.

On the first note, take the following graph, taken from page three of the paper:

Note that in this case all four test cases drive the storage to link speed. That's 2 Gb/s with the Fibre Channel HBA and 1 Gb/s with the other three. In short, if throughput is your goal, make decisions based on link speed. If you check through the rest of the paper, you'll see that response time is similar for all of the configurations, as well. But you will see slight differences in throughput in some of the protocols.

This brings us to the second point from above: less work is done by the CPU when protocol management can be off-loaded to the HBA. This means that FC and HW iSCSI HBAs will have additional CPU cycles for the VMs' work. It can also explain the slight differences in throughput in the other graphs in the paper. The efficiency results quoted in the paper are here:

The increased overheads of running software iSCSI or NFS are due to the VMkernel managing those protocols. It's worth noting that the proliferation of iSCSI in the enterprise has led VMware to spend considerable effort to improve the efficiency of SW iSCSI. Expect its efficiency to improve dramatically in the following releases.

VMFS Scalability

Many in the industry erroneously believe that VMFS won't scale as storage demands grow. Often SCSI reservations and disk locking are cited as the technical-sounding but vaguely-supported reason for this claim. It's worth sampling data from our scalable storage performance paper to debunk this myth.

This chart is a favorite in our world-wide tours as we address VMFS scalability. It's was first introduced in a VMFS scalability blog article that went live in February of 2008. It shows the results of using 64 hosts to generate a variety of traffic on a single VMFS volume. And it's a wealth of information on VMFS and storage access patterns. For instance:

The aggregate number of random writes, in cyan in the middle, maintains perfectly flat linear scalability as the host count grows from 1 to 64.
The aggregate number of random reads is initially limited by the few disks being accessed but ultimately matches the throughput of random writes as many disks come to bear to serve the large number of random reads.
The sequential read activity, which highlights the strengths of today's arrays, demonstrates the largest total throughput which only slightly drops as the array manages so many connections.
But the sequential read activity drops off dramatically as hosts are added.

This last example showing degradation in aggregate sequential read capability is an artifact of the workload that is very important to database administrators: multiple sequential reads approximate random activity. Why is this? As many hosts request more and more sequential data, the array interleaves these requests to maintain response times. This means that the sequential accesses get "shuffled" which results in a random access pattern.

In short, VMFS has no scalability problems as many hosts drive tremendous amounts of traffic to a single volume. If the data isn't convincing enough, consider the following: there are no SCSI reservations used during normal data access. This means that there are no scalability limitations as a result of virtual machine storage access. A word of caution, though: the file system is locked during administrative operations that change the metadata on the volume. This means that virtual machine creation or destruction can will result in file system locks. Perform these operations off of peak hours.

Best Practices for IBM Lotus Domino

Mon, 09 Mar 2009 22:19:19 GMT

Introduction

This page provides the best practices for virtualizing IBM Lotus Domino using VMware Infrastructure. This list is based on my VMworld 2008 session EA2348.

General Recommendations

Use newer hardware

Supports latest hardware assist technologies, larger on-processor cache
64-bit may not perform better with older hardware

Use VMware ESX , that uses bare-metal or hypervisor architecture

You can start with VMware ESXi - the free version.
Do not use the VMware Workstation or Server, that use the hosted architecture.

VMware ESX allows you the choice of virtualization technology best suited for your workload

Hardware Assist (AMD, Intel) (both CPU and MMU virtualization) if your hardware supports it
Paravirtualization (if you use SLES for your Domino deployment)
Binary Translation

Migrate to latest version of ESX

E.g. ESX 3.5 defaults to 2nd Generation Hardware Assist if available, has several I/O Performance improvements

Lotus Domino: Plan to migrate to version 8.0

Significant performance improvements, specially disk I/O

Provide Redundancy to the ESX host

Power supplies, HBAs, NICs, Network and SAN switches
E.g. NIC teaming, HBA multi-pathing

Leverage VMotion, Storage VMotion, DRS and HA for higher Domino availability

VM configuration

64-bit OS recommended

VI3 supports all x86 OSs that Domino supports: Windows, SLES, RHEL
Improved memory limits in 64-bit OS helps cache more data, and thus avoid disk IO. Reduces response times, and hence increasing the number of users
Increase VM memory when running in 64-bit guest OS

64-bit may not perform better with older hardware

E.g. 64-bit Windows more sensitive to onboard L2/L3 chip caches
Microsoft reports 10-15% degradation with older hardware

Guest Operating System:

Windows: Use 2003 SP2
- Microsoft eliminated most APIC TPR accesses, improves virtual performance
Linux: Use 2.6.18-53.1.4 kernel or later to use divider patch
- Some older Linux versions have a 1Khz timer rate
- Put divider=10 on the end of the kernel line in grub.conf and reboot

VM Time Synchronization

Use VMware Tools time synchronization within the virtual machine
Enable ESX server NTP daemon to sync with external stratum NTP source (VMware Knowledge Base ID# 1339)
Disable OS Time Service
- Windows: w32time service
- Linux: NTP daemon

Storage

Storage configuration is absolutely critical; most performance problems traced to this

Number of spindles, RAID configuration, drive speed, controller cache settings, queue depths – all make a big difference

Align partitions

For VMFS, use Virtual Center to create partitions
Check Recommendations for Aligning VMFS Partitions

Use separate, dedicated LUNs for OS/Domino, data and transaction logs

Separate the IO at physical disk level, not simply logical LUNs
Make sure these LUNs have enough spindles to support the IO demands
Fewer spindles or too many VMDK files on single VMFS LUN can substantially increase disk IO latencies
Check Scalable Storage Performance to understand the details

RAID configuration

RAID 1+0 for Data, RAID 0 for Log

Cache settings

Write policy to "write back“, read policy to "read ahead“

Queue Depths

Increase to 255

Storage Protocol: Fibre Channel or iSCSI

Link speed typically limits the scalability, NOT VMware ESX
Link speed is maintained up to 32 virtual machines for each storage connection option
For details check Comparison of Storage Protocol Performance

Storage Partition: VMFS or RDM

VMFS is recommended
- Leverage templates and quick provisioning
- Fewer LUNs means you don’t have to watch Heap
Performance difference between VMFS and RDM not significant
- For details check Performance Characterization of VMFS and RDM Using a SAN

VI3 supports latest storage technologies: leverage these if you have already invested or plan to invest

Fibre channel – 8Gbps connectivity
ISCSI – 10GigE network connectivity, Jumbo Frames
Infiniband support

Virtual CPUs

The number of vCPUs per VM depends on the number of users to be supported

Start with uni-processor, may be enough
Try not to over-provision vCPUs in the guest CPU

Verify CPU compatibility for VMotion

Memory

Increasing memory to avoid disk I/O is most technique to improve performance
More available memory = more Lotus Domino Cache

Increase NSF_DbCache_Maxentries value

Leverage the higher VI 3.5 support 64GB memory limit per VM in VI 3.5 when using 64-bit guest OS for Domino

64-bit OSs can take advantage of larger memory limits for file caching

Leverage NUMA optimizations in VI3

When using NUMA, try to fit the VM within a single node to avoid latencies accessing memory on remote nodes

Networking

Use dedicated NICs based on the network traffic

E.g. separate NICs for mail and replication traffic

Use NIC Teaming & VLAN Trunking

Use Enhanced VMXNET driver with TSO and Jumbo Frames support

Enable TCP transmit coalescing

Check the SPECweb paper for details.

Co-located VMs outperform physical 1Gbps network speed

Resource Management

Use proportional and absolute mechanisms to control VM priorities

Shares, reservations, and limits for CPU and memory
Shares for virtual disks
Traffic shaping for network

Faster migration resulting in better load balancing when using

Smaller VMs
Lesser memory reservations for VMs

Affinity rules for VM placement

E.g. Directory, Mail Server VMs on same ESX

Deployment

Virtualization Assessment

Capacity Planner
Benchmark against Information Warehouse

Easy migration

VMware Converter – both hot and cold cloning
Start with RDM to point to existing data/ transaction log LUNs, but move to VMFS later

Easier change management and quicker provisioning

Templates and clones for easy provisioning

Post VM Build Configuration

deloachjd@yahoo.com — Thu, 26 Feb 2009 20:11:06 GMT

Real-World Virtualization Testing on the HP ProLiant DL785 G5 - A network and systems architecture consultant’s review

Sat, 21 Feb 2009 02:24:37 GMT

i can't install vmware tools in fedora 9

Fri, 20 Feb 2009 05:13:15 GMT

i installed vmware tools rpm ..

but.. it doesn't work.

then i Extracted and installed vmware tools-6.0.0-45731.tar.gz

but there was a problem

What is the location of the directory of C header files that match your running
kernel? [/usr/src/linux/include]

and i downloaded (vmware-any-any-update117.tar.gz )

but.. i can't install (vmware-any-any-update117.tar.gz )

I am so tired

if there anysone knwo about that,pls do let me know.

best regard.

RIM - Key Resources for Virtualizing BlackBerry Enterprise

Mon, 16 Feb 2009 18:32:51 GMT

Support & Licensing Information:

General BlackBerry Support for BES on VMware:

http://www.blackberry.com/btsc/dynamickc.do?externalId=KB04405&sliceId=SAL_Public&command=show&forward=nonthreadedKC&kcId=KB04405

BlackBerry Enterprise Server for Microsoft Exchange - Compatibility Matrix:

http://na.blackberry.com/eng/support/software/server_exchange_ver_december08.pdf

Installation Guide BES 4.1.6

http://na.blackberry.com/eng/deliverables/2751/BESX_Install_Guide_427146_11.pdf

Technical Publications:

Technical case study on how VMware IT is virtualizing BlackBerry Enterprise:

http://www.vmware.com/resources/techresources/10032

Success Stories:

Tuscon Electric Power

http://www.vmware.com/files/pdf/customers/09Q1_isv_vmw_TEP_english.pdf

Interfaith Medical Center

http://www.vmware.com/files/pdf/customers/08Q4_isv_vmw_Interfaith_medical_english.pdf

Interior Health Authority

http://www.vmware.com/files/pdf/customers/08Q4_isv_vmw_IHA_english.pdf

Peerless Clothing

http://www.vmware.com/files/pdf/customers/08Q3_ss_vmw_peerless_english.pdf

The George Washington University

http://www.vmware.com/files/pdf/customers/08Q3_ss_vmw_gwu_english.pdf

Nuvasive

http://www.vmware.com/files/pdf/customers/08Q3_isv_vmw_nuvasive_english.pdf

Alstom

http://www.vmware.com/files/pdf/customers/08Q3_cs_vmw_alstom_english.pdf

Kronos

http://vmware.com/files/pdf/customers/08Q2_ss_vmw_kronos_english.pdf

Hackensack University Medical Center

http://www.vmware.com/files/pdf/customers/08Q2_ss_vmw_humc_english.pdf

Georgian College

http://www.vmware.com/files/pdf/customers/08Q2_ss_vmw_georgian_college_english_2logos.pdf

Uptime Software

http://www.vmware.com/files/pdf/customers/08Q1_ss_vmw_uptime_english.pdf

Ryerson University

http://www.vmware.com/files/pdf/customers/08Q1_ss_vmw_ryerson_english.pdf

Los Angeles University Preschool

http://www.vmware.com/files/pdf/customers/08Q2_ss_vmw_LAUP_English.pdf

Guest_OS install problems

Thu, 05 Feb 2009 17:51:01 GMT

VMWare ESXi Server install on HP DL380 G5 platform. 32 GB Ram, 8 146GB SAS HD (Raid 6), 2 Xeon 5450 Processors.

As the first Guest_OS I created a virtual machine put an ISOs (Disk1 and Disk2) on the Datastore pointed the CD/DVD at the ISOs checked "connect at power on". Powered on the machine started installing the OS. Got to the first interactive response "Press Enter to install Windows Server 2003 SE".

Enter would not workfrom my laptop that has the VMWare Infastructure Client. In fact no options were available to me.

I restarted the virtual server the ISO did not pick up and have not been able to get the ISO to start the install since.

1- Any suggestions on getting the ISO to boot?
2- After I get the ISO to boot again any ideas on the keyboard.

Thanks for your responses.

VMware Health Check Report v0.9.4

Sat, 31 Jan 2009 00:35:22 GMT

Description:

This script generates a health check report of all attributes pertaining to a single ESX/ESXi 3.x+ host or VMware vCenter and its managed entities (i.e. clusters and individual ESX/ESXi 3.x+ hosts). The generated report is based off of some earlier Powershell script work created by Ivo Beerens/Duncan Eppping.

I was primarily motivated on this project because it allowed me to start learning about how to utilize the VI API through the VI Perl Toolkit. Existing information provided by the RCLI and VI Perl Toolkit default utilities are useful though they can be limited. This gave me an opportunity to learn how to write my own Perl scripts that may ultimately lead to more functional tools that the community as well as VI administrators can utilize.

The generated health check report contains most of what Ivo Beerens has included in his latest release. For this project, I am attempting to create a non-Powershell solution as opposed to the direction taken by Ivo and Duncan. This Perl script will generally be a work in progress as I continue to learn about the API. Moreover, I hope that this will be something of use to administrators especially those seeking to query important pieces of information such as VM(s) containing snapshots and/or RDM(s).

The script reports on the following:

vCenter Build/Release
License summary
Active Sessions
ESX/ESXi Build/Release
Cluster(s) Name/Statistics (Hosts,CPU and MEM availabity, HA,DRS and DPM enabled, Resource Pools, Health)
ESX/ESXi Hardware configuration (NICs/HBAs)
ESX/ESXi Hardware Health Sensor via CIM
ESX/ESXi State
ESX/ESXi Configurations (for detailed information, use detail-hosts option)
ESX/ESXi Multipathing Info (only available in host or detail-hosts option)
ESX/ESXi Datastore summary
ESX/ESXi LUN summary
ESX/ESXi Portgroup summary
ESX/ESXi Hostd logs
CDP Summary
Recent Tasks
Virtual Machine summary
VM Storage summary
VM Network summary
VM w/Snapshots
VM w/Snapshot delta age
VM w/RDMs
VM w/NPIV enabled
VM w/connected CD-ROMs
VM w/connected Floppys

For more details, please take a look at the sample reports located here and here

Requirements:

VI Perl Toolkit or VMware VIMA
VMware vCenter 2.5+ and/or ESX 3.x+ / ESXi 3.5+

Usage

[vi-admin@vima-primp-industries ~]$ ./vmwareHealthCheck.pl
Required command option 'type' not specified.

Synopsis: ./vmwareHealthCheck.pl OPTIONS


Command-specific options:
   --cluster
      The name of a vCenter cluster
   --datacenter
      The name of a vCenter datacenter
   --logcount
      The number of lines to output from hostd logs
   --report
      The name of the report to output
   --type (required)
      Type: [vcenter|datacenter|cluster|host|detail-hosts]


Common VI options:
   --config (variable VI_CONFIG)
      Location of the VI Perl configuration file
   --encoding (variable VI_ENCODING, default 'utf8')
      Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
   --help
      Display usage information for the script
   --passthroughauth (variable VI_PASSTHROUGHAUTH)
      Attempt to use pass-through authentication
   --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
      Pass-through authentication negotiation package
   --password (variable VI_PASSWORD)
      Password
   --portnumber (variable VI_PORTNUMBER)
      Port used to connect to server
   --protocol (variable VI_PROTOCOL, default 'https')
      Protocol used to connect to server
   --savesessionfile (variable VI_SAVESESSIONFILE)
      File to save session ID/cookie to utilize
   --server (variable VI_SERVER, default 'localhost')
      VI server to connect to. Required if url is not present
   --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
      Service path used to connect to server
   --sessionfile (variable VI_SESSIONFILE)
      File containing session ID/cookie to utilize
   --url (variable VI_URL)
      VI SDK URL to connect to. Required if server is not present
   --username (variable VI_USERNAME)
      Username
   --verbose (variable VI_VERBOSE)
      Display additional debugging information
   --version
      Display version information for the script

Supported Use Cases:

1) This will collect information about all cluster(s)/host(s) providing vCenter Server:

./vmwareHealthCheck.pl --server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type vcenter

2) This will collect information about a specific datacenter provided with vCenter Server:

./vmwareHealthCheck.pl --server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type datacenter --datacenter DATACENTER_NAME

3) This will collect information about a specific cluster provided with vCenter Server:

./vmwareHealthCheck.pl --server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type cluster --cluster CLUSTER_NAME

4) This will collect information about a specific host provided with ESX/ESXi Server:

./vmwareHealthCheck.pl --server ESX_ESXi_SERVER --username ESX_ESXi_USERNAME --password ESX_ESXi_PASSWORD --type host

./vmwareHealthCheck.pl -server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type detail-hosts

To specify the number of the latest lines in the hostd logs, use --logcount [number_of_lines]
(e.g.)

./vmwareHealthCheck.pl -server VC_SERVER --username VC_USERNAME --password VC_PASSWORD --type detail-hosts --logcount 20

Custom Configurations

Edit the following at the top of the script to configure the warning levels for the various resource consumptions (disk,memory,cpu):

Edit the following at the top of the script to configure the number of days aged for a snapshot delta file:

Changes

##########################################################################

03-24-2009 - v0.9.4
Fixes:
-There was a bug reported by Duncan Epping and others regarding hosts that were appearing in the wrong cluster with respect to the portgroup listings, this should be fixed.

Enhancements:
-Detail Hardware Health sensor readings provided by CIM
-CDP Summary (individual cdp.pl available)

##########################################################################

02-28-2009 - v0.9
Fixes:
-Due to VMware API bug, the VMDK count per VM is not displaying correctly, a work around has been setup to print # of disks

Enhancements:
-Retrieve script version by using --version
-Cluster Health
-Recent Tasks

##########################################################################

02-11-2009 - v0.8
Fixes:
-Fixed an issue where disconnected VM(s) may cause the script to halt

Enhancements:
-Added hostd logs with --logcount flag to specify the last number of lines (default 15 lines)
-Performance enhancement, script should execute much faster (YMMV)

Example report here

##########################################################################

02-09-2009 - v0.7
Fixes:
-Removed invalid cluster stats (output was not regarding consumption but availablity)
-Fixed an issue that may allow the script to execute on Windows VI Perl Toolkit again(no guarantees)

Enhancements:
-Licensing/features information
-Active Sessions
-ESX UUID
-Additional ESX/ESXi host configurations (available with host or detail-hosts flag)
-Performance increase on script execution (YMMV)
-Added detail-hosts flag which provides additional host configurations (comparable to esxhealthscript)

VMotion enabled
Service Console network summary
VMkernel summary
DNS
Offload capabilities
Diagostic Partition
vswif summary
vSwitch/cdp summary
Firewall known services
...much more

Example report here

##########################################################################

02-05-2009 - v0.6
Fixes:
-Fixed minor typographic errors
-Fixed issue if no additional hardware vendor information is available
-Fixed potential issue with displaying information about datastores not being accessible

Enhancements:
-Display snapshot deleta file(s) age (options configurable by users)
-Display root resource pool information if appliacable

##########################################################################

02-04-2009 - v0.5
Fixes:
-Cleaned up a little of the html structure
-Fixed issue when no ntpServers configured
-Fixed issue when VMware tool status is not available
-Fixed the order of the Adv options
-Fixed Host to Datastore access (If a datastore is presented to hosts that reside in separate clusters, false positives may be included in the output)

Enhancements:
-Added support for --datacenter argument for specifying a specific datacenter in vCenter
-Additional hardware info (possibly AssetTag if configured)
-Display Adv options: Disk.SchedNumReqOutstanding and NFS.LockDisable

##########################################################################

02-01-2009 - v0.4
Fixes:
-Only display summary on valid clusters (w/hosts attached)

Enhancements:
-Display hosts in cluster with inconsistent storage presentation
-Display hosts in cluster with inconsistent portgroup configurations
-Display Disk.UseDeviceReset
-Display Disk.UseLunReset

##########################################################################

02-02-2009 - v0.4
Fixes:
-Fixed color output to highlight the correct values
-Fixed the output of the script execution time to output correct time format

Enhancements:
-Display MB,GB,TB and MHz,GHz instead of just (MB,MHz)
-Display cluster resources % free
-Display hosts in cluster with inconsistent LUN access

TO-DO/WIP

-None atm

Known Issues

1) Firewall information is limited to only known and blessed services, this is a limitation of the VI API

2) Multiple Service Console interfaces may not display correctly

3) SNMP infor (may or may not work)

Twitter: Micro-blogging with VMware and Virtualization Experts

Wed, 28 Jan 2009 01:58:07 GMT

Domino - Key Resources for Deploying Domino on VMware

Tue, 27 Jan 2009 20:47:38 GMT

Support & Licensing Information:

General software support for IBM SWG products in a VMware environment:

http://www-01.ibm.com/support/docview.wss?&uid=wws1e333ce0912f7b152852571f60074d175

VMware product support information for IBM Lotus Domino-based server products:

http://www-01.ibm.com/support/docview.wss?rs=203&uid=swg21106182

Support statement for release 8.0.1 of Notes client, Domino server, and Domino Web Access:

http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21252786

Pricing policies for IBM Software on virtualization, including VMware:

http://www-01.ibm.com/software/lotus/passportadvantage/Counting_Software_licenses_using_specific_virtualization_technologies.html

Pricing policies for IBM support:

http://www-01.ibm.com/software/lotus/passportadvantage/pvu_licensing_for_customers.html

Technical documents:

IBM Lotus Domino and IBM Lotus Sametime on VMware Infrastructure 3: From Planning to Successful Deployment

VMworld Session: Best Practices for Virtualizing IBM Lotus Domino with VI3

http://www.vmworld.com/docs/DOC-2252

VMware Communities document: Best Practices for IBM Lotus Domino:

http://communities.vmware.com/docs/DOC-9671

Success Stories:

VideoTron:

http://www.vmware.com/files/pdf/customers/06Q2_cs_vmw_videotron_english.pdf

T-Systems:

http://www.vmware.com/files/pdf/customers/06Q4_cs_vmw_T-systemsGlobal_English.pdf

Department of Energy - Savannah River Site:

http://www.vmware.com/files/pdf/customers/07Q1_ss_vmw_doe_english.pdf

Wyndham City Council:

http://www.vmware.com/files/pdf/customers/07Q2_ss_vmw_wyndham_city_council_english.pdf

Nworks:

http://www.vmware.com/files/pdf/customers/07Q3_ss_vmw_Nworks_English.pdf

NationWide:

http://www.vmware.com/files/pdf/customers/07Q4_cp_vmw_Nationwide.pdf

London Clubs International:

http://www.vmware.com/files/pdf/customers/07Q2_cs_vmw_london_clubs_international_english.pdf

Denso:

http://www.vmware.com/files/pdf/customers/06Q2_cs_vmw_Denso_english.pdf

EMI Music Publishing:

http://www.vmware.com/files/pdf/customers/08Q1_ss_vmw_emi_english.pdf

PowerSeraya Utilities:

http://www.vmware.com/files/pdf/customers/apac_sg_08Q2_cs_vmw_power_seraya_english.pdf

Exchange - Key Resources for Deploying Exchange on VMware

Tue, 27 Jan 2009 00:06:46 GMT

Welcome the VMTN forum for virtualizing enterprise class e-mail and messaging applications. This post provides an overview of the key resources for running Exchange on VMware.

Most of these resources are also available on the VMware.com Exchange page.

Support & Licensing Information:

White Papers and Technical Documents:

Success Stories:

Virtual SAP presentation by CCI

Mon, 26 Jan 2009 16:28:08 GMT

Procedure for Shrinking Windows VM System/Boot Drive

Thu, 15 Jan 2009 14:26:26 GMT

change First two digits

Mon, 12 Jan 2009 20:42:57 GMT

How change the HDD serial number under VMWare workstation?

I already found where serial number is stored in vmware-vmx file but i can not change First two digits

its always 00 (or 01 for slave, 10 for secondary master and 11 for secondary slave).

http://www.6arab.song.googlepages.com/1111.JPG

Help please, anybody

First two digits of HDD Serial number

Mon, 12 Jan 2009 20:36:31 GMT

How change the HDD serial number under VMWare workstation?

I already found where serial number is stored in vmware-vmx file but i can not change First two digits

its always 00 (or 01 for slave, 10 for secondary master and 11 for secondary slave).

http://www.6arab.song.googlepages.com/1111.JPG

Help please, anybody

Interpreting esxtop Statistics

Wed, 31 Dec 2008 22:48:19 GMT

Section 1. Introduction
Section 2. CPU
Section 2.1 Worlds and Groups
Section 2.2 Global Statistics
Section 2.3 World Statistics
Section 3. Memory
Section 3.1 Machine Memory and Guest Physical Memory
Section 3.2 Global Statistics
Section 3.3 Group Statistics
Section 4 Disk
Section 4.1 Adapter, Device, VM screens
Section 4.2 Disk Statistics
Section 4.2.1 I/O Throughput Statistics
Section 4.2.2 Latency Statistics
Section 4.2.3 Queue Statistics
Section 4.2.4 Error Statistics
Section 4.2.5 PAE Statistics
Section 4.2.6 Split Statistics
Section 4.3 Batch Mode Output
Section 5 Network
Section 5.1 Port
Section 5.2 Port Statistics
Section 6. Interrupt
Section 7. Batch Mode

Section 1. Introduction

Esxtop allows monitoring and collection of data for all system resources: CPU, memory, disk and network. When used interactively, this data can be viewed on different types of screens; one each for CPU statistics, memory statistics, network statistics and disk adapter statistics. In addition to the disk adapter statistics in earlier versions, starting with ESX3.5, disk statistics at the device and VM level are also available. Starting with ESX 4.0, esxtop has an interrupt statistics screen. In the batch mode, data can be redirected to a file for offline uses.

Many esxtop statistics are computed as rates, e.g. CPU statistics %USED. A rate is computed based on the refresh interval, the time between successive snapshots. For example, %USED = ( CPU used time at snapshot 2 - CPU used time at snapshot 1 ) / time elapsed between snapshots. The default refresh interval can be changed by the command line option "-d", or the interactive command 's'. The return key can be pressed to force a refresh.

In each screen, data is presented at different levels of aggregation. It is possible to drill down to expanded views of this data. Each screen provides different expansion options.

It is possible to select all or some fields for which data collection is done. In the case of interactive use of esxtop, the order in which the selected fields are displayed can be selected.

In the following sections, this document will describe the esxtop statistics shown by each screen and their usage.

Section 2. CPU

Section 2.1 Worlds and Groups

Esxtop uses worlds and groups as the entities to show CPU usage. A world is an ESX Server VMkernel schedulable entity, similar to a process or thread in other operating systems. A group contains multiple worlds.

Let's use a VM as an example. A powered-on VM has a corresponding group, which contains multiple worlds. In ESX 4.0, there is one vcpu (hypervisor) world corresponding to each VCPU of the VM. The guest activities are represented mostly by the vcpu worlds. (In ESX 3.5, esxtop shows a vmm world and a vcpu world for each VCPU. The guest activities are represented mostly by the vmm worlds.) Besides the vcpu worlds, there are other assisting worlds, such as a MKS world and a VMX world. The MKS world assists mouse/keyboard/screen virtualization. The VMX world assists the vcpu worlds (the hypervisor). The usage of the VMX world is out of the scope of this document. In ESX 4.0, there is only one vmx world. (In ESX 3.5, there are two vmx worlds for each VM.)

There are other groups besides VM groups. Let's go through a few examples:

The "idle" group is the container for the idle worlds, each of which corresponds to one PCPU.
The "system" group contains the VMKernel system worlds.
The "helper" group contains the helper worlds that assist VMKernel operations.
In classic ESX, the "console" group is for the console OS, which runs ESX management processes. In ESXi, these ESX management processes are running as user worlds directly on VMKernel. So, on an ESXi box you can see much more groups than on a classic ESX, but not the "console" group.

Note that groups can be organized in a hierarchical manner in ESX. However, esxtop shows, in a flat form, the groups that contain some worlds. More detailed discussion on the groups are out of the scope.

Q: Why can't we find any vmm worlds for a VM in ESX 4.0?
A: Before ESX 4.0, each VCPU has two worlds "vmm" and "vcpu". In ESX 4.0, cpu scheduler merges their statistics to one vcpu world. So, CPU stats won't show vmm worlds. This is not a problem.

Section 2.2 Global Statistics

"up time"

The elapsed time since the server has been powered on.

"number of worlds"

The total number of worlds on ESX Server.

"CPU load average"

The arithmetic mean of CPU loads in 1 minute, 5 minutes, and 15 minutes, based on 6-second samples. CPU load accounts the run time and ready time for all the groups on the host.

"PCPU(%)"

The percentage CPU utilization per physical CPU.

Q: What does it mean if PCPU% is high?
A: It means that you are using lots of resource. (a) If all of the PCPUs are near 100%, it is possible that you are overcommiting your cpu resource. You need to check RDY% of the groups in the system to verify cpu overcommitment. Refer to RDY% below. (b) If some PCPUs stay near 100%, but others are not, there might be an imbalance issue. Note that you'd better monitor the system for a few minutes to verify whether the same PCPUs are using ~100% CPU. If so, check VM CPU affinity settings.

"used total"

Sum( PCPU(%) ) / number of PCPUs

"LCPU(%)"

The percentage CPU utilization per logical CPU. The CPU used percentages for the logical CPUs belonging to a package add up to 100%. This line is displayed only if hyper-threading is present and enabled.

"CCPU(%)"

Percentages of total CPU time as reported by the ESX Service Console. "us" is for percentage user time, "sy" is for percentage system time, "id" is for percentage idle time and "wa" is for percentage wait time. "cs/sec" is for the context switches per second recorded by the ESX Service Console.

Q: What's the difference of CCPU% and the console group stats?
A: CCPU% is measured by the COS. "console" group CPU stats is measured by VMKernel. The stats are related, but not the same.

Section 2.3 World Statistics

A group statistics is the sum of world statistics for all the worlds contained in that group. So, this section focuses on worlds. You may apply the description to the group as well, unless stated otherwise.

ESX can make use of the Hyperthreading technology, so, the performance counters takes Hyperthreading into consideration as well. But, to simplify this document, we will ignore HT related issues. Please refer to "Resource Management Guide" for more details.

"%USED"

The percentage physical CPU time accounted to the world. If a system service runs on behalf of this world, the time spent by that service (i.e. %SYS) should be charged to this world. If not, the time spent (i.e. %OVRLP) should not be charged against this world. See notes on %SYS and %OVRLP.

%USED = %RUN + %SYS - %OVRLP

Q: Is it possible that %USED of a world is greater than 100%?
A: Yes, if the system service runs on a different PCPU for this world. It may happen when your VM has heavy I/O.

Q: For an SMP VM, why does VCPU 0 have higher CPU usage than others?
A: The system services are accounted to VCPU 0. You may see higher %USED on VCPU 0 than others, although the run time (%RUN) are balanced for all the VCPUs. This is not a problem for CPU scheduling, but only the way VMKernel does the CPU accounting.

Q: What is the maximum %USED for a VM group?
A: The group stats is the sum of the worlds. So, the maximum %USED = NWLD * 100%. NWLD is the number of worlds in the group.

Typically, worlds other than VCPU worlds are waiting for events most of time, not costing too much CPU cycles. Among all the worlds, VCPU worlds represent best the guest. Therefore, %USED for a VM group usually do not exceed Number of VCPUs * 100%.

Q: What does it mean if %USED of a VM is high?
A: The VM is using lots of CPU resource. You may expand to worlds to see what worlds are using most of them.

"%SYS"

The percentage of time spent by system services on behalf of the world. The possible system services are interrupt handlers, bottom halves, and system worlds.

Q: What does it mean if %SYS is high?
A: It usually means that your VM has heavy I/O.

Q: Are %USED and %SYS similar to user time and system time in Linux?
A: No. They are totally different. For Linux OS, user (system) time for a process is the time spent in user (kernel) mode. For ESX, %USED is for the accounted time and %SYS is for the system service time.

"%OVRLP"

The percentage of time spent by system services on behalf of other worlds. In more detail, let's use an example.

When World 'W1' is running, a system service 'S' interrupts 'W1' and services World 'W2'. The time spent by 'S', annotated as 't', is included in the run time of 'W1'. We use %OVRLP of 'W1' to show this time. This time 't' is accounted to %SYS of 'W2', as well.

Again, let's take a look at "%USED = %RUN + %SYS - %OVRLP". For 'W1', 't' is included in %RUN and %OVRLP, not in %SYS. By subtracting %OVRLP from %RUN, we do not account 't' in %USED of 'W1'. For 'W2', 't' is included in %SYS, not in %RUN or %OVRLP. By adding %SYS, we accounted 't' to %USED of 'W2'.

Q: What does it mean if %OVRLP of a VM is high?
A: It usually means the host has heavy I/O. So, the system services are busy handling I/O. Note that %OVRLP of a VM group may or may not be spent on behalf of this VM. It is the sum of %OVRLP for all the worlds in this group.

"%RUN"

The percentage of total scheduled time for the world to run.

Q: What is the difference between %USED and %RUN?
A: %USED = %RUN + %SYS - %OVRLP. (%USED takes care of the system service time.) Details above.

Q: What does it mean if %RUN of a VM is high?
A: The VM is using lots of CPU resource. It does not necessarily mean the VM is under resource constraint. Check the description of %RDY below, for determining CPU contention.

"%RDY"

The percentage of time the world was ready to run.

A world in a run queue is waiting for CPU scheduler to let it run on a PCPU. %RDY accounts the percentage of this time. So, it is always smaller than 100%.

Q: How do I know CPU resource is under contention?
A: %RDY is a main indicator. But, it is not sufficient by itself.

If a "CPU Limit" is set to a VM's resource settings, the VM will be deliberately held from scheduled to a PCPU when it uses up its allocated CPU resource. This may happen even when there is plenty of free CPU cycles. This time deliberately held by scheduler is shown by "%MLMTD", which will be describe next. Note that %RDY includes %MLMTD. For, for CPU contention, we will use "%RDY - %MLMTD". So, if "%RDY - %MLMTD" is high, e.g., larger than 20%, you may experience CPU contention.

What is the recommended threshold? Well, it depends. As a try, we could start with 20%. If your application speed in the VM is OK, you may tolerate higher threshold. Otherwise, lower.

Q: How do we break down 100% for the world state times?
A: A world can be in different states, either scheduled to run, ready to run but not scheduled, or not ready to run (waiting for some events).
100% = %RUN + %READY + %CSTP + %WAIT.
Check the description of %CSTP and %WAIT below.

Q: What does it mean if %RDY of a VM is high?
A: It means the VM is possibly under resource contention. Check "%MLMTD" as well. If "%MLMTD" is high, you may raise the "CPU limit" setting for the VM. If "%RDY - %MLMTD" is high, the VM is under CPU contention.

"%MLMTD"

The percentage of time the world was ready to run but deliberately wasn't scheduled because that would violate the "CPU limit" settings.

Note that %MLMTD is included in %RDY.

Q: What does it mean if %MLMTD of a VM is high?
A: The VM cannot run because of the "CPU limit" setting. If you want to improve the performance of this VM, you may increase its limit. However, keep in mind that it may reduce the performance of others.

"%CSTP"

The percentage of time the world spent in ready, co-deschedule state. This co-deschedule state is only meaningful for SMP VMs. Roughly speaking, ESX CPU scheduler deliberately puts a VCPU in this state, if this VCPU advances much farther than other VCPUs.

Q: What does it mean if %CSTP is high?
A: It usually means the VM workload does not use VCPUs in a balanced fashion. The VCPU with high %CSTP is used much more often than the others. Do you really need all those VCPUs? Do you pin the guest application to the VCPUs?

"%WAIT"

The percentage of time the world spent in wait state.

This %WAIT is the total wait time. I.e., the world is waiting for some VMKernel resource. This wait time includes I/O wait time, idle time and among other resources. Idle time is presented as %IDLE.

Q: How do I know the VCPU world is waiting for I/O events?
A: %WAIT - %IDLE can give you an estimate on how much CPU time is spent in waiting I/O events. This is an estimate only, because the world may be waiting for resources other than I/O. Note that we should only do this for VMM worlds, not the other kind of worlds. Because VMM worlds represent the guest behavior the best. For disk I/O, another alternative is to read the disk latency stats which we will explain in the disk section.

Q: How do I know the VM group is waiting for I/O events?
A: For a VM, there are other worlds besides the VCPUs, such as a mks world and a VMX world. Most of time, the other worlds are waiting for events. So, you will see ~100% %WAIT for those worlds. If you want to know whether the guest is waiting for I/O events, you'd better expand the group and analyze the VCPU worlds as stated above.

Since %IDLE makes no sense to the worlds other than VCPUs, we may use the group stats to estimate the guest I/O wait by "%WAIT - %IDLE - 100% * (NWLD - NVCPU)". Here, NWLD is the number of worlds in the group; NVCPU is the number of VCPUs. This is a very rough estimate, due to two reasons. (1) The world may be waiting for resources other than I/O. (2) We assume the other assisting worlds are not active, which may not be true.

Again, for disk I/O, another alternative is to read the disk latency stats which we will explain in the disk section.

Q: Why do I always see a high %WAIT for VMX/mks worlds?
A: This is normal. That means there are not too much activities on them.

Q: Why do I see a high %WAIT for a VM group?
A: For a VM, there are other worlds besides the VCPUs, such as a mks world and VMX worlds. These worlds are waiting for events most of time.

"%IDLE"

The percentage of time the VCPU world is in idle loop. Note that %IDLE is included in %WAIT. Also note that %IDLE only makes sense to VCPU world. The other worlds do not have idle loops, so, %IDLE is zero for them.

"%SWPWT"

The percentage of time the world is waiting for the ESX VMKernel swapping memory. The %SWPWT (swap wait) time is included in the %WAIT time. This is a new statistics added in ESX 4.0.

Q: Why do I see a high %SWPWT for a VM group?
A: The VM is swapping memory.

Section 3. Memory

Section 3.1 Machine Memory and Guest Physical Memory

It is important to note that some statistics refer to guest physical memory while others refer to machine memory. "Guest physical memory" is the virtual-hardware physical memory presented to the VM. "Machine memory" is actual physical RAM in the ESX host. Let's use the following figure to explain. In the figure, two VMs are running on an ESX host, where each block represents 4 KB of memory and each color represents a different set of data on a block.

Inside each VM, the guest OS maps the virutal memory to its physical memory. ESX Kernel maps the guest physical memory to machine memory. Due to ESX Page Sharing technology, guest physical pages with the same content can be mapped to the same machine page.

Section 3.2 Global Statistics

"MEM overcommit avg"

Average memory overcommit level in 1-min, 5-min, 15-min (EWMA).

Memory overcommit is the ratio of total requested memory and the "managed memory" minus 1. VMKernel computes the total requested memory as a sum of the following components: (a) VM configured memory (or memory limit setting if set), (b) the user world memory, (c) the reserved overhead memory. (Overhead memory will be discussed in more detail for "OVHD" and "OVHDMAX" in Section 3.3.)

"managed memory" will be defined in "VMKMEM" section.

Q: What does it mean if overcommit is not 0?
A: It means that total requested guest physical memory is more than the machine memory available. This is fine, because ballooning and page sharing allows memory overcommit.

This metric does not necessarily mean that you will have performance issues. Use "SWAP" and "MEMCTL" to find whether you are experiencing memory problems.

Q: What's the meaning of overcommit?
A: See above description for details. Roughly speaking, it reflects the ratio of requested memory and the available memory.

"PMEM" (MB)

The machine memory statistics for the host.

"total": the total amount of machine memory in the server. It is the machine memory reported by BIOS.

"cos" : the amount of machine memory allocated to the ESX Service Console.

"vmk" : the amount of machine memory being used by the ESX VMKernel. "vmk" includes kernel code section, kernel data and heap, and other VMKernel management memory.

"other": the amount of machine memory being used by everything other than the ESX Service Console and ESX VMKernel. "other" contains not only the memory used by VM but also the user worlds that run directly on VMKernel.

"free" : the amount of machine memory that is free.

Q: Why is total not the same as RAM size plugged in my memory slots?
A: This is because some memory range is not available for use. It is fine, if the difference is small. If the difference is big, there might be some hardware issue. Check your BIOS.

Q: Why can't I find the cos part?
A: COS is only available in classic ESX. You are using ESXi.

Q: How do I break down the total memory?
A: total = cos + vmk + other + free

Q: Which one contains the memory used by VMs?
A: "other" contains the machine memory that backs guest physical memory of VMs. Note that "other" also includes the overhead memory.

Q: How do I know my "free" memory is low? Is it a problem if it is low?
A: You could use the "state" field, which will be explained next, to see whether the free memory is low. Basically, it is fine if you do not experience memory swapping or ballooning. Check "SWAP" and "MEMCTL" to find whether you are experiencing memory problems.

"VMKMEM" (MB)

The machine memory statistics for VMKernel.

"managed": the total amount of machine memory managed by VMKernel. VMKernel "managed" memory can be dynamically allocated for VM, VMKernel, and User Worlds.

"minfree": the minimum amount of machine memory that VMKernel would like to keep free. This is because VMKernel needs to keep some amount of free memory for critical uses.

"rsvd" : the amount of machine memory that is currently reserved. "rsvd" is the sum of three parts: (a) the reservation setting of the groups; (b) the overhead reservation of the groups; (c) "minfree".

"ursvd" : the amount of machine memory that is currently unreserved. It is the memory available for reservation.

Please note that the VM admission control is done at resource pool level. So, this statistics is not used directly by admission control. "ursvd" can be used
as a system level indicator.

"state" : the free memory state. Possible values are high, soft, hard and low. The memory "state" is "high", if the free memory is greater than or equal to 6% of "total" - "cos". If is "soft" at 4%, "hard" at 2%, and "low" at 1%. So, high implies that the machine memory is not under any pressure and low implies that the machine memory is under pressure.

While the host's memory state is not used to determine whether memory should be reclaimed from VMs (that decision is made at the resource pool level), it can affect what mechanisms are used to reclaim memory if necessary. In the high and soft states, ballooning is favored over swapping. In the hard and low states, swapping is favored over ballooning.

Please note that "minfree" is part of "free" memory; while "rsvd" and "ursvd" memory may or may not be part of "free" memory. "reservation" is different from memory allocation.

Q: Why is "managed" memory less than the sum of "vmk", "other" and "free" in the PMEM line? Is it normal?
+A: It is normal, just the way we do accounting. A more precise definition for "managed" is the free memory after VMKernel initialization. So, this amount of memory can be dynamically allocated for use of VMs, VMKernel, and user worlds. "managed" = "some part of vmk" + "other" + "free".+

+So, "managed" < "vmk" + "other" + "free". Or, in an equivalent form, "managed" < "total" - "cos".+

Q: How do I break down the managed memory in terms of reservation?
A: "managed" = "rsvd" + "ursvd" + "vmkernel usage"

VMKernel machine memory manager needs to use some part of memory, which should not be subject to reservation, so, it is not in "rsvd", nor in "ursvd". In the above equation, we put this part under "vmkernel usage". Unfortunately, it is not shown directly in esxtop.

Note that the vmkernel usage in managed memory is part of "vmk".

Q: What does it mean if "ursvd" is low?
A: VMKernel admission control prohibits a VM PowerOn operation, if it cannot meet the memory reservation of that VM. The memory reservation includes the reservation setting, a.k.a. "min", and the monitor overhead memory reservation. Note that even if "min" is not set, VMKernel still needs to reserve some amount
of memory for monitor uses.

So, it is possible that even though you have enough free memory, a new VM cannot power on due to the violation of memory reservation.

Q: Why do I fail admission control even though "ursvd" is high?
A: The VM admission control is done at resource pool level. Please check the "min" setting of all its parent resource pools.

Q: Why is "managed" greater than the sum of "rsvd" and "ursvd"? Is it normal?
A: It is normal. See above question. VMKernel may use some of the managed memory. It is not accounted in "rsvd" and "ursvd".

Q: What is the meaning of "state"?
A: See the description of "state" above.

Q: How do I know my ESX box is under memory pressure?
A: It is usually safe to say the ESX box is under memory pressure, if "state" is "hard" or "low". But, you need also check "SWAP" and "MEMCTL" to find whether you are experiencing memory problems. Basically, if there is not enough free memory and ESX are experiencing swapping or ballooning, ESX box is under memory pressure.

Note that ballooning does not have as big performance hit as swapping does. Ballooning may cause guest swapping. ESX swapping means host swapping.

Also note that A VM may be swapping or ballooning, even though there is enough free memory. This is due to the reservation setting.

"COSMEM" (MB)

The memory statistics reported by the ESX Service Console.

"free" : the amount of idle machine memory.

"swap_t": the total swap configured.

"swap_f": the amount of swap free.

"r/s" : the rate at which memory is swapped in from disk.

"w/s" : the rate at which memory is swapped out to disk.

Note that these stats essentially come from the COS proc nodes.

Q: What does it mean if I see a high r/s or w/s?
A: Your console OS is swapping. It is highly likely that your COS free memory is low. You may either configure more memory for COS and restart your ESX box, or stop some programs running inside your COS.

Q: Why can't I see this COSMEM line?
A: You are using ESXi not classic ESX.

"NUMA" (MB)

The ESX NUMA statistics. For each NUMA node there are two statistics: (1) the "total" amount of machine memory managed by ESX; (2) the amount of machine memory currently "free".

Note that ESX NUMA scheduler optimizes the uses of NUMA feature to improve guest performance. Please refer to "Resource Management Guide" for details.

Q: Why can't I see this NUMA line?
A: You are not using a NUMA machine, or your BIOS disables it.

Q: Why is the sum of NUMA memory not equal to "total" in the PMEM line?
A: The PMEM "total" is the memory reported by BIOS, while the NUMA "total" is the memory managed by VMKernel machine memory manager. There are two major parts of memory seen by BIOS but not given to machine memory manager: (1) COS uses, and (2) VMKernel uses during early initialization.

So, Sum("NUMA total") < "PMEM total" - "cos".

Note that the free memory on all the nodes can be added up as the "free" memory in the PMEM line.

"PSHARE" (MB)

The ESX page-sharing statistics.

"shared": the amount of guest physical memory that is being shared.

"common": the amount of machine memory that is common across World(s).

"saving": the amount of machine memory that is saved due to page-sharing.

The monitor maps guest physical memory to machine memory. VMKernel selects to map guest physical pages with the same content to the same machine page. In other words, those guest physical pages are sharing the same machine page. This kind of sharing can happen within the same VM or among the VMs.

Since each VM's "shared" memory measures guest physical memory, the host's "shared" memory may be larger than the total amount of machine memory if memory is overcommitted. "saving" illustrates the effectiveness of page sharing for saving machine memory.

"shared" = "common" + "saving".

Note that esxtop only shows the pshare stats for VMs, excluding the pshare stats for user worlds.

"SWAP" (MB)

The ESX swap usage statistics.

"curr" : the current swap usage. This is the total swapped machine memory of all the groups. So, it includes VMs and user worlds.

"target": the swap usage expected to be. This is the total swap target of all the groups. So, it includes VMs and user worlds.

"r/s" : the rate at which machine memory is swapped in from disk.

"w/s" : the rate at which machine memory is swapped out to disk.

Note that swap here is host swap, not guest swap inside the VM.

Q: What does it mean if "curr" is not the same as "target"?
A: It means ESX will swap memory to meet the swap target. Note that the actual swapping is done at the group level. So, you should check "SWCUR" and "SWTGT" for each group. We will discuss this in the next section.

Q: Is it bad if "r/s" is high?
A: Yes, it is very bad. This usually means that you have memory resource contention. Because swapin is synchronous, it will hurt guest performance a lot.

Do two things: (1) Check your "free" memory or "state" as mentioned above. If free memory is low, you need to move VMs to other hosts or add more memory to the host. (2) If free memory is not low, check your resource setting of your VMs or user worlds. You may have set a low "limit", which causes swapping.

Q: Is it bad if "w/s" is high?
A: Yes, it is also very bad. This usually means that you have memory resource contention. Do the similar actions as mentioned above.

"MEMCTL" (MB)

The memory balloon statistics.

"curr" : the total amount of physical memory reclaimed by balloon driver. This is the total ballooned memory by the VMs.

"target": total amount of ballooned memory expected to be. This is the total ballooned targets of the VMs.

"max" : the maximum amount of physical memory reclaimable.

Note that ballooning may or may not lead to guest swapping, which is decided by the guest OS.

Q: What does it mean if "curr" is not the same as "target"?
A: It means ESX will balloon memory to meet the balloon target. Note that the actual ballooning is done for the VM group. So, you should check "MCTLSZ" and "MCTLTGT" for each group. We will discuss this in the next section.

Q: How do I know the host is ballooning memory?
A: If the "curr" is changing, you can know it is ballooning. Since ballooning is done at VM level, a better way is to monitor "MCTLSZ" for each group. We will discuss this in the next section.

Q: Is it bad if we have lots of ballooning activities?
A: Usually it is fine. Ballooning tends to take unused memory from one VM and make them available for others. The possible side effects are (a) reducing the memory cache used by guest OS, (b) guest swapping. In either cases, it may hurt guest performance. Please note that (a) and (b) may or may not happen, depending on your workload inside VM.

On the other hand, under memory contention, ballooning is much better than swapping in terms of performance.

Section 3.3 Group Statistics

Esxtop shows the groups that use memory managed by VMKernel memory scheduler. These groups can be used for VMs or purely for user worlds running directly on VMKernel. You may see many pure user world groups on ESXi, not on classic ESX.

Tip: use 'V' command to show only the VM groups.

"MEMSZ" (MB)

For a VM, it is the amount of configured guest physical memory.

For a user world, it includes not only the virtual memory that is backed by the machine memory, but also the reserved backing store size.

Q: How do I break down "MEMSZ" of a VM?
A: A VM's guest physical memory could be mapped to machine memory, reclaimed by balloon driver, or swapped to disk, or never touched. The guest physical memory can be "never touched", because (1) the VM has never used it since power on; or, (2) it was reclaimed by balloon driver before, but has not been used since the balloon driver releases it last time. This part of memory is not measured directly by VMKernel.

"MEMSZ" = "GRANT" + "MCTLSZ" + "SWCUR" + "never touched"

Please refer to "GRANT", "MCTLSZ", "SWCUR".

"GRANT" (MB)

For a VM, it is the amount of guest physical memory granted to the group, i.e., mapped to machine memory. The overhead memory, "OVHD" is not included in GRANT. The shared memory, "SHRD", is part of "GRANT". This statistics is added to esxtop in ESX 4.0.

The consumed machine memory for the VM, not including the overhead memory, can be estimated as "GRANT" - "SHRDSVD". Please refer to "SHRDSVD".

For a user world, it is the amount of virtual memory that is backed by machine memory.

Q: Why is "GRANT" less than "MEMSZ"?
A: Some guest physical memory has never been used, or is reclaimed by balloon driver, or is swapped out to the VM swap file. Note that this kind of swap is host swap, not the guest swap by the guest OS.

"MEMSZ" = "GRANT" + "MCTLSZ" + "SWCUR" + "never touched"

Q: How do I know how much machine memory is consumed by this VM?
A: GRANT accounts the guest physical memory, it may not be the same as the mapped machine memory, due to page sharing.

The consumed machine memory can be estimated as "GRANT" - "SHRDSVD". Please note that this is an estimate. Please refer to "SHRDSVD".

Note that overhead memory, "OVHD", is not part of the above consumed machine memory.

"SZTGT" (MB)

The amount of machine memory to be allocated. (TGT is short for "target".) Note that "SZTGT" includes the overhead memory for a VM.

This is an internal counter, which is computed by ESX memory scheduler. Usually, there is no need to worry about this. Roughly speaking, "SZTGT" of all the VMs is computed based on the resource usage, available memory, and the "limit/reservation/shares" settings. This computed "SZTGT" is compared against the current memory consumption plus overhead memory for a VM to determine the swap and balloon target, so that VMKernel may balloon or swap appropriate amount
of memory to meet its memory demand. Please refer to "Resource Management Guide" for details.

Q: How come my "SZTGT" is larger than "MEMSZ"?
A: "SZTGT" includes the overhead memory, while "MEMSZ" does not. So, it is possible for "SZTGT" be larger than "MEMSZ".

Q: How do I use "SZTGT"?
A: This is an internal counter. You don't need to use it.

This counter is used to determine future swapping and ballooning activities. Check "SWTGT" and "MCTLTGT".

"TCHD" (MB)

The amount of guest physical memory recently used by the VM, which is estimated by VMKernel statical sampling.

VMKernel estimates active memory usage for a VM by sampling a random subset of the VM's memory resident in machine memory to detect the number of memory reads and writes. VMKernel then scales this number by the size of VM's configured memory and averages it with previous samples. Over time, this average will approximate the amount of active memory for the VM.

Note that ballooned memory is considered inactive, so, it is excluded from "TCHD".

Because sampling and averaging takes time, "TCHD" won't be exact, but becomes more accurate over time.

VMKernel memory scheduler charges the VM by the sum of (1) the "TCHD" memory and (2) idle memory tax. This charged memory is one of the factors that memory scheduler uses for computing the "SZTGT".

Q: What is the difference between "TCHD" and working set estimate by guest OS?
A: "TCHD" is the working set estimated by VMKernel. This number may be different from guest working set estimate. Sometimes the difference may be big, because (1) guest OS uses a different working set estimate algorithm, (2) guest OS has a different view of active guest physical memory, due to ballooning and host swapping,

Q: How is "TCHD" used?
A: "TCHD" is a working set estimate, which indicates how actively the VM is using its memory. See above for the internal use of this counter.

"%ACTV"

Percentage of active guest physical memory, current value.

"TCHD" is actually computed based on a few parameters, coming from statistical sampling. The exact equation is out of scope of this document. Esxtop shows some of those parameters, %ACTV, %ACTVS, %ACTVF, %ACTVN. Here, this document provides simple descriptions without further discussion.

%ACTV reflects the current sample.
%ACTVS is an EWMA of %ACTV for long term estimate.
%ACTVF is an EWMA of %ACTV for short term estimate.
%ACTVN is a predict of what %ACTVF will be at next sample.

Since they are very internal to VMKernel memory scheduler, we do not discuss their usage here.

"%ACTVS"

Percentage of active guest physical memory, slow moving average. See above.

"%ACTVF"

Percentage of active guest physical memory, fast moving average. See above.

"%ACTVN"

Percentage of active guest physical memory in the near future. This is an estimated value. See above.

"MCTL?"

Memory balloon driver is installed or not.

If not, install VMware tools which contains the balloon driver.

"MCTLSZ" (MB)

The amount of guest physical memory reclaimed by balloon driver.

This can be called "balloon size". A large "MCTLSZ" means lots of this VM's guest physical memory is "stolen" to decrease host memory pressure. This usually is not a problem, because balloon driver tends to smartly steal guest physical memory that cause little performance problems.

Q: How do I know the VM is ballooning?
A: If "MCTLSZ" is changing, balloon driver is actively reclaiming or releasing memory. I.e., the VM is ballooning. Please note that the ballooning rate for a short term can be estimated by the change of "MCTLSZ", assuming it is either increasing or decreasing. But, for a long term, we cannot do it this way, because that monotonically increase/decrease assumption may not hold.

Q: Does ballooning hurt VM performance?
A: If guest working set is smaller than guest physical memory after ballooning, guest applications won't observe any performance degradation. Otherwise, it may cause guest swapping and hurt guest application performance.

Please check what causes ballooning and take appropriate actions to reduce memory pressure. There are two possible reasons: (1) The host does not have enough machine memory for use. (2) Memory used by the VM reaches the "limit" setting of itself or "limit" of the resource pools that contain this VM. In either case, ballooning is necessary and preferred over swapping.

"MCTLTGT" (MB)

The amount of guest physical memory to be kept in balloon driver. (TGT is short for "target".)

This is an internal counter, which is computed by ESX memory scheduler. Usually, there is no need to worry about this.

Roughly speaking, "MCTLTGT" is computed based on "SZTGT" and current memory usage, so that the VM can balloon appropriate amount of memory. If "MCTLTGT" is greater than "MCTLSZ", VMKernel initiates inflating the balloon immediately, causing more VM memory to be reclaimed. If "MCTLTGT" is less than "MCTLSZ", VMKernel will deflate the balloon when the guest is requesting memory, allowing the VM to map/consume additional memory if it needs it. Please refer to "Resource Management Guide" for details.

Q: Why is it possible for "MCTLTGT" to be less than "MCTLSZ" for a long time?
A: If "MCTLTGT" is less than "MCTLSZ", VMKernel allows the balloon to deflate. But, balloon deflation happens lazily until the VM requests new memory. So, it is possible for "MCTLTGT" to be less than "MCTLSZ" for a long time, when the VM is not requesting new memory.

"MCTLMAX" (MB)

The maximum amount of guest physical memory reclaimable by balloon driver.

This value can be set via vmx option "sched.mem.maxmemctl". If not set, it is determined by the guest operating system type. "MCTLTGT" will never be larger than "MCTLMAX".

If the VM suffers from ballooning, "sched.mem.maxmemctl" can be set to a smaller value to reduce this possibility. Remember that doing so may result in host swapping during resource contention.

"SWCUR" (MB)

Current swap usage.

For a VM, it is the current amount of guest physical memory swapped out to the backing store. Note that it is the VMKernel swapping not the guest OS swapping.

It is the sum of swap slots used in the vswp file or system swap, and migration swap. Migration swap is used for a VMotioned VM to hold swapped out memory on the destination host, in case the destination host is under memory pressure.

Q: What does it mean if "SWCUR" of my VM is high?
A: It means the VM's guest physical memory is not resident in machine memory, but on disk. If those memory will not be used in the near future, it is not an issue. Otherwise, those memory will be swapped in for guest's use. In that case, you will see some swap-in activities via "SWR/s", which may hurt the VM's performance.

"SWTGT" (MB)

The expected swap usage. (TGT is short for "target".)

This is an internal counter, which is computed by ESX memory scheduler. Usually, there is no need to worry about this.

Roughly speaking, "SWTGT" is computed based on "SZTGT" and current memory usage, so that the VM can swap appropriate amount of memory. Again, note that it is the VMKernel swapping not the guest swapping. If "SWTGT" is greater than "SWCUR", VMKernel starts swapping immediately, causing more VM memory to be swapped out. If "SWTGT" is less than "SWCUR", VMKernel will stop swapping. Please refer to "Resource Management Guide" for details.

Q: Why is it possible for "SWTGT" to be less than "SWCUR" for a long time?
A: Since swapped memory stays swapped until the VM accesses it, it is possible for "SWTGT" be less than "SWCUR" for a long time.

"SWR/s" (MB)

Rate at which memory is being swapped in from disk. Note that this stats refers to the VMKernel swapping not the guest swapping.

When a VM is requesting machine memory to back its guest physical memory that was swapped out to disk, VMKernel reads in the page. Note that the swap-in operation is synchronous.

Q: What does it mean if SWR/s is high?
A: It is very bad for VM's performance. Because swap-in is synchronous, the VM needs to wait until the requested pages are read into machine memory. This happens when VMKernel swapped out the VM's memory before and the VM needs them now. Please refer to "SWW/s".

"SWW/s" (MB)

Rate at which memory is being swapped out to disk. Note that this stats refers to the VMKernel swapping not the guest swapping.

As discussed in "SWTGT", if "SWTGT" is greater than "SWCUR", VMKernel will swap out memory to disk. It happens usually in two situations. (1) The host does not have enough machine memory for use. (2) Memory used by the VM reaches the "limit" setting of itself or "limit" of the resource pools that contain this VM.

Q: What does it mean if SWW/s is high?
A: It is very bad for VM performance. Please check the above two reasons and fix your problem accordingly.

If this VM is swapping out memory due to resource contention, it usually means VMKernel does not have enough machine memory to meet memory demands from all the VMs. So, it will swap out mapped guest physical memory pages to make room for the recent requests.

"SHRD" (MB)

Amount of guest physical memory that are shared.

VMKernel page sharing module scans and finds guest physical pages with the same content and backs them with the same machine page. "SHRD" accounts the total guest physical pages that are shared by the page sharing module.

"ZERO" (MB)

Amount of guest physical zero memory that are shared. Thisis an internal counter.

A zero page is simply the memory page that is all zeros. If a zero guest physical page is detected by VMKernel page sharing module, this page will be backed by the same machine page on each NUMA node. Note that "ZERO" is included in "SHRD".

"SHRDSVD" (MB)

Estimated amount of machine memory that are saved due to page sharing.

Because a machine page is shared by multiple guest physical pages, we only charge "1/ref" page as the consumed machine memory for each of the guest physical pages, where "ref" is the number of references. So, the saved machine memory will be "1 - 1/ref" page."SHRDSVD" estimates the total saved machine memory for the VM.

The consumed machine memory by the VM can be estimated as "GRANT" - "SHRDSVD".

"COWH" (MB)

Amount of guest physical hint pages for page sharing. This is an internal counter.

"OVHDUW" (MB)

Amount of overhead memory reserved for the vmx user world of a VM group. This is an internal counter.

"OVHDUW" is part of "OVHDMAX".

"OVHD" (MB)

Amount of overhead memory currently consumed by a VM.

"OVHD" includes the overhead memory consumed by the monitor, the VMkernel and the vmx user world.

"OVHDMAX" (MB)

Amount of reserved overhead memory for the entire VM.

"OVHDMAX" is the overhead memory a VM wants to consume in the future. This amount of reserved overhead memory includes the overhead memory reserved by the monitor, the VMkernel, and the vmx user world. Note that the actual overhead memory consumption is less than "OVHDMAX". "OVHD" < "OVHDMAX".

"OVHDMAX" can be used as a conservative estimate of the total overhead memory.

Section 4 Disk

Section 4.1 Adapter, Device, VM screens

The ESX storage stack adds a few layers of code between a virtual machine and bare hardware. All virtual disks in virtual machines are seen as virtual SCSI disks. The ESX storage stack allows these virtual disks to be located on any of the multiple storage options available.

For performance analysis purposes, an IO request from an application in a virtual machine traverses through multiple levels of queues, each associated with a resource, in the guest OS, the VMkernel and the physical storage. (Note that physical storage could be an FC- or IP- SAN or disk array.) Each queue has an associated latency, dictated by its size and whether the IO load is low or high, which affects the throughput and latency seen by applications inside VMs.

Esxtop shows the storage statistics in three different screens: adapter screen, device screen, and vm screen. Interactive command 'd' can be used to switch to the adapter screen, 'u' for the device screen, and 'v' for the vm screen.

The main difference in the data seen in these three screens is the level at which it is aggregated, even though these screens have similar counters. By default, data is rolled up to the highest level possible for each screen. (1) On the adapter screen, by default, the statistics are aggregated per storage adapter but they can also be expanded to display data per storage channel, target, path or world using a LUN. See interacitive commands, 'e', 'E', 'P', 'a', 't', 'l', for the expand operations. (2) On the device screen, by default, statistics are aggregated per storage device. Statistics can also be viewed per path, world, or partition. See interactive commands, 'e', 'p', 't', for the expand operations. (3) On the VM screen, statistics are aggregated on a per-group basis by default. One VM has one corresponding group, so they are equivalent to per-VM statistics. You can use interactive command 'V' to show only statistics related to VMs. Statistics can also be expanded so that a row is displayed for each world or a per-world-per-device basis. See interactive commands, 'e' and 'l'.

Please refer to esxtop man page for the details of the interactive commands.

Section 4.2 Disk Statistics

Due to the similarities in the counters of the three disk screens, this section discusses the counters without distinguishing the screens. Similar to other esxtop screens, the storage counters are also organized in different sets, each of which contains related counters. The counters can be selected as a set by selecting the appropriate field option in esxtop. If esxtop is used in batch mode, make sure that the esxtop configuration file includes all counters of interest.

Each group of counters in the following subsections corresponds to a particular field option.

Section 4.2.1 I/O Throughput Statistics

CMDS/s

Number of commands issued per second.

READS/s

Number of read commands issued per second.

WRITES/s

Number of write commands issued per second.

MBREAD/s

Megabytes read per second.

MBWRTN/s

Megabytes written per second.

Section 4.2.2 Latency Statistics

This group of counters report latency values measured at three different points in the ESX storage stack. In the context of the figure below, the latency counters in esxtop report the Guest, ESX Kernel and Device latencies. These are under the labels GAVG, KAVG and DAVG, respectively. Note that GAVG is the sum of DAVG and KAVG counters.

Note that esxtop shows the latency statistics for different objects, such as adapters, devices, paths, and worlds. They may not perfectly match with each other, since their latencies are measured at the different layers of the ESX storage stack. To do the correlation, you need to be very familiar with the storage layers in ESX Kernel, which is out of our scope.

Latency values are reported for all IOs, read IOs and all write IOs. All values are averages over the measurement interval.

All IOs: KAVG/cmd, DAVG/cmd, GAVG/cmd, QAVG/cmd
Read IOs: KAVG/rd, DAVG/rd, GAVG/rd, QAVG/rd
Write IOs: KAVG/wr, DAVG/wr, GAVG/wr, QAVG/wr

GAVG

This is the round-trip latency that the guest sees for all IO requests sent to the virtual storage device.

GAVG should be close to the R metric in the figure.

Q: What is the relationship between GAVG, KAVG and DAVG?
A: GAVG = KAVG + DAVG

KAVG

These counters track the latencies due to the ESX Kernel's command.

The KAVG value should be very small in comparison to the DAVG value and should be close to zero. When there is a lot of queuing in ESX, KAVG can be as high, or even higher than DAVG. If this happens, please check the queue statistics, which will be discussed next.

DAVG

This is the latency seen at the device driver level. It includes the roundtrip time between the HBA and the storage.

DAVG is a good indicator of performance of the backend storage. If IO latencies are suspected to be causing performance problems, DAVG should be examined. Compare IO latencies with corresponding data from the storage array. If they are close, check the array for misconfiguration or faults. If not, compare DAVG with corresponding data from points in between the array and the ESX Server, e.g., FC switches. If this intermediate data also matches DAVG values, it is likely that the storage is under-configured for the application. Adding disk spindles or changing the RAID level may help in such cases.

QAVG

The average queue latency. QAVG is part of KAVG.

Response time is the sum of the time spent in queues in the storage stack and the service time spent by each resource in servicing the request. The largest component of the service time is the time spent in retrieving data from physical storage. If QAVG is high, another line of investigation is to examine the queue depths at each level in the storage stack.

Section 4.2.3 Queue Statistics

AQLEN

The storage adapter queue depth. This is the maximum number of ESX Server VMKernel active commands that the adapter driver is configured to support.

LQLEN

The LUN queue depth. This is the maximum number of ESX Server VMKernel active commands that the LUN is allowed to have. (Note that, in this document, the terminologies of LUN and Storage device can be used interchangeably.)

WQLEN

The World queue depth. This is the maximum number of ESX Server VMKernel active commands that the World is allowed to have. Note that this is a per LUN maximum for the World.

ACTV

The number of commands in the ESX Server VMKernel that are currently active. This statistic is only applicable to worlds and LUNs.

Please refer to %USD.

QUED

The number of commands in the VMKernel that are currently queued. This statistic is only applicable to worlds and LUNs.

Queued commands are commands waiting for an open slot in the queue. A large number of queued commands may be an indication that the storage system is overloaded. A sustained high value for the QUED counter signals a storage bottleneck which may be alleviated by increasing the queue depth. Check that LOAD < 1 after increasing the queue depth. This should also be accompanied by improved performance in terms of increased cmd/s.

Note that there are queues in different storage layers. You might want to check the QUED stats for devices, and worlds.

%USD

The percentage of queue depth used by ESX Server VMKernel active commands. This statistic is only applicable to worlds and LUNs.

%USD = ACTV / QLEN * 100%

For world stats, WQLEN is used as the denominator. For LUN (aka device) stats, LQLEN is used as the denominator.

%USD is a measure of how many of the available command queue "slots" are in use. Sustained high values indicate the potential for queueing; you may need to adjust the queue depths for system’s HBAs if QUED is also found to be consistently > 1 at the same time. Queue sizes can be adjusted in a few places in the IO path and can be used to alleviate performance problems related to latency. For detailed information on this topic please refer to the VMware whitepaper entitled "Scalable Storage Performance".

LOAD

The ratio of the sum of VMKernel active commands and VMKernel queued commands to the queue depth. This statistic is only applicable to worlds and LUNs.

The sum of the active and queued commands gives the total number of outstanding commands issued by that virtual machine. The LOAD counter values is the ratio of this value with respect to the queue depth. If LOAD > 1, check the value of the QUED counter.

Section 4.2.4 Error Statistics

ABRTS/s

The number of commands aborted per second.

It can indicate that the storage system is unable to meet the demands of the guest operating system. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time, e.g. 60 seconds on some windows OS’s. Also, resets issued by a guest OS on its virtual SCSI adapter will be translated to aborts of all the commands outstanding on that virtual SCSI adapter.

RESETS/s

The number of commands reset per second.

Section 4.2.5 PAE Statistics

PAECMD/s

The number of PAE commands per second.

It may point to hardware misconfiguration. When the guest allocates a buffer, the vmkernel assigns some machine memory, which might come from a “highmem” region. If you have a driver that is not PAE-aware, then this counter is updated if accesses to this memory region result in copies by the vmkernel into a lower memory location before issuing the request to the adapter. This might happen if you do not populate the DIMMs with low memory first, then you may artificially cause “highmem” memory accesses.

PAECP/s

The number of PAE copies per second.

Section 4.2.6 Split Statistics

SPLTCMD/s

The number of split commands per second.

Commands can be split when they reach the vmkernel. This might impact perceived latency to the guest. The guest may be issuing commands of large block sizes which have to be broken down by the vmkernel. For ESX3.0.x, guest requests greater than 128KB are split into 128KB chunks. Since few applications do larger than 128KB ops, this is unlikely to be an issue. Splitting can also occur when IOs fall across partition boundaries but these are easily differentiated from the splitting as a result of the IO size.

SPLTCP/s

The number of split copies per second.

Section 4.3 Batch Mode Output

Esxtop batch mode output can be loaded in perfmon directly. It uses a csv (comma separated values) format. The instance type can be ideitified via its name. Because there are quite a number of instances related to disk statistics, let's list a few examples below. You may easily match the format in your own environment.

LUN (aka device): "\\<host>\Physical Disk(DEV-vmhba0:0:0)\<counter>"
Partition: "\\<host>\Physical Disk(PN-vmhba0:0:0-1)\<counter>"
Path: "\\<host>\Physical Disk(PH-vmhba0:C0:T0:L0)\<counter>"
Per-World-Per-Device: "\\<host>\Physical Disk(WD-vmhba0:0:0-1024)\<counter>"
Adapter: "\\<host>\Physical Disk(vmhba0)\<counter>"

Section 5 Network

Section 5.1 Port

We arrange the network stats per port of a virtual switch. "PORT-ID" identifies the port and "DNAME" shows the virtual switch name. A port can be linked to a physical NIC as an uplink, or can be connected by a virtual NIC. "UPLINK" indicates whether the port is an uplink.

If the port is an uplink, i.e., "UPLINK" is 'Y', "USED-BY" shows the physical NIC name.

If the port is connected by a virtual NIC, i.e., "UPLINK" is 'N', "USED-BY" shows the port client name. (a) If the port is used by a virtual machine, the client name contains a world id and the VM name. The world id identifies the leader world of the VM group. Note that "vswif" is used by COS (on classic ESX). (b) If the port is used by VMKernel system, there is no world id. The client name can be used to identify the use of the port. To give two examples.

"vmk" is a port used by vmkernel. Users can create vmk NICs for their uses, such as VMotion. On ESXi, there will be at least one vmk NIC to communicate with outside of the host.
"Management" is a management port for a portset. This is internal. Usually no need to worry about it.

For each non-uplink port, the NIC teaming policy determines which physical NIC is in charge of the port. "TEAM-PNIC" shows the physical NIC name, if valid. Please refer to NIC teaming documentation for details.

Section 5.2 Port Statistics

"SPEED" (Mbps)

The link speed in Megabits per second. This information is only valid for a physical NIC.

"FDUPLX"

'Y' implies the corresponding link is operating at full duplex. 'N' implies it is not. This information is only valid for a physical NIC.

"UP"

'Y' implies the corresponding link is up. 'N' implies it is not. This information is only valid for a physical NIC.

"PKTTX/s"

The number of packets transmitted per second.

"PKTRX/s"

The number of packets received per second.

"MbTX/s" (Mbps)

The MegaBits transmitted per second.

"MbRX/s" (Mbps)

The MegaBits received per second.

Q: Why does MbRX/s not match PKTRX/s for different workloads?
A: This is because the packet size may not be the same. The average packet size can be computed as follows: average_packet_size = MbRX/s / PKTRX/s . A large packet size may improve CPU efficiency of processing the packet. However, it may potentially increase latency.

"%DRPTX"

The percentage of transmit packets dropped.

"%DRPTX" = "dropped Tx packets" / ("success Tx packets" + "dropped Tx packets")

Q: What does it mean if %DRPTX is high?
A: This usually means the network transmit performance is bad. Please check whether the phsycial NICs are fully utilizing their capacity. You probably need physical NICs with better performance. Or, you may add more physical NICs and use a good NIC teaming load balancing policy.

"%DRPRX"

The percentage of receive packets dropped.

"%DRPRX" = "dropped Rx packets" / ("success Rx packets" + "dropped Rx packets")

Q: What does it mean if %DRPRX is high?
A: This usally means the network recieve performance is bad. Try to give more CPU resource to the impacted VM, or increase the ring buffer size.

"ACTN/s"

Number of actions per second. The actions here are VMkernel actions. It is an internal counter. We won't discuss it further here.

Section 6. Interrupt

Interrupt screens are under development for our next release.

Section 7. Batch Mode

Esxtop batch mode output uses a csv (comma separated values) format. The first line contains the names of the performance counters and their instances. Each of the following lines contains the performance data for those counter instances in one snapshot.

One way to read the batch mode output file is to load it in Windows perfmon. (1) Run perfmon; (2) Type "Ctrl + L" to view log data; (3) Add the file to the "Log files" and click OK; (4) Choose the counters to show the performance data. Each batch mode counter has a category name (listed as a performance object in perfmon) and a counter name (listed in the counter list in perfmon).

The counter names in esxtop batch mode are different from the ones in interactive mode listed in the sections above. The tables below describe their relationships. The first column is the interactive mode counter name; the second column is the batch mode counter category; the last column is the batch mode counter name.

Table 7-1 CPU Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
CPU load average	Physical Cpu Load	Cpu Load (1 Minute Avg)
		Cpu Load (5 Minute Avg)
		Cpu Load (15 Minute Avg)
PCPU(%)	Physical Cpu	% Processor Time
LCPU(%)	Logical Cpu	% Processor Time
CCPU(%) us	Console Physical Cpu	% User Time
CCPU(%) sy	Console Physical Cpu	% System Time
CCPU(%) id	Console Physical Cpu	% Idle Time
CCPU(%) wa	Console Physical Cpu	% I/O Wait Time
CCPU(%) cs/sec	Console Physical Cpu	% Context Switches/sec

%USED	Group Cpu (or Vcpu)	% Used
%SYS	Group Cpu (or Vcpu)	% System
%OVRLP	Group Cpu (or Vcpu)	% Overlap
%RUN	Group Cpu (or Vcpu)	% Run
%RDY	Group Cpu (or Vcpu)	% Ready
%MLMTD	Group Cpu (or Vcpu)	% Max Limited
%CSTP	Group Cpu (or Vcpu)	% CoStop
%WAIT	Group Cpu (or Vcpu)	% Wait
%IDLE	Group Cpu (or Vcpu)	% Idle
%SWPWT	Group Cpu (or Vcpu)	% Swap Wait

Table 7-2 Memory Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
MEM overcommit avg	Memory	Memory Overcommit (1 Minute Avg)
		Memory Overcommit (5 Minute Avg)
		Memory Overcommit (15 Minute Avg)
PMEM total	Memory	Machine MBytes
PMEM cos	Memory	Console MBytes
PMEM vmk	Memory	Kernel MBytes
PMEM other	Memory	NonKernel MBytes
PMEM free	Memory	Free MBytes
VMKMEM managed	Memory	Kernel Managed MBytes
VMKMEM minfree	Memory	Kernel MinFree MBytes
VMKMEM rsvd	Memory	Kernel Reserved MBytes
VMKMEM ursvd	Memory	Kernel Unreserved MBytes
VMKMEM state	Memory	Kernel State (0: high, 1: soft, 2:hard, 3: low)
COSMEM free	Console Memory	Free MBytes
COSMEM swap_t	Console Memory	Swap Total MBytes
COSMEM swap_f	Console Memory	Swap Free MBytes
COSMEM r/s	Console Memory	Swap MBytes Read/sec
COSMEM w/s	Console Memory	Swap MBytes Write/sec
NUMA	Numa Node	Total MBytes
		Free MBytes
PSHARE shared	Memory	PShare Shared MBytes
PSHARE common	Memory	PShare Common MBytes
PSHARE saving	Memory	PShare Savings MBytes
SWAP curr	Memory	Swap Used MBytes
SWAP target	Memory	Swap Target MBytes
SWAP r/s	Memory	Swap MBytes Read/sec
SWAP w/s	Memory	Swap MBytes Write/sec
MEMCTL curr	Memory	Memctl Current MBytes
MEMCTL target	Memory	Memctl Target MBytes
MEMCTL max	Memory	Memctl Max MBytes

MEMSZ	Group Memory	Memory Size MBytes
GRANT	Group Memory	Memory Granted Size MBytes
SZTGT	Group Memory	Target Size MBytes
TCHD	Group Memory	Touched MBytes
%ACTV	Group Memory	% Active Estimate
%ACTVS	Group Memory	% Active Slow Estimate
%ACTVF	Group Memory	% Active Fast Estimate
%ACTVN	Group Memory	% Active Next Estimate
MCTL?	Group Memory	Memctl?
MCTLSZ	Group Memory	Memctl MBytes
MCTLTGT	Group Memory	Memctl Target MBytes
MCTLMAX	Group Memory	Memctl Max MBytes
SWCUR	Group Memory	Swapped MBytes
SWTGT	Group Memory	Swap Target MBytes
SWR/s	Group Memory	Swap Read MBytes/sec
SWW/s	Group Memory	Swap Written MBytes/sec
SHRD	Group Memory	Shared MBytes
ZERO	Group Memory	Zero MBytes
SHRDSVD	Group Memory	Shared Saved MBytes
COWH	Group Memory	Copy On Write Hint MBytes
OVHDUW	Group Memory	Overhead UW MBytes
OVHD	Group Memory	Overhead MBytes
OVHDMAX	Group Memory	Overhead Max MBytes

Table 7-3 Disk Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
CMDS/s	Physical Disk	Commands/sec
READS/s	Physical Disk	Reads/sec
WRITES/s	Physical Disk	Writes/sec
MBREAD/s	Physical Disk	MBytes Read/sec
MBWRTN/s	Physical Disk	MBytes Written/sec
KAVG/cmd	Physical Disk	Average Kernel MilliSec/Command
DAVG/cmd	Physical Disk	Average Driver MilliSec/Command
GAVG/cmd	Physical Disk	Average Guest MilliSec/Command
QAVG/cmd	Physical Disk	Average Queue MilliSec/Command
KAVG/rd	Physical Disk	Average Kernel MilliSec/Read
DAVG/rd	Physical Disk	Average Driver MilliSec/Read
GAVG/rd	Physical Disk	Average Guest MilliSec/Read
QAVG/rd	Physical Disk	Average Queue MilliSec/Read
KAVG/wr	Physical Disk	Average Kernel MilliSec/Write
DAVG/wr	Physical Disk	Average Driver MilliSec/Write
GAVG/wr	Physical Disk	Average Guest MilliSec/Write
QAVG/wr	Physical Disk	Average Queue MilliSec/Write
AQLEN	Physical Disk	Adapter Q Depth
LQLEN	Physical Disk	Lun Q Depth
DQLEN	Physical Disk	Device Q Depth
WQLEN	Physical Disk	World Q Depth
ACTV	Physical Disk	Active Commands
QUED	Physical Disk	Queued Commands
%USD	Physical Disk	% Used
LOAD	Physical Disk	Load
LOAD	Physical Disk	Load
ABRTS/s	Physical Disk	Aborts/sec
RESETS/s	Physical Disk	Resets/sec
PAECMD/s	Physical Disk	PAE Commands/sec
PAECP/s	Physical Disk	PAE Copies/sec
SPLTCMD/s	Physical Disk	Split Commands/sec
SPLTCP/s	Physical Disk	Split Copies/sec

Table 7-4 Network Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
SPEED	Network Port	Link Speed (Mb/s)
FDUPLX	Network Port	Full Duplex?
UP	Network Port	Link Up?
PKTTX/s	Network Port	Packets Transmitted/sec
PKTRX/s	Network Port	Packets Received/sec
MbTX/s	Network Port	MBits Transmitted/sec
MbRX/s	Network Port	MBits Received/sec
%DRPTX	Network Port	% Outbound Packets Dropped
%DRPRX	Network Port	% Received Packets Dropped
ACTN/s	Network Port	Actions Posted/sec

v-tomo 20081222

Tue, 23 Dec 2008 08:36:37 GMT

VMware Products and Hardware-Assisted Virtualization (VT-x/AMD-V)

jmattson@vmware.com — Thu, 11 Dec 2008 22:05:08 GMT

Introduction

Both major x86 chip vendors now provide hardware-assisted virtualization support. Intel's x86 hardware virtualization technology is known as VT-x or, more commonly, simply VT (Virtualization Technology). Previously, it was code-named "Vanderpool." AMD's x86 virtualization technology is known as AMD-V. Previously, it was known as "SVM" (Secure Virtual Machine), and before that, it was code-named "Pacifica." These competing technologies are remarkably similar under the hood, particularly in the first generation. Performance of the first generation hardware was somewhat lackluster, and for that reason, VMware products still default where possible to binary translation without hardware assistance on older HV-capable hardware.

Both vendors have made incremental improvements to their hardware-assisted virtualization technology, and now both vendors offer a major advancement in the form of virtual MMU support, or nested paging. AMD was the first to come out with nested paging in their Family 10H (Phenom) processors, and they dub their nested paging technology RVI (Rapid Virtualization Indexing). Intel introduced a comparable feature in their Core i7 processors, and they dub their nested paging technology EPT (Extended Page Tables). With nested paging support, along with improvements in VM-entry/VM-exit latencies, hardware-assisted virtualization now outperforms binary translation in most situations. However, TLB misses are much more expensive in a nested paging environment, so workloads that over-subscribe the TLB are potentially still good candidates for binary translation without hardware assistance.

As the hardware has evolved, VMware software support of hardware-assisted virtualization has also evolved. The following sections detail the hardware-virtualization support in some of our products.

Workstation 5.5 and ESX 3.0

Intel VT-x support was first introduced in Workstation 5.5 to support 64-bit guests on Intel hardware. Since Intel does not provide segment limit checks in 64-bit mode, VT-x is a requirement to provide isolation between 64-bit guests and the virtual machine monitor (or hypervisor). Though unsupported, it is also possible to run a 32-bit guest using VT-x. The configuration option for running a 32-bit guest with VT-x is:

monitor_control.vt32 = "TRUE"

There is no support for EPT or AMD-V in these products.

Workstation 6.0 and Fusion 1.0

Workstation 6.0 has the same VT-x support as Workstation 5.5. Experimental support was added for AMD-V and RVI. The configuration option for running a guest (either 32-bit or 64-bit) with AMD-V is:

monitor_control.enable_svm = "TRUE"

RVI will be used by default on RVI-capable hardware. Note that this is the only VMware product that supports (experimentally) AMD-V on Family 0FH processors (Athlon/Opteron) with AMD-V support. Apple does not currently sell AMD hardware, but this code is in Fusion 1.0 nonetheless.

ESX 3.5

ESX 3.5 only supports VT-x for running 64-bit guests. The experimental 32-bit VT-x support was eliminated. ESX3.5 also provides the first official AMD-V support, but only for chips with RVI. With this release, we dropped the "monitor_control" flags for hardware-virtualization. The configuration option for running a guest (either 32-bit or 64-bit) with AMD-V and RVI is:

monitor.virtual_mmu = "hardware"

Unlike the boolean monitor_control flags used previously, the monitor.virtual_mmu option has three settings: automatic, software and hardware. Selecting RVI implicitly selects AMD-V, since RVI cannot be used with binary translation.

Workstation 6.5, Server 2.0, ESX 4.0 and Fusion 2.0

These products add official support for VT-x and AMD-V for all guests, both 32-bit and 64-bit. EPT support is introduced for EPT-capable hardware. VT-x or AMD-V can be used in conjunction with a software MMU or with nested paging (RVI or EPT) on hardware that supports it. Note, however, that hardware-assisted virtualization is only supported on 64-bit hardware, and there is no support for AMD-V on Family 0FH processors (Athlon/Opteron).

The configuration option for selecting binary translation is:

monitor.virtual_exec = "software"

The configuration option for selecting VT-x/AMD-V is:

monitor.virtual_exec = "hardware"

The configuration option for selecting the software MMU is:

monitor.virtual_mmu = "software"

The configuration option for selecting RVI/EPT is:

monitor.virtual_mmu = "hardware"

Note that you cannot mix binary translation with RVI/EPT.

Workstation 6.5 allows you to make these selections through the UI, in the virtual hardware "processors" configuration. ESX 4.0 provides a configuration selection through the vSphere client. For the other products, you have to edit the VM configuration files by hand.

Whether you edit the configuration file by hand or use the UI, you should be careful to select an execution mode that is supported by your hardware. In particular, if you select RVI/EPT on hardware without nested paging support, the execution mode will revert to binary translation. If you select binary translation for a 64-bit guest on Intel hardware, the execution mode will dynamically switch to VT-x as soon as the guest enters long mode (typically partway through the guest boot process).

If you do not select a particular execution mode, a default mode will be chosen for you. The defaults are incredibly complex, depending on your hardware capabilities and the guest OS type. In general, the default mode is likely to be the best (highest performing) mode for your hardware and guest OS, but the heuristics are not infallible. You should feel free to benchmark your workload using the execution modes available on your hardware and choose what works best for you.

For more information, please see http://www.vmware.com/resources/techresources/10036.

remove phantom peripherals by script

Tue, 02 Dec 2008 18:07:13 GMT

Best Practices for SQL Server

Mon, 01 Dec 2008 20:45:05 GMT

Introduction

At VMworld 2008 in Las Vegas several of us in our virtual performance team met with a variety of customers to talk about Microsoft SQL Server. We already had a large base of customers running very many SQL Server DBs on our products and we wanted to collect information on the challenges posed in the process of virtualizing this critical workload. We were pleased to see that ESX Server handled SQL VMs with excellent performance. But, for many customers, the first efforts at virtualizing SQL didn't yield high-performing SQL VM. After careful investigation and many, many discussions we've started to put together the puzzle as to where SQL Server performance problems come from. This page will document these common problems, borrowing slides from our presentations on the subject.

Virtualizing SQL: The Checklist

We've talked with dozens of customers in the past months to document the issues that resulted in poor SQL performance. Happily, none of the issues were due to underlying technologies. Here is a list of issues and an explanation of the impacts. These items are roughly listed in the order of decreasing likelihood of occurrence.

Item 1: Configure Storage Correctly

Storage configuration problems are the number one cause of SQL performance issues. Usually these problems arise because the DBA requests a virtual disk of the VI admin, the VI admin places the VMDK on a LUN that may or may not meet the DBA's performance needs. For instance:

VMs' VMDK files placed on VMFS volumes without enough spindles.
Many VMDK files placed on a single VMFS volume which could use more spindles.
Database and log files placed on the same LUN which, you guessed it, could use more spindles.

This may be obvious to some, but this problem occurs again and again. The VI administrator should be aware of a few technical items that can help understand and avoid this problem:

Based on the IO demands of the DB files, a certain number of spindles should be guaranteed to this file. This means that its VMDK must be placed on a VMFS volume to accout for the SQL Server's demands and all of the other demands on that volume.
Mixing sequential activity (such as log file update) and random activity (such as database access) results in random behavior. This means that the LUN configuration in the pre-virtual physical environment may not be sufficient for the consolidated environment. This is discussed some in Storage Performance: VMFS and Protocols.
When storage isn't meeting the SQL Server's demands, the device latency or kernel latency (queueing time) will increase. Read up on these counters in Storage Performance Analysis and Monitoring.

Item 2: Use Recent Hardware

Often companies that are dipping their metaphorical toes into
virtualization want to run proof-of-concept (POC) experiments to verify
that the virtual platform can meet their performance expectations. But
its surprising how many times these experiments are run on older,
poorly-performing hardware. Presumably the shiny, new systems were in
use for production applications so only the mothballed, cobweb-covered
servers from a previous generation were available for the POC. This
causes many problems. Check out this slide from a talk on SQL Server at VMworld Europe 2009:

The slide points out a couple of things. First, the larger caches and shorter pipelines on newer Intel processors results in a considerable drops in performance overheads. Second, the latency of the VMEXIT instruction, which determines the amount of time it takes to transition from the VM to the VMkernel, has shrunk by a large amount with subsequent generations of hardware. And don't forget the other additions from Intel and AMD such as hardware assisted memory management and IO virtualization.

Item 3: Follow SQL Server Best Practices

Microsoft has kindly provided a web page of best practices for SQL Storage configuration. These be practices should still be followed when configuring your virtual SQL deployments!

Item 4: Configure VM Identically to Native and Run The Right Test

For many SQL Server POCs the goal is to measure the VM's ability to perform, with respect to the virtual platform. If this comparison is to be performed, its critical that the VM be configured identically to the physical hardware. Obviously this means that the VM should be run on the same hardware using identically configured LUNs. Its also important to ensure that the VM has the same number of vCPUs and amount of memory as the physical baseline. This means restricting the number of pCPUs and amount of memory with NUMPROC and MAXMEM, respectively, in boot.ini.

It also means that the test being applied should be understood. If a benchmark is chosen that uses a very small database, the content will be cached and the storage system won't be used. This can skew the results and produce recommendations not consistent with production deployments. Here is another slide from the same VMworld Europe 2009 presentation detailing some of what we know about the SQL Server benchmarking alternatives:

We at VMware prefer DVD Store.

Item 5: Use VMware's ESX Server

VMware's hosting products, VMware Server, VMware Workstation, and even VMware Fusion, are all capable of running SQL Server. But if the database is going to be run in production on enterprise-class hardware, use VMware's enterprise-class hypervisor: ESX Server. These products are not often confused by the initiated but rogue members of large companies often run off-the-books proof-of-concept experiments on VMware's hosted products. When they produce results they don't like, the results get spread throughout the company which can slow the virtual deployment.

Consider the following data, again from the VMworld Europe 2009 SQL Server presentation:

This information is getting a bit dated now, as it was performed years ago on ESX Server 3.0. But the point stands: before believing results claiming that "VMware cannot run SQL Server" its worth investigating the platform used to generate the results.

Item 6: Understand Memory Management and Configure Correctly

Database performance is heavily dependent on the amount of memory available. Almost without exception, providing more memory to SQL Server will improve performance. However, if that memory is coming from a host that is already over-committed or is being provided through workarounds to 32-bit limitations, performance may suffer. Here are a few keys for SQL Server memory management:

If more than 3 GB is desired, use 64-bit versions of the OS and application.
If memory is over-committed on the box, set reservations for performance-critical SQL Server VMs to guarantee that those VMs' memory isn't ballooned or swapped out.
If SQL Server's "lock pages in memory" parameter has been set, provide set the VM's reservations to the amount of memory in the VM. This setting can adversely interfere with ESX Server's balloon driver. Setting reservations will stop the balloon driver from inflating into the VM's memory space.

Item 7: Align Disk Partitions

This item is really a special but very important case of item two, follow best practices. Partition alignment can impact storage performance which can be critical to some SQL Server VMs' performance. See VMware's paper on partition alignment for more information on this.

Whitepapers

Verifying that VT-x is Enabled and Locked at Boot

jmattson@vmware.com — Mon, 01 Dec 2008 19:48:09 GMT

Intel first introduced VT-x on late-model P4 processors (codenamed Paxville). Many Core, Core 2, Core i7 and Core i5 processors are VT-capable (as are their Xeon-branded counterparts). However, some skus of these models are not VT-capable. (For example, the Q8200 Intel Core 2 Quad Processor is not VT-capable). You can check to see if your processor is VT-capable here. If your processor is VT-capable, "Intel Virtualization Technology" will be listed as one of the supported features.

On Intel CPUs, it is expected that the BIOS (or other firmware) will either enable or disable VT-x at boot, and will then lock the setting so that it cannot be subsequently changed without powering down the CPU. Note that VT-x must be both enabled and locked before software can make use of the VMX extensions. This operation must be performed for each core of a multi-core system, and for each thread of a multi-threaded system.

Because the VT-x setting is typically locked at power on, it is necessary to fully power down the system after changing any VT-x options in the BIOS. A simple reboot is not sufficient! After saving your BIOS changes, I recommend that you actually pull the power cord(s) out of the wall and wait ten seconds. For laptop systems, you may have to remove the battery as well.

Some BIOSes do not have VT-x enable/disable options. Some BIOSes with such an option fail to enable/disable VT-x on each core of a multi-core system or on each thread of a multi-threaded system. To verify that VT-x is enabled and locked on each core/thread of your system, download the attached ISO, burn it to a CD, and boot your host from the CD.

If the ISO reports that your CPU is not VT-capable, and your CPU should be VT-capable, you may have encountered an Intel chip erratum (possibly AW67 or AV69, depending on the CPU). The effect of this erratum is that some CPU features are reported incorrectly after a particular machine specific register has been updated by the BIOS. The solution to this problem is to obtain an updated BIOS from your system vendor.

If the ISO reports that VT-x is unlocked on one or more threads/cores, a workaround is available for current VMware hosted products. Simply add the following option to your system-wide configuration file:

hv.enableIfUnlocked = TRUE

On Linux systems, the system-wide configuration file is /etc/vmware/config. On Windows systems, the system-wide configuration file varies according to VMware product and Windows version. For VMware Workstation on XP hosts, the system-wide configuration file is C:\Documents and Settings\All Users\Application Data\VMware\VMware Workstation\config.ini. For VMware Workstation on Vista hosts, the system-wide configuration file is C:\ProgramData\VMware\VMware Workstation\config.ini. For other VMware products, adjust the path appropriately.

Note that this workaround can also be helpful if your BIOS fails to enable and lock VT-x when waking from hibernation. However, the ISO is not helpful in diagnosing this failure. If VT-x works after a power-cycle but fails after resuming from hibernation, then you should try the above workaround.

If your BIOS specifically disables VT-x and locks it in that state, then it is impossible to subsequently enable VT-x through software.

Note that VT-x is unavailable to normal software if you have enabled "trusted execution," which restricts VT-x to "trusted" code. You should disable trusted execution in the BIOS and power-cycle the system.

Running Nested VMs

jmattson@vmware.com — Mon, 01 Dec 2008 02:17:29 GMT

Requirements

It is only possible to run nested VMs when the outer VM uses hardware virtualization (Intel's VT-x or AMD's AMD-V) and the inner VM uses the classical BT (binary translation) monitor. You may run any 32-bit or 64-bit operating system as the outer guest. You may only run 32-bit operating systems as the inner guest. These configurations are entirely unsupported.

Nested VMs are only possible when the host is running ESX 3.0 (with some caveats), ESX 3.5 (some AMD CPUs only), Workstation 6.x, or Fusion. Note that you cannot run nested VMs on Intel systems when the host is running ESX 3.5.

See the appropriate section below for the configuration options required for the two guests.

Intel VT-x Hosts

Intel VT-x is supported on most Core, Core 2, Core i7 and Core i5 processors, and on some late-model P4 processors. Current VMware products only support VT-x on 64-bit (EM64T) Intel processors.

Before you can use VT-x, you have to enable it in the host BIOS. For the BIOS change to take effect, you must power-cycle your system after enabling VT-x. Some systems do not have a VT-x setting in the BIOS. If your system falls into this category, complain to the system vendor. Many BIOSes have bugs in their VT-x support. Common errors include enabling VT-x only on a subset of the logical cores or neglecting to re-enable VT-x when waking from hibernation. If your system falls into this category, complain to the system vendor.

ESX 3.0

The configuration file for the outer guest should contain the following line:

monitor_control.vt32 = TRUE

The configuration file for the inner guest should contain the following line:

vmx.allowNested = TRUE

Note that the 'vmx.allowNested' option is not supported by older VMware products. For nested VMs to work on an ESX 3.0 host, the VMware product running in the outer guest must be Workstation 6.0 or later or ESX 3.5 or later. Furthermore, you must not use the backdoor in the inner guest. In particular, this means that you must not install VMware tools in the inner guest.

Workstation 6.0 and Fusion 1.x

The configuration file for the outer guest should contain the following lines:

monitor_control.vt32 = TRUE
monitor_control.restrict_backdoor = TRUE

Workstation 6.5, Fusion 2.x and ESX 4.0

The configuration file for the outer guest should contain the following lines:

monitor.virtual_exec = hardware
monitor_control.restrict_backdoor = TRUE

AMD-V Hosts

AMD-V is supported on Rev F and later Family 0FH AMD64 processors, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona or Phenom). Thus, you can use AMD-V on Family 0FH Rev F and Rev G processors with Workstation 6.0. However, you need a Family 10H processor to use AMD-V with ESX 3.5 and Workstation 6.5.

ESX 3.0

The configuration file for the outer guest should contain the following line:

monitor_control.enable_svm = TRUE

The configuration file for the inner guest should contain the following line:

vmx.allowNested = TRUE

Workstation 6.0

The configuration file for the outer guest should contain the following lines:

monitor_control.enable_svm = TRUE
monitor_control.restrict_backdoor = TRUE

ESX 3.5

The configuration file for the outer guest should contain the following lines:

monitor.virtual_mmu = hardware
monitor_control.restrict_backdoor = TRUE

Workstation 6.5 and ESX 4.0

The configuration file for the outer guest should contain the following lines:

monitor.virtual_exec = hardware
monitor_control.restrict_backdoor = TRUE

Upgradation of ESX Server from 3.0.2 to 3.5

Sun, 26 Oct 2008 00:37:40 GMT

Hello Everybody,

Presently we are running ESX server having version 3.0.2. planning to upgrade the ESX server from 3.0.2 to 3.5. Could anybody of us let us know what are the steps need to be taken care before upgrading and how to gohead with the upgrade proceedure.

Thanks in Advance

Raghu

Here is a link to VMware's upgrade that has the steps for a successful upgrade - http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_upgrade_guide.pdf

If you find this or any other answer helpful please consider awarding points by marking the answer correct or helpful

This document was generated from the following thread: Upgradation of ESX Server from 3.0.2 to 3.5

Can a single VM use all the resources of multiple hosts, then keep running if a node is lost?

Fri, 10 Oct 2008 21:59:02 GMT

Hi!
I have gotten conflicting answers from VMware education materials ("A VM can only run on a single host at a time." - VI3: Install and Config, pg 217) and VMware sales reps ("Sure, that should work fine.") for the following scenario (my hope is that someone out there has tried this configuration and can let me know if it will work before I commit to buying 2 ESX VC licenses).

Situation: Have a high end server that currently maxes out the resources available on the box. Have another identical server available as a spare.

Goal: Combine hardware resources of both servers and make them all available as a single machine.

Possibility that needs a thumbs up or down: Would like to install ESX3 and VC on both hosts and cluster all the resources of both hosts together into a resource pool. Then create a single VM that would use all the resources in the pool. In case one of the hosts in the cluster failed, the VM would need to keep running on just the resources on the single host (half of what is in the pool).
I know this is somewhat backwards to what VMware is usually used for. However, does anyone know if such a configuration would work, and if it would be supported by VMware? Or any other suggestions on how to achieve the Goal listed above?

Thank you in advance for your kind help!!!
Mak

Rule of thumb - do not listen to sales people - what you read in the documentation is correct a VM can only run on a single host at a time - this is true even when you create a cluster with in Virtual Center - what you gain particularly with a DRS cluster is if a host is becoming overloaded DRS can automatically move the Vm to another host that has more available resources -

Hi

One VM can ONLY run on one Host.
You can't reach that goal .
How can one server max out your hardware ? How many cores is in the Host and how many vcpu's in the vm

Best regards
Lars Liljeroth

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

The current machine is not running as a VM - it was just an idea make it a VM to hopefully make more processing available and create some redundancy. The app is a custom db with lots of transactions that was only supported on a single CPU AMD 2.8 Opteron. I know, lame config, but that is what the vendor said they would support.

Hello,

VMware HA will reboot the VM if a host fails.
VMware DRS will move a VM to another host if there is CPU or memory contention

VMware FT (not available yet) will run in fault tolerant mode.

So there are ways to get close to what you want but there is no way to get there now.

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354
As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

Can you tell me more about VMware FT (Functionality, Availability, etc, since that may be what I need)?

Hello,

VMware FT was announced at VMworld as part of the VDC-OS tool suite. On http://www.vmworld.com you will find some presentations on FT. http://www.vmworld.com/static/sessions/2008/BC2621.html would be where I would start.

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354
As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

Rule of thumb - do not listen to sales people

... EVER ...

This document was generated from the following thread: Can a single VM use all the resources of multiple hosts, then keep running if a node is lost?

vi reference card

Fri, 10 Oct 2008 11:36:17 GMT

Putty & Certificate Authentication.

Wed, 24 Sep 2008 13:39:10 GMT

Can any one help?

Scenario

I have 2 ESX 3.0.2.Upd1 host machines that are managed from a laptop. Using PuttyGen I have created my id_rsa.txt, id_rsa.pub and id_rsa.ppk keys. I have then FTP'd my id_rsa.txt to /$home/.ssh directory and changed the permissions to -rw-------- and ownership to owner:owner, and renamed teh file authorized_keys. I have then started up Putty in the normal way and configured it to use the id_rsa.ppk previously created.

Problem

I get either "Server Refused Key" or "No Authentication Methods Available".

Other Info

I have accepted the Key request that appears on opening a Putty connection for the first time. But this is not using the keys that I have created and thus doesn't fulfil the security criteria that have been set. The end result must be certificated authentication with all other means disabled. Currently I have bypassed this by ammending the sshd_config file.

Is there a file or cache that holds the keys that can be flushed or deleted or ammended to clear out any spurious or out of date keys?

Hello,

Please give the exact puttygen steps you took.

Also the exact changes you made to sshd_config.

Otherwise we are shooting in the dark here.

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

To create my keys I did teh following:

1 Open PuttyGen and select Generate, move the cursor around the space to generate the key.

2. Enter a "Key comment" for Reference purposes.

3. Enter a "Key Passphrase" then "Confirm Passphrase".

4. Click on "Save Public Key" and save it as a id_rsa.pub on the desktop.

5. Click on "Save Private Key" and save it as a id_rsa.ppk on the desktop.

6. Highlight, and copy the contents of the key window "ssh-rsa....to end, and paste into Notepad. Save as id_rsa.txt.

The id_rsa.txt is then ftp'd over to the ESX Host into the /$home/.ssh and renamed authorized_keys

As I said previously change ownership and permissions, and as far as I am aware, that should be it.

To use, open Putty, enter IP Address, select SSH/Auth from the Catagory tree, browse to my id_rsa.ppk file and click Open.

That is as far as I get.

The problem is that this has been working in our test environment, and I am doing nothing different. The systems are identical except for one factor and that is that the test environment was set up and is being managed from the same laptop. However the Dev environment was set up with one laptop, and managed with another having moved teh .ppk file across. I thought there may be some tie in between hardware and key so I generated a new set of keys with the new laptop, but it still fails. That is why I enquired if there was some sort of cache of keyfile that gets written to, that could be cleared.

The only change I made to the sshd_config was to enable Password Authentication so that if and when the certificate failed I could still log on. This eventually will need to be disabled once I have sorted out this problem.

Hello,

THere is no keyfile cache. Are you trying to get in as root or another user? Root will not work by default.

Look in ~/.ssh/authorized_keys and see if you have two lines with the same comment. Sometimes that is the issue.

Did you add it to the proper users authorized_keys file?

Simple things, I know but there is no cache mechanism for sshd. You could always do 'service sshd restart' and that will clear anything out that was there.

Hi Edward,

I am using a user that has admin privilages, and once logged on su to root. I have looked at the authorized keys file, and there is only one key in it, the one that I created using Putty?

I cannot see any reason why this configuration doesn't work. I have done all the things that you suggest, during my fault diagnosis but cannot coome up with a cause.

My interntion is to build a stand alone host and try to either re-create the fault or see if re-building the host and starting with a clean sheet will cure the problem.

I will keep you posted as to my success/failure...

Hi again Edward,

Thanks for your help thus far, just to give a quick update. I haven't been able to reproduce the problem, but I have been able to get everything to work as expected. This leads me to believe my original thought about a user profile issue was the culprit although not proven. I think that the solution to my problem will be to remove the offending profile and recreate it. Then generate a fresh pair of keys, and take it from there.

Many thanks for your advice and assistance, this isn't the first timie you've got me out of a hole.

Kind regards

Steve Pickering

Hello Steve,

Just another thought, all items pertaining to SSH logins are logged to /var/log/secure and /var/log/messages. Do either of these files have errors pertaining to your login? It could be a permission problem with some aspect of the ssh subsystem. But recreating the profile and keys is a good way to go.

Ready Time

Wed, 27 Aug 2008 17:50:36 GMT

Introduction

Ready time is as important as it is confusing. I'm going to collect a few thoughts on ready time in this collection point with the hopes that some of the confusion around this important part of virtual system performance can be eliminated.

Details

Stated simply, ready time is the amount of time a VM wants to run but has not be provided CPU resources on which to execute. Somewhat confusingly, ready time is reported in two different values between esxtop and VirtualCenter. In esxtop is reported in an easily-consumed percentage format. A number of 5% means the VM spent 5% of its last sample period waiting for available CPU resources. In VirtualCenter ready time is reported as a time measurement. In VC's real-time data, which produces sample values every 20,000 ms, a number of 1,000 ms is reported for a 5% ready time.

There is so much more to know about ready time that I'm not going to reproduce here. Read the whitepaper on the subject for more details. There have been no changes in the details on ready time since ESX 3.0 that make that paper out-of-date.

Interpreting Ready Time Values

The most common question we get on ready time is, "what ready time numbers constitute a problem?" While there is no easy answer to this, we can offer some guidance on the acceptable values. But before I lay that out, let me say that ready time should not be the ultimate measurement of system performance. As always, user experience and latency should be. There are some situations where user experience is horrible on a system with no load and virtually zero ready time. This could happen with a mis-configured array, as an example. And occasionally we see aggressively-consolidated hosts showing very high ready times that are meeting user needs. There are no absolutes with ready time.

But, there are a few general regions into which ready time values can be binned. Note that these ready time values are per vCPU. esxtop reports ready time for a VM once its been summed up across all vCPUs. That means that 5% ready on each of four vCPUs will be reported as 20% ready at the VM level. This is the high end of a very light amount of ready time.

Value, per vCPU	Description
r == 0%	This doesn't happen. The very presence of a hypervisor between the operating system and the hardware means that there is a non-zero ready time on all operations. But on healthy systems this number is so small that end-users don't know their workload has been virtualized. See the next section.
0% < r <= 5%	This is the "normal" region for ready time. Very small single digit numbers result in a minimal impact to user experience. If performance problems exist on the system and ready time falls into this region, your problems lie elsewhere.
5% < r <= 10%	In this region ready time is starting to be worth watching. Most systems function healthily with ready time in this region but highly sensitive measurements may be suffering.
10% < r	While some systems continue to meet expectations, double-digit ready time percentages often mean some action is required to address performance issues. See the last section for guidance.

Again, remember that VirtualCenter performance numbers must be re-calculated to percentages to find the category on the above table. But since VC reports ready time per vCPU, no special arithmetic is needed to account for the number of vCPUs in the VM (as is needed with esxtop.)

Causes and Correction

There are two general areas that can cause unnecessarily high ready times:

Overloaded hosts.
Excessive use of SMP.

Host Overloading

The most common cause of high ready time is trying to get too much work out of too little hardware. Consider the following simple case: on a hypothetical system with only one physical CPU, if two 1-way VMs are fully loaded by their users then each wants to have an entire CPU. Because only one is available, ESX will time share that resource and give each of them only 50% of the CPU. As a result, each VM will spend 50% of its time waiting for the processor. This would be reported as 50% ready time.

Often this condition is observable when ready time is high and total host CPU utilization is also very high. The only fix for this is to back off the load on the system. VMs should be migrated off or processor resources should be increased.

Excessive SMP

In ESX Server 2.5, SMP guests had to be co-scheduled to start at the exact same moment. If a 2-way VM was ready to run but only one physical core was available, the VM would not be scheduled until a second core was freed up. This would increase its ready time. In ESX Server 3.0 and later versions, relaxed co-scheduling was introduced which meant that a subset of a VM's vCPUs could be scheduled ahead of others. However, guest operating systems still require some degree of co-scheduling which means that the relaxation isn't absolute. In short, increasing vCPUs still puts some burden on the scheduler to try and co-schedule the vCPUs that can increase ready time. This is one ready why VMware advises only allocating vCPUs to VMs that are using them. Read Co-scheduling SMP VMs in VMware ESX Server for more information on co-scheduling.

This condition is manifested by hosts that have sub-optimal CPU utilization and lots of SMP VMs. A host may have a dozen 4-way VMs with each showing high ready time but only be at an aggregate 40% CPU utilization. This is a clear sign that the scheduler is spending a great deal of time managing unneeded vCPUs.

"Permission to perform this operation was denied" when users try to mount CD/DVD in VC 2.5

Thu, 21 Aug 2008 21:52:37 GMT

We are running ESX 3.0.2 and VC 2.5. Just upgraded to VC 2.5 last week.

Today I started receiving reports from users where if they click "Connect CD\DVD 1" and select either a local drive or an ISO file, they get the permission denied message. I have tried increasing their permissions all the way up to Administrator of the VM and no difference. I am able to login to VC and connect media without any issues as an admin on Virtual Center server, but can't find a way to give anyone else the ability to do this after the upgrade. Is this a known bug? Users were able to do this in VC 2.0 with Virtual Machine User permission.

hmm...they should be able to do this with the permissions you have stated. Wonder if you found a glitch between VC 2.5 and the older ESX 3.0.x.
Do you know for certain if this was working at any point after your VC upgrade and now? If not, maybe you need to restart some services on ESX. It seems like there is a disconnect between them somewhere.

I would try the following:
-restart VC service on VC server
-on ESX: service mgmt-vmware restart or better yet just reboot the thing if you can to rule out any other service that needs to be restarted.

I too am having this exact issue, except I am running ESX 3.5

I found a workaround from VMWare, unfortunately it is not ideal and I would prefer some kind of patch to fix this issue.

I upgraded a server to 3.5 and restarted Virtual Center and the problem persisted. After opening up a support case here is the info I got:

Go to the Hosts & Clusters folder in Virtual Center.

On the Permissions tab, add a user or group of users and give them the 'Virtual Machine Power User' role, but be sure to UNCHECK the 'Propogate to Child Objects" option. This way, the users don't receive access to all VMs in your datacenter.

This will allow them to connect removable media, but they must be kept there for it to work.

rpmello,

I had this same issue when usered tried to open a console window as well as mounting an ISO in VC 2.5. I did this workaround and its resolved my issue thanks! I hope VMware properly adresses this issue as well.

rpmello,

Your solution worked, thanks for saving me a phone call to support. Yes, this is indeed an issue with the upgrade and I hope a patch comes out.

Thanks, this worked for me as well. Very frustrating bug.

I had the same issue. This workaround worked for me.

I don't like this ugly workaround. I hope a patch is released soon.

Slight modification, they do not appear to need 'Virtual Machine Power User' role at the Hosts & Cluster level. Simply doing Read Only (and unchecking the propogate) seems to work in our testing of this today.

After I added the "ReadOnly" on "Hosts & Clusters", I'm able to add an ISO or Power of a VM.

The User hast "VM Administrator" rights on a VM, but if this user tries to open the settings of a VM, he gets "Exeption of type "Vmomi.Fault.NoPermission" was thrown.

Any ideas?

Thanks,

Chris

Read-only may not give you all the rights that you require. I tried read-only with and without propagate enabled and got the Vmomi.Fault.NoPermission when I tried to clone a VM. I added just the permission to view the datastore (all privileges\datastore\browse datastore) and got "Permission to perform this operation was denied."

I cloned the Virtual Machine Administrator role, pulled out some of the options that allow administration of the host server, and assigned it at the Datacenter level with no propogate. That fixed everything.

This new VC 2.5 is a disaster. It used to be you could just assign Administrator rights to a cluster and the cluster administrator could do everything he needs. Now, we need work-arounds. It looks like this upgrade was a rush job, much like NT4 SP6. I'm wondering when VC 2.5a is going to be out.

I experienced this issue in 2.0.1 so I am not convinced it is only VC2.5's bug.

Hi Guys,

Anyone find a solution for this...?

According to the release notes some of this is fixed in VC 2.5.0 Update1 I don't know yet because my change window for VC 2.5.0U1 is 5/1. If someone upgraded can you please update us on this situation.

Yes it is fixed.

See

http://kb.vmware.com/kb/1004327

For this issue, minimum privilege required at "Hosts & Clusters" level is "Datastore/Browse Datastore" without propogation. I have a custom role "CD/DVD Media Manager" with "Datastore/Browse Datastore" privilege. I assign this to users at top level "Hosts & Clusters" without propogation. At VM level you need more privileges such as "Configure CD Media" and "Modify Device Settings".

I think, there is an underlying assumption that all image files are stored in datastores.

Hello,

Did you try this and it worked or are you just reading what they posted? We have VC 2.5 U1 and ESX hosts 3.0.2 U1 and it does not work....still looking. I guess I'll open another ticket. I'll post if they fix it

Regards,

J Kelso

I installed VI2.5u1 and upgraded the host to 3.5u1 a while back and it did not have the issue.

Oh, OK. That may be why it works for you and not me The solution I was given by tech support was to give the user in question permission at every level, Hosts & Clusters down to the Resource pool with propogate to child unchecked all the way down until the resource pool I was trying to give it to him on. Very cumbersome but it works.

Regards,

JKelso

This document was generated from the following thread: "Permission to perform this operation was denied" when users try to mount CD/DVD in VC 2.5

Large Memory Pages

Fri, 08 Aug 2008 16:02:31 GMT

In ESX Server 3.5 VMware introduced support for large memory pages in the guest. Large memory pages, an architecteral feature available in x86 microprocessors for decades, can be used to improve performance on workloads that make use of them. With CPU, hypervisor, OS, and application support, throughputs can go up and CPU utilization can go down. Since applications such as Oracle databases and Java have been using large pages on Linux and Windows for years, the introduction of this support on ESX Server allows for increased gains in performance over previous virtual installs. VMware is currently the only virtualization vendor to support large pages.

VMware's support for large memory pages is detailed in the Large Page Performance performance study. That paper includes data on throughput gains in SPECjbb. The results are duplicated here:

At LinuxWorld 2008 VMware presented further data on the value of large memory pages with Oracle databases. Here is a chart showing those gains with VMware binary translation (BT):

Data also shared at LinuxWorld on the value of large memory pages with AMD Rapid Virtualization Indexing (RVI; formerly called NPT):

The gains derived from the presence of large pages in these Oracle/Swingbench results are atypical. Large pages have been documented in numerous locations to provide benefits between 5-20% on most database applications. The increases shown here (of up to 350%) are due to the specialized configuration which is less about demonstrating real-world application performance and more about stressing the underlying configuration to uncover strengths and weaknesses.

Storage Queues and Performance

Thu, 24 Jul 2008 00:51:16 GMT

Introduction

VMware recently published a paper titled Scalable Storage Performance that delivered a wealth of information on storage with respect to the ESX Server architecture. This paper contains details about the storage queues that are a mystery to many of VMware's customers and partners. I wanted to start a wiki article on some aspects of this paper that may be interesting to storage enthusiasts and performance freaks.

Two Important Queues

Let's use the following figure as a starting point for this discussion.

For the purposes of this paper, I'm going to call the two different queue types the "kernel queue" and the "device driver queue". The device driver queue is specified in the device itself and has historically been configured through Linux-like module commands in the console operating system. More on that in "Changing Queue Depth" below. The kernel queue should be thought of as infinitely long, for all practical purposes. Any time the device driver queue gets full, commands to the storage will queue up in the kernel.

Note that each LUN gets its own queue. This means that when you change the queue depth in the device driver, you're changing the queue depths for many queues. The underlying device (HBA) is going to have a hard limit on the number of active commands it will allow at one time. This should be considered when setting queue depth. If your HBA can support only 2,000 active commands but it is addressing 40 LUNs, a specified queue depth of 64 won't allow that many commands to all LUNs. This being due to the fact that 64*40 = 2,560--which is more than the 2,000 maximum commands. In practice this is rarely a concern, though, as rarely are so many LUNs being simultaneously addressed through a single HBA and so many outstanding commands being issued to these LUNs.

Device Driver Queue Function

The device driver queue is used for a low-level interaction with the storage device. It controls how many active, or "in flight", commands there can be at any one time. This is effectively the concurrency of the storage stack. Set the device queue to 1 and each storage command becomes sequential: each one must complete before the next starts.

But if the device queue is left at its default of 32, as an example, 32 commands will be concurrently processed by the storage system. All 32 will be shipped off to the storage device by the kernel and new commands are queued when completions arrive.

Kernel Queue Function

The kernel queue can be thought of as kind of an overflow queue for the device driver queues. But it's not just an overflow queue. ESX Server contains all kinds of cool optimizations to get the most out of your storage. And these features apply to commands in the kernel queue only. Here are some examples of features provided to commands queued at the kernel queues:

Multi-pathing for failover and load balancing.
Prioritization of storage activities based on VM and cluster shares.
Optimizations to improve efficiency for long sequential operations.

There are others, as well.

Impacts of Queue Depths

So, increasing queue depths in the device driver can greatly improve the performance of the storage at the device level. Decreasing the device driver queue will result in increases in usage of the kernel queues. This decreases the device efficiency, but introduces opportunities for optimizations across multiple VMs and devices. So, what's the right ratio of these two depths? We think that the sweet spot lies with a depth 32 device driver queue. That's why we've set 32 as the default device driver queue length.

But your configuration and workloads may benefit from a change to this default queue depth. I'll refer you to the aforementioned storage paper for information on when you might want to change the driver queue depth. I'll just point out a couple of broad observations here:

With fewer, very high IO VMs on a host, larger queues at the device driver will improve performance.
As the VM count grows and storage performance features--like shares, load balancing, failover, etc.--become more important, the default queue depth is best.
With too many servers each having too large of device queues, your storage array could easily be overloaded and see its performance suffer.

Improving Storage Performance

Now that we've covered how storage queuing works, you may be wondering how you can monkey around with these queue sizes for optimal performance. I can tell you as someone that has been involved with many, many performance analysis projects that changing queue size is rarely a fix to an acute storage performance problem. You should first go through the analysis techniques in Storage Performance Analysis and Monitoring. That may or may not lead to changing queue depths.

But, in the event that you do end up changing queue depths...

Changing Queue Depth

We have a helpful knowledge base article that describes the process of changing the device driver queue. Unfortunately, as of today (7/24/08) this document only describes how to change queues through the console operating system. No information is provided for ESXi. I've contacted the KB owner and will have that document updated ASAP.

memory usage & vm right sizing

Wed, 16 Jul 2008 16:52:05 GMT

I'm trying to get my head around VM memory usage as it relates to DRS & ESX 3.

I have a cluster of 4 hosts with 48 Gb & 43 Ghz available. Currently 60 VMs running here.
CPU util is around 20%.
Memory util is around 74%.
Active memory shows around 5Gb.

The 74% number exactly matches the amount of RAM allocated to the vms vs the physical RAM in the server.

With a default ESX config, what's the over committment allowed? How is this adjusted? Where is this documented?

Is my memory usage apparently high just because there's no contention and therefore no ballooning or swapping is happening?

Has anyone developed a good process for determining if vms have the right amount of memory allocated?

I believe everything is working fine, but I need to understand how much I can push things before performance will erode.

Are you using memory reservations at all? If you make reservations for your VM's that are equal to the amount of RAM assigned to them then swapping and page sharing does not occur. You can overcommit pretty heavily if you are willing to pay the performance penalty. If you do not set reservations then ESX create a VSWP file equal to the difference between the amount of physical memory assigned to the VM and the reservation it has. By default memory reservations are set to 0. So if you have a VM with 2GB of memory without a reservation it will create a 2GB vswp file when it is powered on. It will start using the vswp file if the server is out of physical RAM. If you set a 1GB reservation then it will create a 1GB vswp file instead. The vswp files are what allows for memory over-commitment.

Good Windows memory doc...

Understanding memory usage in Windows 2000 - http://www.microsoft.com/technet/prodtechnol/windows2000serv/tips/w2mem.mspx

Also see this post...

http://www.vmware.com/community/thread.jspa?messageID=549391򆈏

These are good memory documents...

The Role of Memory in ESX Server 3 - http://www.vmware.com/pdf/esx3_memory.pdf
Memory resource management in Vmware ESX Server - http://www.vmware.com/pdf/usenix_resource_mgmt.pdf

thanks. This is very helpful. I've read thru the whitepapers listed before, when I ran into swapping under ESX 2 and there wasn't much to do about it.

Now I'm getting close to saturating resources and we'll still need to bring some more vms into the cluster before the new blades show. I can either stop adding new vms to the cluster or deal with the situation with new features.

I have not configured many reservations.

Going forward I think using resource pools to segregate systems of different priority will be the way to go, then fine-tuning the pecking order in those with individual reservations. This way the performance drain on a swapping system can be focused on those less important or demanding systems.

With this discussion I finally get what all the knobs are good for. I knew what they did, but didn't get the cohesive picture.

1. Use resource pools to group machines & guarantee resources
2. Use reservations & limits to control individual system resources
3. Use priority to balance demanding vms

None of it matters without contention except for memory max - reservation = swap size.

Thanks. What's the e stand for? esiebert7625 doesn't exactly roll of the keyboard.

This document was generated from the following thread: memory usage & vm right sizing

Feature Request - naming SCSI targets

Tue, 15 Jul 2008 18:52:21 GMT

A customer requested a feature be added whereby a name can be provided for SCSI targets; a name that would be made aware by all ESX servers in a cluster. That way, the appropriate RDM can be selected in the list when configuring additional guest machines.

Windows Time in Guest OS

Sun, 22 Jun 2008 21:27:23 GMT

I followed KB1339 to get the ESX host syncing time with an NTP host. Infact, it's pointed to the same place our domain PDC emulator is pointing.

The article says that after you configured VMWare Tools to sync time with the ESX host, you should disable the windows time service. My question is, is this part necessary? My co-worker seems to think that the VMWare tools time sync will override the windows time service so disabling it isn't necessary. I'm not convinced that's true. Any thoughts?

Hi,

If you enable the time synch with VMWare Tools, you can disable windows service. The windows service use NTP protocol, and by this way, synchrosize on the network. VMWareTools, doesn't use NTP server but use internal protocol to comunicate with ESX server. The ESX server use NTP protocole.

If you use both (VMWare Tools and Windows Service) you can have 2 ways to keep you VM on time : Domain and ESX. It can create some special effect (too much time synch, time synch very long, ...). If VMWare Tools works correctly, it should desactivate the possibility to your VM to synchronize with the domain. I didn't check this point.

My point of view, the windows time service should running and don't use VMWare Tools because some fonctionnality of windows need this service. Maybe you don't use them. So, to simplify the administration, I prefere have the same configuration on all servers and keep windows time service. But if you want to use VMWare Tools, you can disable the windows time service.

Olivier

###############
If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

We have all our ESX hosts synching to our NTP server and all VMs synch to the ESX host and the windows time service is disabled. Been running that way for over 2 years without any issues.

The problem with leaving both running is that they are not aware of each other and both can move the time on the VMs. Check out Timekeeping inf Virtual Machines for an in-depth discussion of the way it works.

If your servers are part of an AD domain then they should have Windows Time Service enabled and VMware Tools Time Sync disabled. The exception to this being your PDC emulator which is at the top of the AD time pyramid. Your PDC emulator needs Windows Time Service enabled and VMware Tools Time Sync enabled to sync NTP time with the ESX server. Accurate timekeeping is critical to the proper operation of Kerberos and your member servers should only sync from one source (WTS). More on this below...

Configuring windows time service (not in ESX) - http://www.windowsnetworking.com/articles_tutorials/Configuring-Windows-Time-Service.html
Vmware time sync and windows time service (understand windows time first) - http://kb.vmware.com/kb/1318
Time sync - http://www.vmware.com/community/thread.jspa?forumID=21&threadID=16115&messageID=186017
Timekeeping in VMware virtual machines - http://www.vmware.com/pdf/vmware_timekeeping.pdf
How to configure an authoritative time server in Windows Server 2003 - http://support.microsoft.com/kb/816042
Virutalization of Active Directory - http://www.vmware.com/community/thread.jspa?messageID=352424񖂨
Considerations when hosting Active Directory domain controller in virtual hosting environments - http://support.microsoft.com/kb/888794
Virtualizing a Windows Active Directory Domain Infrastructure - http://download3.vmware.com/vmworld/2006/tac9710.pdf

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Thanks, Eric
Visit my website: http://vmware-land.com
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Hi,

Can I clarify this, as in our environment I have Virtualised our PDC emulator and used the Vmtools time sync option etc this is all working fine its our member servers that we have trouble with. They are still relying on w32 time service under windows the problem we get on the member servers is the time drift associated with not having a hardware clock, my only solution that I could come up with was to setup a scheduled task to run a timesynce at certain periods during a day to keep the guests clock in sync.

The last reply sort of contradicts my findings in our environment, again I repeat the Domain controllers are not a problem and have configured them as per Vmware recommendations and the MS guides, it is only the member servers.

My way of thinking is I have a fix with a scheduled task or I disable w32time service under windows and just use the VMware tools option for time sync.

regards

Mark

I'd stick with Windows Time Service on the member servers...just double check they are using the PDC Emulator by

net time /querysntp - should say perversely --- 'no time server configured for this host'

If it says anything else use..

net time /setsntp

You can always trying reducing the time period between sync's

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters

DWORD Period = 65532

This will give you a SYNC once every 45 minutes until 3 good synchronizations occur, then once every 8 hours (3 per day)

http://support.microsoft.com/kb/223184

_{If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points}

This document was generated from the following thread: Windows Time in Guest OS

Domain Controller time Sync In a VM

Wed, 18 Jun 2008 17:06:57 GMT

I've seen a lot of different configurations for DC time on ESX server both on this forum and others on the internet. I'm getting somewhat confused and would like to get some opinions before making any bad decisions.

My current setup is 2 Windows 2003 DCs on bare metal. I am looking to leave the PDC emulator on a bare metal system, promote a third DC on a virtual machine in EXS 3.5 and demote the second bare metal machine so that I end up with 2 DCs, one of which is a virtual machine that can be cloned for disaster recovery and/or testing purposes.

My concern is the time drift issue that affects virtual machines. I've seen various opinions but the best that I can figure out after reading http://download3.vmware.com/vmworld/2006/tac9710.pdf is that I should:

1) set the PDC running on bare metal to sync time with tock.usno.navy.mil by configuring the registry entries:

HKLM\System\CurrentControlSet\Services\W32Time\Parameters
Type = NTP
NtpServer = tock.usno.navy.mil,0x1

HKLM\System\CurrentControlSet\Services\W32Time\Config
AnnounceFlags = 5

Then stop and start the w32time service and force the time update with "w32tm /resync /rediscover"

2) configure the EXS server as an NTP client syncing to the PDC

3) configure the second DC running as a Virtual machine to sync with the host ESX server by setting:

HKLM\System\CurrentControlSet\Services\W32Time\Parameters
Type = NoSync

Then setting the "time synchronization between virtual machine and EXS server" option in VMware tools

My questions are: Am I missing anything? Does anybody have a similar setup running? Do any of these settings prevent client PCs or member servers from syncing to the domain controllers? The document I linked to above says to set the registry entries in the domain controller group policy but this would put the same settings on all DCs would it not? Can I do it as manual registry configurations as indicated above?

Thanks

Hi

We use the same setup except from step 3. All our vm's use the AD timesync and NO
VM Tools sync...

Works just fine for us for the last 2 years...

I'd go along with Lars. We have the PDC emulator running on metal with an external sync. All other DC and servers are virtual and take the sync from the metal PDC. No server uses the VM Tools to to sync time or takes any other source. Setting the option on all the servers and clients, except the PDC, is a simple as making sure each server is using AD time.

net time /setsntp

net time /querysntp > This computer is not currently configured to use a specific timeserver (+therefore I'll query DNS for a AD timesource+)

Make sure you only have one source any everybody talks to that source. Remember if you have a time drift, so long as everybody is drifting at the same rate - it's not good - but things will still work. You have problems when one half of the network is going one way (esx time) and one half the other (AD time). Keep an eye on your event logs for W32Time error. Works for me.

As both the other posters have have stated do not mix time sychronisation.

Configure the 2nd DC and all other servers to use the PDC as the time source.

The only time I would consider the use of ESX time is when I have no Physical DC's and my ESX Hosts are set to get their time from a NTP source. Even, then I would rather have my PDC set to a NTP source and point all other server to it this way if there is any time drift all server would suffer the same drift and as perviously stated things would still work.

My reasoning for the above is as follows, a misconfiguration of a ESX host (ie not pointing it at the same NTP server or an NTP communication failure could result in time drift between ESX hosts, therefore the posibility of a time jump when DRS or VMotion moves the PDC emulator could give rise to the risk of time drift.

Tom Howarth
VMware Communities User Moderator

Thanks for your input

So if I understand what you are saying correctly I should leave all the default settings on the Virtual Machine DC after I promote it and all time issues will be fine. It will sync with the PDC emulator often enough to avoid any time drift problems.

That is correct,

Please remember to use the helpful or correct buttons if you found any of the information enparted here useful

Tom Howarth
VMware Communities User Moderator

This document was generated from the following thread: Domain Controller time - need input