After two days of frustration, I gave up on ESXi 4 and installed Win2008 R2 with HyperV

[dougclutter]

First off, I'm not posting this because I'm a Windows fanboy...nor am I trying to bash VMware. I have used Workstation since v5 (v4?) and Server v1 for years. I've always been very pleased with the power, elegance, and stability of the VMware products. However, I think VMware needs to hear this story, so they can adjust their product plans.

I was using an old Dell SC420 server for development work. It had Win2003 R2 on the bare metal and it ran 3 Win2003 R2 VMs. Not bad for a 486 HT chip with only 2.5 Gb of RAM! Since this rig is 5 years old, I decided it was time to upgrade.

I purchased a Dell T710 w/1 Quad core HT (E5520) cpu and 12 Gb RAM. Quite an upgrade to be sure.

I never even thought twice about how to configure this. It was going to be an ESXi 4 machine and I was going to just migrate 1 physical and 3 VM machines right over to it. I even considered paying Dell the $148 to have them send along the pre-installed flash version, but I cheaped out at the last minute.

The installation was a breeze. Migrating the machines was difficult only because I didn't know that the destination should be "Infrastructure machine". After about 8 hours, I had everything migrated over. All the machines were up and working, and I was a tired but happy guy.

The next day was not so good. I had two goals that morning: 1) Make sure I had a reasonable backup plan, and 2) Make sure the UPS did a graceful shutdown of the whole rig.

On the old server, I had a simple BAT script that ran a Windows backup job. The BAT script launched every night as a scheduled task. Every couple weeks, I'd rotate one of the three USB drives assigned to this task from my home and all was good.

Anybody who knows anything about ESXi knows where this is headed, so I'll just wrap up. Here's a summary of the troubles I faced:

• There is no easy way to backup the ESXi drive to a USB drive. I was faced with several poor options. The two most promising options were: buy NAS units (too expensive), or rewrite the backup scripts and run them inside the instances (too time consuming). This is an omission that seems simply bizzarre to me. Almost every electronic device we buy (cameras, PDAs, computers, flash drives, external drives), all come with a USB port. Why would ESXi not support USB?

• There isn't even the most rudimentary backup program included. I never thought I would miss the Windows backup program, but ESXi made me yearn for it!

• There is no easy way to get ESXi to shutdown when an APC Back UPS XS 1300 tells it to...via a USB cable. Once again, we've had UPS units for eons. They have all switched over to USB ports to let the computer know that the battery is nearly depleted. Then the computer can shutdown nice and clean so we don't lose anything.

• And finally, one of the reasons I bought the new server was so I could start getting a little more aggressive with testing. We need to automate the process of starting a virtual machine, running a bunch of tests, collecting the results, and shutting it down. If I'm reading things correctly, ESXi 4 has yanked away the APIs that would make this possible.

Ultimately, I decided that Win2008 R2 and Hyper-V made more sense for me. Within a few hours, I had the new server up and running and all my issues with backups, UPSs, and programmable APIs just went away. Perhaps I'm the exception in the world, but I was really disappointed to leave ESXi behind and I don't think I would have had to if VMware had really thought things out.

One final note, I was shocked that there was absolutely no way to purchase a "single" server license for ESX. Why force us to pay for an "Essentials" license for three physical servers when we only have one? And I won't even get started on the pricing...$995! You can't be serious.

[Rumple]

So, thought this thread, we've seen a couple of things being pointed out:

#1 - The VMWare and their marketers have made a few points about ESX FREE that really stretch its abilities in an SMB Market. Weall know the SMB Market does not have tape drives (unless its purchased inside the system they bought. They also don't have Backup software, again unless its purchased inside the server they bought. Having a tape drive and backupexec software for the server you just bought isn't going to help you when running ESX. That leaves an SMB with the very typical situation of USB Drives run in a rotation...again, doesn't help you at all.

ESX Essentials IS geared towards the SMB Market because it provides them with the bare basics of the ability to backup their VM's (and its cheaper then buying 2 complete servers for the SMB office). At the very least it gives the IT guy the ability to plug usb drives into the receptionists or office owner's PC and tell them to swap them out on a daly basis.

#2 - ESX vs Hyper-V - I still consider them to be different products because inspite of everything, Hyper-V still required that main operating system to be running...its not an independant process running with low overhead...therefore its very much like VMWARE Server...with the same capabilities in an SMB Market. Single Server, local Storage, unmanaged switches. Unfortunately without the Essentials product, most Windows Admin's looking after an SMB market are going to have a hard time figuring out how to make RCLI scripts run against ESX server because of inexperience, and or having it setup so the company owner can do something when they are not around.

Ps - you can shutdown esx remotely for ups by using a ups plugged into a desktop and that in tern calls the RCLI script that shuts down the server...and you can probably also have your desktop run RCLI scripts to perform a backup to a share on the desktop with the usb drives plugged in.

Just for clarity, I support many SMB clients, and I am also a consultant who's supported many Enterprise environments and I would not put esx free into any environment because if I am not around its gonig to be a bitch for them to figure out (or me to talk them through it). At least with an Essentials product I can setup a desktop for VC, and show them how to check backups, shutdown the server, restart the backup appliance,etc.

Personally I think esx FREE should not be available to anyone who's not a VCP....at least you have someone with experience with esx, who's at least done the course, and understands the limitations prior to implementation...so these types of threads don't get going.

If someone wants a free hypervisor then they can use Hyper-V or VMWARE Server, and overspend on the server to get the number of VM's they require if needed. Otherwise, they underspend a little on the server and put the money into a hypervisor that can run considerably more Virtual machines...although in reality an SMB, typically only wants 3-4 VM's anyhow on the same vlan, with local storage...

[s1xth]

I just wanted to chime in on this post, hopefully some people in Vmware are reading this. I agree with the reponses from everyone, except on one thing, ESXi Free LOCKED down backup API's. If anyone remembers prior to U4 and vSphere 4 the backup API's were NOT locked and Veeam Backup, ESXexpress worked wonders for the SMB. Now that they are locked and you need to purchase an 'essentials' license. I understand that Vmware wanted to make money, and saw their 'software' counterparts making money off a free product that somehow breaks the ESXi free license, I call that BS. If you want your product to go up against HyperV you NEED to unlock the backup API's.

I was at Vmworld and I brought this up to Veeam, Vizioncore, Symantec and Vmware themselves, I got the same answer from EVERYONE "yeah we know, it sucks, it really does". Even Vmware engineers said it sucks, and it was "out of their hands".

Vmware keeps talking about SMB customers, well they to start to understand that not every company is an Enterprise customer, and the SMB doesnt have a gigantic budget. I am not bashing them, I am happy Vmware customer, but I see this as their biggest WEAKNESS is the current status of ESXi. (heck dont even get me started on how I heard MANY vmware engineers telling other attendees that ESX isnt going anywhere, and all the labs were based off ESX, when all I hear is ESXi is the future).

Just my .02 cents....still love their products, just need to draw a finer line on some things, and clean some areas up for the SMB.

[williambishop]

This might sound rude, but I really don't care about my credibility with you. I don't mean it to be rude, but here's the problem. Anyone who's built large infrastructures and most of the people here in fact, didn't just "hope" vmware was the right product, we tested. Then tested, then tested some more. Most enterprise groups have incredibly tight and thorough requirements to show "just cause"(you absolutely wouldn't believe the hassle it is to get a project confirmed, then completed, nor how many checkpoints there are during the entire process)...

I say hyper V is shoddy, because during what seemed an unendurable amount of testing, hyper-v quickly disappointed everyone when it proved it was not very feature rich, but the workload on it was horrendous. Where I get 60+ desktops on an esx host, on the same one I might get 10 with Hyper-V before we started seeing performance drops--and then the bloody thing would just go tits up at whatever random interval it had decided upon to maximize my annoyance with it. We won't even get into the server testing, where the load was even more abysmal than on desktops. I don't know about you but I don't need my databases or authentication environment going down just whenever it feels like it. Yes, you got it on a three servers, and being an SMB, that might be enough...But it doesn't cut it in most environments. Plus, most companies need no outages, which basically MANDATES that you use ESX with vmotion. Yes, I know you can just pay someone 50,000 dollars a year to move them around for you (a la microsoft), but that's not for most folks. The price of esx and the required feature sets are not expensive at all when you determine the value involved...

All that said, if microsoft produces a better product, I'll be first in line to use it, and I'm no fan of VMware since the true guidance was forced out. They're after the dollar more than the customer now, and a lot of their moves are geared to maximize that dollar, often at the cost of the customers satisfaction. But Hyper-V? Seriously? Not just no, but hell no.

BTW, I've got guest vm's that have been up since they were built years ago, with NO downtime. That's VMWARE.

Lastly, I'm not anywhere above average, and I figured the stuff out, I firmly believe you can to.

[s1xth]

Just wanted to say this, I have 10 ESXi hosts with 6 month up time. (Havent been restarted/rebooted since U4 came out. Rock solid). Now the VM's themselves? Solid, as Windows can be :-). I have a couple Ubuntu/CentOS boxes that run 24/7 at 90-100% CPU never reboot them, ever. Some of my SQL 2005/W2K8 boxes need reboots once a month, but thats Windows. Havent ever had a major issue (knock on wood).

[Anders_Gregerse]

As many you started out with the assumtion that vmware server and ESX was two products alike and as you discovered they were not. ESX does have a more steep learning curve and does miss some of the ease to use features that Vmware server have, but they are two different products. I've not surprised that you find Hyper-V more to your liking, a lot of people will, not because it's good product, but because it run on/with windows, support a vast amount of hardware and have the features that you required. For you it doesn't matter that you can't use live migration or that ESXi has a small footprint and respect to that. You are not alone and many more will join you because of those needs.

I'm still using ESX because it currently cover my need better that Hyper-V R2 (without vmm).

I hope the vmware guys are listening, because this is a typical SMB story that will repeat itself.

[dougclutter]

Anders - I think you've summed this thread up perfectly. I too hope the vmware guys are listening, and that they take steps to address some of the shortcomings of ESXi. That was my major reason for posting...not to request technical assistance...nor to start a heated "holy war". I like VMware, and I want them to succeed. I hope this thread helps in some small way.

[kghammond2009]

After reading this thread, I don't think anyone actually explained why there are design limitations with USB devices.

ESX / ESXi (including the free edition) are designed for Enterprise deployments that include 2+ ESX servers running vMotion.

If you are unaware, vMotion allows you to move a VM from one physical ESX server to another on the fly (Assuming you have a SAN/NAS, NFS, iSCSI, FC). In order for vMotion to work effectively, you need the ESX servers to have similar physical hardware and devices.

I hope my explanation makes sense, maybe someone else can put the technical challenges of presenting USB devices to multiple ESX VM's on seperate servers as well as to the ESX service console.

Expanding on the APC question... If you have two ESXi free servers plugged into one APC device, and you want them to power off through the USB, you only have one USB cable, correct? This simply is not scalable and does not work well in a Enterprise application.

I hope this helps some,

Kevin

[kghammond2009]

Also expanding on Anders point. ESXi is not designed to be a stand alone hypervisor solution such as VMware Server and Hyper-V. It is an interesting question of what VMware is trying to accomplish with the ESXi free edition.

Once you have two ESXi free servers, most of the value of VMware comes within the paid vCenter product, but there is a substantial cost difference in the infrastructure to support a two node ESX cluster versus the infrastructure to support an ESXi free server. For many SMB's this cost difference is prohibitive to expand to a true two node ESX cluster.

It would be interesting if VMware developed an optimized stand alone ESXi free solution that would assume the product cannot be added to vCenter, thus it could treat local devices differently, but I doubt they will ever spend resources developing a product like this for SMB use.

In reality you can accomplish making a ESXi free be a stand alone product already with 3rd party products and scripts if you want to invest the time, but VMware Server and Hyper-V have a much shorter time investment to get a single server up and running fully integrated with a typical SMB Windows environment with backups, etc.

[rrstapleton]

I just downloaded a trial copy of the Work Station Software and tried installing it on 3 seperate machines, you might ask what does this have to do with your issue, well I called their support line and pretty well got the bum's rush, trial copy no support, makes me want to run out and purchase it right away, like ^. I have worked with Rockwell, Schneider Electric as well as Siemens, all in Executive and Technical areas, if we treated customers the way I was by their phone support I would be ashamed.

So I understand your frustration, does anybody have any alternative to VM Ware that works?

[dougclutter]

After reading this thread, I don't think anyone actually explained why there are design limitations with USB devices.

ESX / ESXi (including the free edition) are designed for Enterprise deployments that include 2+ ESX servers running vMotion.

...

Expanding on the APC question... If you have two ESXi free servers plugged into one APC device, and you want them to power off through the USB, you only have one USB cable, correct? This simply is not scalable and does not work well in a Enterprise application.

Kevin - you are absolutely correct in saying that ESXi appears to be designed with the vMotion/HA functionality in mind. Since we currently (in our department), only require a single physical server, we were hoping to start small with a single-server ESXi deployment that we could later grow into a multi-server (paid!) deployment. Given the shortcomings previously discussed, this isn't feasible for us, so our time, effort, and energy will now be expended on getting Hyper-V to work. To paraphrase the poet...knowing how way leads on to way, I doubt that I shall ever come back.

Also, I think you are misunderstanding my intended use of USB devices. I do NOT want to expose them within the VMs....rather I want to utilize them from the underlying ESXi OS. I just want to backup the internal drive (array) and use a UPS to shutdown the ESXi OS and all the VMs.

Best regards, Doug

[dougclutter]

I just downloaded a trial copy of the Work Station Software and tried installing it on 3 seperate machines, you might ask what does this have to do with your issue, well I called their support line and pretty well got the bum's rush, trial copy no support, makes me want to run out and purchase it right away, like ^. I have worked with Rockwell, Schneider Electric as well as Siemens, all in Executive and Technical areas, if we treated customers the way I was by their phone support I would be ashamed.

So I understand your frustration, does anybody have any alternative to VM Ware that works?

Here's something that I know a little about. The VMware Workstation product is great! I don't know what your intended use is....but I have used it for years and I am very, very satisfied with it. In my environment, it is used for development work and customer support. It is wonderful to be able to spin up a number of different OSs at will and run them from a window on my desktop. Also, it has a wonderfully useful snapshot feature that can be used even when the virtual machine is running. Do you need to see how something runs in a memory constrained envirnoment? Easy....just lower the VMs memory allocation and launch it. Need to see if multiple processors work as expected....again an easy setting. Obviously, you will need a beefy machine to support it, but it is worth it.

I can't say anything about their support because I've never used it.

[kghammond2009]

Kevin - you are absolutely correct in saying that ESXi appears to be designed with the vMotion/HA functionality in mind. Since we currently (in our department), only require a single physical server, we were hoping to start small with a single-server ESXi deployment that we could later grow into a multi-server (paid!) deployment. Given the shortcomings previously discussed, this isn't feasible for us, so our time, effort, and energy will now be expended on getting Hyper-V to work. To paraphrase the poet...knowing how way leads on to way, I doubt that I shall ever come back.

Also, I think you are misunderstanding my intended use of USB devices. I do NOT want to expose them within the VMs....rather I want to utilize them from the underlying ESXi OS. I just want to backup the internal drive (array) and use a UPS to shutdown the ESXi OS and all the VMs.

Best regards, Doug

I think I do understand very well what you are trying to accomplish. Let me restate this a little differently.

If you have one APC UPS and you expose it to the service console via USB to shutdown the ESX server. For humor sake, assume this all works. Now you are have enough needs to expand to a second ESX server. You are a small shop and your APC UPS is big enough to power two ESX servers. Great. OK, well your APC UPS only has one USB port. So now, how do you shut down the second ESX server via one USB cable?

Since VMware assumes ESX is designed for multiple server installations, they also assume that if you are going to shut down your ESX server(s) via a UPS system, that system needs to be scalable to shutdown all your ESX servers. Thus a USB interface is not sufficient for the task. Thus, they do not spend resources trying to make a USB interface work for shutting down a ESX server. Does that make better sense?

The same goes for backing up to a USB hard drive. Once again, this may work well in a single server scenario. Now as soon as you have two ESX servers, how do you back up both ESX servers to one USB hard drive on one ESX server? It won't work well and it is not scalable, so VMware doesn't develop a solution for this once again, becasue they are focusing on 2+ scalable ESX solutions. Even for kicks and giggles, assume that you put a USB drive on each ESX server to manage your backups. Well what happens when a VM moves from one serverA to ServerB? Do the backups move with it to the USB drive on serverB? Do the old backups on serverA get deleted? If they get deleted, when do they get deleted? Etc, etc.

To summarize, you asked the question why VMware does not support USB backup and shutdown solutions on ESXi. I am answering your question, they do not support USB solutions because they do not scale well beyond one ESX server and they design ESX for 2+ ESX server installations.

If you want to use ESXi instead of Hyper-V, you need to think of solutions that will scale beyond one ESX server. This means network based solutions. If you don't want to worry about scalable solutions because you will only ever have one virtualization server, then Hyper-V and VMware Server are better solutions for you and they are designed to fit your needs.

Simply put, ESXi free is not designed to be used the way you want to use it.

Kevin

[dougclutter]

Thanks for the clarification Kevin. I would go one step further and say that ESXi is intended for use with 2+ physical servers AND the NAS/SAN necessary for backup AND a high end UPS for power control. And, as you and others have pointed out, by the time you get to this level of complexity, you're probably not going to be satisfied with the ESXi (free edition)...you're going to want some vCenter/Infrastructure software juice to pull it all together.

[Dr_Virt]

Some other points regarding ESXi. Please understand that it has a limited scope that greatly appeals to many. Below are a few ways we have deployed ESXi:

1) Dev/Lab systems - ESXi is great for turning midlevel servers in to development and testing playgrounds.

2) Remote/Branch Office IT in a Box - We have remote sites that need Active Directory, File/Print Services, and SCCM options. 1 server, ESXi, and we ship their IT service to them with a simple 1,2,3 setup guide.

3) Prototyping - Similiar to 1 above. Build the environment, validate, and then migrate to production.

We find new uses for it all the time.