Nick Howell's Blog

Interesting night with vMotion...

that1guynick — Sat, 12 Sep 2009 20:43:58 GMT

So just a quick note here. We are staging our NetApp + Nexus 10GbE upgrade/migration, and was going through the motions, moving VM's off in order to add the 10GbE NIC's, and start running into Host CPU incompatability errors. What I wanted to do here was basically post everything we discovered, and tried (unsuccessfully), to get it to live vMotion.

Symptoms: live vMotion throws an error about Host CPU incompatability at 'ecx' however, cold migration goes without a hitch. This was unacceptable however, because we promised NDU's to the new NetApp.

The digging begins. First things first, I KNOW that Intel-VT is enabled in the BIOS, because we would not be able to host x64 VM's were it not. And we have several, and they have lived on every host. But you know what? I'll check anyway. I was right. Intel-VT is enabled.

At the suggestion of Rick Scherer, I tried to create a brand new EVC-enabled cluster. Apparently the X7350 3GHz Xeon's don't support this, as I was unable to add the first host to this new cluster.

Back to the drawing board. OK, let's start looking into the NX-bit bypass, and dig through the vmx files.

Tried to power down a non-critical VM, change the CPUID Mask to "Hide the NX/XD flag from guest." Powered it back up, and tried to vMotion again....failed.

Now I'm really getting frustrated, and digging into the vmx files with a fine tooth comb. Then I stumbled across something......odd....

tools.syncTime = "FALSE"
uuid.location = "56 4d 65 df 6b c0 4c 44-7f ee b8 89 f7 af 4d 66"
sched.mem.max = "1024"
sched.swap.derivedName = "/vmfs/volumes/cd9c6686-f1131816/Web4-01/Web4-01-99e2ad9d.vswp"
hostCPUID.0 = "0000000a756e65476c65746e49656e69"
guestCPUID.0 = "0000000a756e65476c65746e49656e69"
userCPUID.0 = "0000000a756e65476c65746e49656e69"
hostCPUID.1 = "000006fb000408000004e3bdbfebfbff"
guestCPUID.1 = "000006fb00010800800022010febbbff"
userCPUID.1 = "000006fb000408000004e3bdbfebfbff"
hostCPUID.80000001 = "00000000000000000000000120000800"
guestCPUID.80000001 = "00000000000000000000000120000800"
userCPUID.80000001 = "00000000000000000000000120000800"
evcCompatibilityMode = "FALSE"

I actually have no idea what this truly means, or where it comes from, but on just about every other 1vCPU VM I inspect, I see like values across the board. I did make a copy and try changing it to match the other values, and the migration still failed.

The tech I worked with last night was supposidly allocating this to a Bug ID of 380853 (which I can't access...partners?!) and submitting a set of exported logs to the engineering team.

I don't have time to sit around and wait, so I'm proactively moving things off. If they need to be quickly powered-down to be moved, I'm clearing it with the responsible party, and plowing through it. Certainly not how I would like to go about it, but hey, bigger fish to fry.

A couple of things to note. There are (3) HP DL580 G5 servers in this cluster. They're identical. Same box, same (4) X7350's per host (+OR ARE THEY?!+), same 128GB RAM per host. Each of them had different BIOS revisions, so that is being brought up to the latest (May 2009). Also, I decided to check out the /proc/vmware/cpuinfo on each host.

Dammit, Intel. This is annoying. Please stop doing this. See below:

ESX host 1. This is the oldest one, and set the stage for the other two with the X7350's. We were specific about getting identical hardware so we didnt have to deal with performance degradations and "workarounds" such as EVC.

cat /proc/vmware/cpuinfo

	ESX1	ESX2	ESX3
pcpu	00	00	00
family	06	06	06
model	15	15	15
type	00	00	00
stepping	11	11	11
tscKhz	2933332	2933434	2933434
processorKhz	2933332	2933434	2933434
busKhz	266666	266675	266675
name	GenuineIntel	GenuineIntel	GenuineIntel
ebx	0x00040800	0x00040800	0x00040800
ecxFeat	0x0004e3bd	0x0004e3bd	0x0004e3bd
edxFeat	0xbfebfbff	0xbfebfbff	0xbfebfbff
initApic	0x00000000	0x00000000	0x00000000
apicID	0x00000000	0x00000000	0x00000000

So I sent a big long email draft to my local Intel field engineer and we'll see what the deal is. For now, I'm putting a smile back on my face and headed to the datacenter to turnup some 10GbE!

-Nick

Oracle on VMware - One Man's battle against a titan.

that1guynick — Wed, 09 Sep 2009 22:56:47 GMT

Is it just me, or is there still a lot of taboo and misinformation in our industry about VMware, and what it brings to the table, and what it's capable of? As big of keywords as 'virtualization' and 'consolidation' have become, I still get the eerie doubt from a lot of peers about just that. The minute you say, "Hey, why don't we just VM it!" there's a hush that falls over the room as everyone starts throwing random reasons out why they think it wouldn't work. And it really comes down to a lack of education (adoption might be a better word..."the norm" so to speak) throughout the industry. You sales folks have GOT to stop feeding us high-level powerpoint slides with these monotone readings over the top of them. Put the slides away and just TALK SHOP WITH US!

As much as VMware and other big storage players have done for the industry, especially in the last few years, the sales force is still the same as it was10-20 years ago. You're still just using keywords to get our attention, but not really telling us how to use your product to our advantage. Even when we've had vendors and resellers come INTO our office to give 'demonstrations,' it's still nothing more than them plugging their laptop into our big tv, and showing us the same powerpoint they've shown to 100 other potentials. Eventually, the techie's get to take over, and the lonely PSE they let out of the closet for a day to come along finally gets to converse with the company's IT team and we get down to business, if we're not nodding off from watching the slides click by while the droning of the sales rep reading them verbatim seems more like a lullaby.

But that's not what I'm writing about here. This isn't a rant. This was a precursor into a battle I didn't know I was getting myself into.

A little backstory...

Our current environment involves a couple of Sun 18U refrigerators with a fiber-channel Hitachi brick. Pretty cookie-cutter for a high-end, mission-critical Oracle database. Layer on a bunch of Oracle database, application, and DR software as well as all the manual processes that go along with them, and you've got a rough idea of what we're starting with.

Before this upgrade process started, we had finally gone down the consolidated storage path, joined the NetApp country club (as my boss likes to call it), and were using it for mainly shared network drives, user data, and VMware. The NetApp came into play because we also wanted to consolidate the Oracle storage onto the NetApp and off of the standalone brick.

There were a few scenarios that were thrown out in the beginning. There were the new Niagara-based Sun T2000's, which were actually RECOMMENDED to us, and I'm still convinced to this day that this is 100% our fault for not doing the homework we should have done pre-purchase. In case you're not familiar, the Niagara chipset was never really designed to run databases. They turned out to be a huge flop during our performance testing until the right engineer/Sun-guru came along and told us about the whole Niagara story.

It was also during this phase that we started parting the seas of IT between the DBA's and the Operations crowd. I had been doing some serious reading into NFS. I have had a lot of success using NFS + VMware for my datastores. We actually started out using iSCSI + VMware, but I added a new large datastore via NFS and the performance was actually better. And the allure of resizing entire datastores on the fly was enough to make me jump in with both feet. So, the DBA's would never conceive of running anything but fiber-channel to high-end storage. So our initial failed phase of testing with the T2000's was largely attributed to using NFS, because it was the likely scapegoat. It wasn't until we hooked the T2000's up to fiber-channel to the same NetApp configuration that we noticed the performance was still terribad, and only marginally better than our current 5+ year old Sun refrigerators.

After some serious digging, NetApp and Sun engineer involvement, and theorycrafting (more like "ok now what the hell are we supposed to do?"), I started planting the VMware bug.

"No way."
"There's no way VMware can handle the workload."
"Are you kidding me?"
etc.

So, I sat quietly as we went on to the next Sun solution. This time, we did some serious homework, in looking at the Sun M4000. Different architecture, built to run high-end databases, etc etc etc.

At the same time, in another corner of the datacenter, I proactively on my own, built a linux VM. Working with one of the DBA's, we installed a copy of 11gR1, and migrated a copy of our production database to it. We divi'ed up some volumes on the NetApp to host the data, and mounted those via NFS directly to the VM. At the same time, we also configured Oracle's new D-NFS client, to make direct connection from the database to the storage, bypassing the kernel layer.

The results....well, they were nothing short of shocking. The linux VM running 11g completely outworked the M4000 with a 1:1 hardware configuration. Mind you, this was only on a 1GbE connection, single path, to a NetApp filer that was already heavily taxed hosting tons of CIFS shares, and NFS ops to VMware. (Testing was conducted using SwingBench and Real Application Testing)

So, where do we go from here? We had a meeting. Everyone was excited/shocked/appalled by the results (+"How the hell did a little VM run circles around the boxes that launch the space shuttle?!"+). We all threw our hands in the table, and said "GO VMWARE!" and off we went, with a purpose in mind to virtualize Oracle.

We are currently in the process of building out a production-level environment, and once I have cleared it to discuss it more, look for a subsequent post related to the final architecture.

We are definitely excited. Not so much for the consolidation VMware brings to the table, but the agility and easy resilience things like vMotion, HA, and eventually FT, bring to the table, especially for big tier 1 apps such as Oracle OLTP databases.

Stay tuned.

-Nick

Upgrading to vSphere 4 - The Experience (Part 1)

that1guynick — Wed, 08 Jul 2009 06:03:48 GMT

The day had arrived.

After many marketing documents, press releases, and speculative forum posts, we were all waiting with baited breath to see just how VMware could make one of the most revolutionary products to ever hit the market even better.

I attended the vSphere 4 launch tour. I watched the powerpoint slides scroll through with all of the piles of new features. I was sold. I researched a bit, and couldn't find any articles negating the ability to upgrade. Asked all my questions beforehand...

"Can ESX 3.x and 4 hosts co-exist in the same cluster?"
"Is there anything retroactive about running a non-upgraded-VM on a ESX4 host?"
"Can I still join ESX4 hosts to VirtualCenter 2.5?"

This accurately defined my upgrade path, and this is basically a synopsis of how that process went down. There were some surprises, but none of them were bad.

Step 1. Upgrade VirtualCenter 2.5.x to vCenter 4.0

"Wait...what?" This shocked me a bit at first, because the answers I got from the launch tour told me that I could join ESX4 hosts to VirtualCenter 2.5! Why do I need to upgrade VC to vCenter4 first?

And then the golden nugget was dropped on me.

It just so happened that we had coincidentally just purchased an additional host to add to the cluster. I had waited to do the install so that I could truly test the whole "adding an ESX4 host to VC 2.5" thing.

Clean install went flawlessly. Things to note: You can now preload custom drivers DURING the install. Got a weird NIC or HBA? Oddball SCSI card? Here's your chance. No need to go in post-install and do your configuring there. You can do it before the installer launches into loading packages. COOL! Other than that, it was just like installing any of the other versions, aside from a few graphics changes.

First host install complete. Once it's loaded, I connect to it directly via the VIC, to do some simple configuring.

ROADBLOCK!

New client install time! The first time you connect to ESX4 or vCenter4, you're prompted to install the new client. Schnazzy new dashboard, and I see the potential here for lots of vCenter plugins. (more on that in another post)

Client updated, onward! Connect to the host, configure my storage and networking to be identical to the other hosts in the cluster. Easy-peezy, no problems there. Right-click on the cluster, add host, go through the motions, and in a matter of minutes, I now have an HA/DRS-activated ESX4 host in my VC 2.5/ESX3.5.x cluster. Flawless.

WOW! This is great! So I kick one of the 3.5 hosts in Maintenance Mode, go to lunch while the VM's migrate off, come back and pop in the ESX cd. Boot up....

"Existing ESX hosts can no longer be upgraded via this method. Please upgrade through vCenter."

Hmm...a bit of a quick digging made me realize that there was not a choice in upgrade paths anymore. So I read back over the upgrade documentation once again. AHHHHH! So THIS is why they want you to upgrade VC to vCenter4 first!

Apparently, one of the big changes in vCenter4 was making Update Manager NOT USELESS! "Alright....let's give this thing a shot."

So, ditching the idea of upgrading hosts for the moment, I switched gears over to VC. Downloaded the .zip pack and copied them to my existing VC server (which is also a VM). Ran the executable, it found my existing VC installation. Great! Then the prompts about the database come up. So I put in my db user/pass, and the name of the database, ERROR! Ut oh. It prompts me that the vc user must have full 'sa' rights to the MSDB database. I was confused for a bit because I didn't read the fine print closely enough. I thought it was referring to my actual VC database. So, I double-checked that the vc user had owner privs to the VC database, went back and tried again. ERROR! It took me a couple of tries to realize I wasn't paying enough attention, and that it said 'MSDB' database, and not my actual 'VC' database.

Problem solved, moving on. Upgraders, just make sure you're doing the MSDB when you upgrade to vCenter, not your actual VC database.

Licked that, install finished fine, and IIRC, it prompted me to remove the old model licensing server. Note, if you're upgrading all of your vCenter and hosts to v4, there's no need to keep it, but it will not interfere with anything. v4 doesn't even reference it, and it's good to have around in case you need to do something in 3.5 (i.e. test/dev environments, perhaps?)

Restarted the VC VM (sorry, "vCenter" now. Did Steve Jobs start working for VMware under the table? vCenter/vSphere...) and connected to it. Nice dashboard! I like it! Love the new licensing model as well. I went straight to the licensing site (licensing.vmware.com) and consolidated all of my individual licenses into ONE KEY! This was amazing! Gone are the days of managing dozens of license keys! And it actually WORKS now! Also, little did I know, but VMware had also already upgraded my license keys to be vSphere4 keys.

So, VC is upgraded to vCenter 4. Licenses have been properly allocated across all hosts with ONE key, and we're ready to rock.

In my next writeup, I'll go over upgrading your individual hosts.

-Nick