My vCenter has been loading slow for as long as I can remember.   It finally got annoying enough to do something about.  "Certainly, they've made this better by now."

So I got a fresh drink, dug in, and starting searching for blog posts and articles on how to trick out vCenter performance, slow loading, etc.  I stumbled across this KB article from VMware titled:

Defragmenting VirtualCenter performance data indexes on a MS-SQL database

Overview of the KB:

Fragmentation of indexes occurs when the logical order of pages is different than the physical order on the disk. In VirtualCenter fragmentation occurs most noticeably due to the statistics collection and consolidation.    When the indexes are excessively fragmented, performance of queries to the VirtualCenter database is slow.

SWEET!  This could be it!   So I scrolled down to the bottom of the page for the section on VirtualCenter 4.x on SQL 2005, and ran the queries to check for fragmentation....

Use VirtualCenter

go

dbcc showcontig (VPX_HIST_STAT1)

dbcc showcontig (VPX_HIST_STAT2)

dbcc showcontig (VPX_HIST_STAT3)

dbcc showcontig (VPX_HIST_STAT4)

go

What it revealed was SHOCKING...

Now, that's a lot of numbers, but look at the one's I've highlighted.  Per the KB article:

The key pieces of information to determine fragmentation are Scan Density and Logical Scan Fragmentation. In the example, scan density is 27.49%. In an ideal environment the closer this number is to 100% the better the database performs. For Logical Scan Fragmentation, the example shows 38.79%. The lower the percentage the better the system performs.

Holy crap!  My scan density was in the teens when it's supposed to be closer to 100%, and my fragmentation was 99% when it supposed to be closer to 0%!!!

Again, per the KB article, you can run the following optimization routines to "defrag" the indexes:

dbcc indexdefrag ('<database>', 'VPX_HIST_STAT1', 'PK_VPX_HIST_STAT1')

dbcc indexdefrag ('<database>', 'VPX_HIST_STAT2', 'PK_VPX_HIST_STAT2')

dbcc indexdefrag ('<database>', 'VPX_HIST_STAT3', 'PK_VPX_HIST_STAT3')

dbcc indexdefrag ('<database>', 'VPX_HIST_STAT4', 'PK_VPX_HIST_STAT4')

go

So off we went...

Crazy.  I had no idea this was going on in my database.  Maybe it's one of those set-it-and-forget-it things that typical admins don't pay that much attention to, because just about every mgmt app these days has a SQL DB backend requirement.

Here's what the numbers looked like after the defrag.  Only took ~5 minutes. Logical fragmentation went from 99% to 18%

While these aren't as good of results as I expected to see (notice some of the other tables are still 50% fragmented) I'm convinced that there are more methods to trick out vCenter so that it doesn't load so slow.  Every time I click on a different item in the column on the left, I want it to pop.  It's essentially a webpage.  It can be done.

I'm on a mission...

-Nick

This year was certainly an unprecedented, and albeit unEXPECTedly, lucrative conference for me. Whoever says they don't believe in luck, chance, and coincidence should re-evaluate their belief systems.

First:

As I mention in a prior post, my name was drawn from a hat to win the Ultimate vSphere Reference Library. I met up with @gregwstuart today to conduct an interview and to collect my winnings. I will post the video when Greg sends it to me.

Here's a photo:

(not pictured: Windows 7, vDestination mousepad)

Big thanks to all of the authors and providers involved, and to @gregwstuart and @vDestination for putting together such a wonderful contest.

Follow him: @gregwstuart

Blog: vDestination.com

Second:

As if that were not enough, I then won an iPad from NetApp simply by wearing their hat all week, and being COINCIDENTALLY LUCKY enough to bump into Keith Aasen, a camera man, and a beautiful woman carrying a microphone. They all turned to me and I thought maybe they just wanted to interview me about the conference. The Keith asks, "So have you won anything from NetApp yet?" While I was trying to remember exactly what they were giving away, Keith was holding this:

OMGOMGOMGOMGOMGOMGOMGOMGOMGOMGOMGOMG......breath......OMGOMGOMG!

Youtube video here @ 2:25

Third:

And we're not done yet. For the past two years, Kingston has been bringing a professional Guitar Hero player to their booth. Her name is Annie Leung and is known as the "best female Guitar Hero Player in the world." In the booth, you can play against her for a chance to win a 64GB SSD hard drive from Kingston. I tried to keep up on guitar, but just couldn't hang, even though I always play on Expert. So, I thought I would try something unique, and challenge her to a hard Bass track that I had really gotten good at: "2 Minutes to Midnight" by Iron Maiden.

While ultimately, she beat me, I definitely gave her a run for her money. In the end, I received a 64GB SSD hard drive for my efforts.

Annie, thanks for an awesome match. Come find me on XBL for a match anytime. XBL name: krackmnky

You can follow Annie on Twitter and visit her YouTube channel.

This is all above and beyond a year's worth of tshirts, new backpack, badass PowerCLI reference poster, several cool ballcaps, and a lifetime's worth of new friends.

What a conference. So rewarding in every way, shape, and form. And I am eternally grateful for having been a part of such a milestone of a conference.

-Nick

This morning, I sat down at my computer with a cup of coffee and start perusing thru my Inbox, and happened across this email from Greg Stuart of vDestination:

Congrats Nick, you are the lucky winner of the vDestination.com Ultimate vSphere Reference Library.  I just did a blog post with the video of the drawing I did last night.  Are you going to VMworld?  If not, please supply me with your shipping address so I can ship these great prizes to you.  Some of the prizes I'm picking up from a couple of the authors at VMworld, so I won't be shipping it until I get back from VMworld.  Again, congrats, enjoy these awesome prizes. 

See the video of the drawing here:  http://vdestination.com/2010/08/27/we-have-a-winner/

I'm honestly floored!  I never win anything (though I said this last year when winning an iPod Touch @ VMworld), but this is truly something useful, and I hope that I can make the best use of it by expanding my knowledge, finally getting my VCP and working towards VCDX, and continuing to be an evangelist for VMware virtualization.

In case you're interested, here's my submission to the contest:

After virtualizing nearly everything in our environment over the past 4 years, there hasn’t been a lot of time to go back and revisit what we could have done better/differently. By re-educating myself, I hope to improve my environment even more, by optimizing administrative tasks with automation and PowerCLI, building reporting workflows, as well as sharing with my fellow co-workers bringing them up to speed on virtualization. Everybody has a different way of doing things, and the little bit of Scott’s book that I have read has helped me tremendously. I hope to develop some standards around how we build and manage our environment, based on the extensive knowledge of these guru’s!

Thanks again to everyone involved, and to all of you who donated something to the contest.  I promise to truly make the best use out of all of it that I can.

-Nick

I thought I would sit down and make some notes for myself with regards to what I was looking forward to most at VMworld 2010, and then revisit them AFTER the show to see how rewarding it was, what I gained above and beyond expectation, and if all of my hopes and concerns were addressed.

Most of all, I'm looking forward to the Technology Exchange.  Granted, all of us hang out on Twitter all day every day, but it's different when you put hundreds of vendors in the same big hall that all just want to geek-out about virtualization with you.  I can't express enough to vendors how rewarding that is to us end-users and engineers out in the field.

I'm looking forward to the keynote addresses of the major players in the industry, including Paul Maritz and his vision of VMware going forward.  These are also exciting times where new things get revealed, and then we get to run to the Tech Exchange and geek out AGAIN about the next best thing that's coming!  

It's a whirlwind of a good time.  From the informative sessions, tweetups & social media, fun and instructional labs, and wonderful after-party's hosted by our favorite vendors and resellers...not to mention the fact that we're in one of the coolest cities in the world...It's the best money that a company can spend on their engineers.

What am I looking forward to, content-wise:

*New books at the bookstore.  Need some new reading material, and there's lots of new books out!

*NEW BACKPACK!  I can finally retire the neon green for some gunmetal grey hotness!

*Optimization of Storage & Backups & DR.  The never-dying topic of DR...

*NetApp NTAP 8.1 and the future of virtualized storage

*Cool new plugins and features for doing all of the things INSIDE of vCenter

*Last year, I met a bunch of wonderful people and hope to have more conversations with vendors and give direct feedback to problems I have faced.

*Meeting more of my tweeps and getting some face-to-face introductions!

*Promote #NAS4LIFE with proliferation of NFS-based storage!

*Maybe a free iPad or two from contests!

Looking forward to seeing you all in a week!

-Nick

Snapshot. Snapvault. Snapmanager. SnapLock. SnapDrive. SnapMirror.

Are you screaming and confused yet?

Don't worry. Most of their new customers are, as was I, until about a couple of years or so ago when I really took the time to dive in with some SE's and try to understand the logic around it all.

Since I still hear from new and/or potential customers frequently, asking me to help them understand what all of this "Snap" stuff is about, I thought I would take the time to write it all out here.

What I’m not going to do is regurgitate a bunch of the circles and ABC diagrams that you can find on the NOW site or in a typical google search, because that can often times compound the confusion.  We’re going to strictly talk logic, vocabulary, and understanding.  So you other NetApp guys out there, please understand this is more personal theory than “inode level 2” type stuff.

So sit back, grab a coffee, open your mind, and let's get going!

Snapshot.

NetApp defines a Snapshot as:  "a read-only, space-efficient, point-in-time image of the data in a volume or aggregate. It is only a picture of the file system and does not contain the actual contents of data files."

Wait. What do you mean it doesn't contain data?!  What's the point otherwise!?

OK, here's my version:  Think of a snapshot as a differential backup from the last time you took a full backup.  What happens during a differential backup? Only the parts/files that have changed since your last backup get backed up.  Similar concept, only we're talking blocks instead of files. Your live volume will be the equivalent of your full backup, and every time you take a snapshot, it’s another differential copy.  This isn’t a 1-to-1 comparison of the two minds of thought, but it’s an easy way for new folks to understand.

All those words at the top of this post that start with Snap are all based on the Snapshot, and the concepts around it.  Keep that in mind as we talked about the other big Snap words.

So let’s continue on with what is likely the most commonly used…

Snapmirror.

Snapmirror is exactly what you think it would be, just as it sounds.  It is a mirror of a dataset between two different NetApp filers.  While there are different levels and utilizations of Snapmirror, I don’t want to go into it here, because that’s not what we’re talking about.

How are Snapshots and Snapmirror related? That’s what I want to cover here, because everything is based around the Snapshot.

When you first build a Snapmirror relationship of a volume between two filers, this relationship must be “initialized.”  Initializing the Snapmirror is a fancy way of saying, “I need to copy all the data from the source to the destination, before I can actually start syncing the data.” How does it do this effectively?  You guessed it.  Snapshot.  A snapshot is taken at the time of initialization, and this is called your “baseline.” Contents of this snapshot are completely sync’ed between the two filers, including any and all other snapshots you’ve taken on the source.  Since snapshots of a volume are stored within said volume, and you’re making a 1:1 mirror copy of said volume, all the other snapshots from the original source volume are coming along too.  So when you go to snap list <vol name> or Filer > Volume > Snapshots > Manage in FilerView, you will see all of you scheduled snapshots, as well as a new one (“baseline”), and all of these will exist for that volume on both filers hosting the Snapmirror relationship.

Make sense?  If not, ask away in comments!

"But Nick, I don’t want to keep a constant 1:1 full copy sync’ed of all of my volumes on two controllers! I just want a backup mechanism to be able to store MORE snapshots for longer periods of time!"

Ah-HA!  Enter SnapVault.

SnapVault is the disk-to-disk “backup” version of SnapMirror.  Where SnapMirror is the 1:1 scheduled sync’ing of a volume, SnapVault is the scheduled “backup” of a volume, only copying changed blocks in…you guessed it…Snapshots (noticing a trend here?)…from the source to the destination.

SnapVault systems are made up of two major components:

SnapVault Primary’s:  These are you source systems containing volumes/datasets that you want to backup.

SnapVault Secondary’s:  These are the destination system(s) where you long-term archive as many iterations of a dataset as you have available disk to store them on.

To make it easier to understand, think of it as a hub-and-spoke topology.  In the simplest of configurations, your SV_Secondary (destination) filer is the hub of the wheel, and each SV_Primary (source) represents a single spoke, connecting back to the hub.  You can SnapVault many primary systems to a single secondary system, again, for as much disk as you have available on the secondary to store data. For every controller (2 on clustered systems) you must have a SV_Primary license if you intend to use SnapVault to backup volumes on that controller.  However, you will only need the single SV_Secondary license, unless you begin to expand your SnapVault configuration out even more.

SnapVault also gives you the ability to centrally control all snapshot schedules among all of your filers across your entire infrastructure, to maintain naming conventions, standardize schedules, etc.  This gets even moreso centralized with NetApp’s other software product, known as Protection Manager.  But, again, a little out-of-scope of this writing.

Before we get to SnapManager products and SnapDrive, I wanted to cover one that is most-often used in Legal, Hospital/Medical, and government agencies, which require extremely long-term archiving of WORM data.  In case you didn’t know, WORM stands for Write-Once-Read-Many.  It is a way of locking down change access to data completely (even from admins) but still allowing to be accessed in a read-only fashion for reporting, auditing, compliance, and legal purposes.

SnapLock.

SnapLock is NetApp’s way of locking down a dataset in this fashion.  It is almost always used in conjunction with a 3rd party application that is used to manage archived data.  Similar comparisons to this are things like EMC’s DiskXtender product, which uses UDO laser discs as WORM media.  But, this is still cumbersome, as it requires you to manage laserdiscs, and protect them if they ever have to be moved, removed, or destroyed.

SnapLock is still used in the same fashion, only you’re locking this data down onto your Unified shared storage, and this can be mirrored, snapped, or vaulted to other controllers, and will still maintain its SnapLock status.

Honestly, this is the one I have the least experience with, but being a medical company, we are looking at ways to improve our current WORM solution, and SnapLock is definitely on the radar for us in the next couple of years.

SnapDrive and SnapManagers

These two are not directly related to Snapshots, but they affect the ways and processes that Snapshots are taken.  SnapDrive and SnapManager products are middle-men.  They are communication applications that allow applications, operating systems, and NetApp storage systems to all speak the same language and accomplish all of your goals for you.

NetApp has a SnapManager app out there for just about every tier1 application on the marketplace.

Oracle (SMO)

Exchange (SME)

SQL (SM-SQL)

VMware (SMVI)

…and more.

I’m going to pick the one I’m the most familiar with (Oracle) to give some direct steps and examples of how these two coordinate among the App, OS, and Storage system to give you app-level-consistent snapshots of your apps and databases.

SnapDrive is the “agent” of sorts, that gets installed locally on the host of whatever application/OS/DB you’re wanting to take snapshots of.  He will tell the OS and local host disks the correct instructions when the NetApp controller makes calls to it.

SnapManager application software also gets installed on your host, but it is geared more as an agent for the application/db running on said host.  He will tell the Application itself the correct instructions when the NetApp controller makes calls to it.

These two have to work in conjunction with each other, as well as with the OS/application/DB/NetApp, in order to give you consistent backups.

So here we go:  The layman’s language of how SnapManager takes a snapshot of a production Oracle database.

Let’s assume you want to snapshot an Oracle database once per day at midnight.  Oracle databases need to be put into “hot backup mode” in order for the snapshot to be worth anything.  You schedule SMO to take this snapshot of this database named ORACLE on a host named SERVER1 every night at midnight.  The data files for this db are stored on FILER1.

It goes down a little something like this:

SMO> Hey, FILER1, my schedule says it’s time to snapshot ORACLE database on SERVER1.

FILER1> Hmm, ok.  Let me check with SERVER1 and make sure SnapDrive isn’t busy, because he’ll have to pause the mount points to the OS while we do this thing.

SnapDrive> Hey! SERVER1 said you needed me. Yea, I’m ready when you guys are!

SMO> OK, hold on, let me put the database in hotbackup mode. (alter database *) OK, done!

FILER1> OK, SnapDrive, I’m gonna snapshot, you ready on those mount points?

SnapDrive> Ready when you are!

SMO> Go go!

(roughly 60 seconds per 100GB is typically what I see here)

FILER1>  OK, snapshots completed, SMO.  Name them whatever you like and re-open the database when you’re ready!

SMO> Done and done. (alter database open *)  We’re done!  See ya tomorrow night!

So, while SnapManager and SnapDrive don’t really have anything to do with the snapshots themselves (technically…let’s not split hairs here), they are pivotally important to the communications between all systems involved.  And while I didn’t understand the product(s) initially, I have certainly fallen in love with them, and would love to be involved in the “how to make them better” side of things.

Each of the tier1 app SnapManager products must be run and configured autonomously of each other, but there is unconfirmed rumor abounding about a “unified view” version coming soon where you will get a MMC-style version that will encapsulate all SnapManager products into one screen, giving you a single pane of glass to manage all of your snapshotting routines for all of your tier1 apps.

I hope this will serve as a good primer for you newer NetApp customers out there, and help clear up some of the confusion about what each of the Snap products does and how they interact with other.

At the end of the day, spend most of your time understanding what a Snapshot is, but more importantly what it isn’t.  Once you have that down-pat, the rest just comes along naturally.

Is it just me, or is there still a lot of taboo and misinformation in our industry about VMware, and what it brings to the table, and what it's capable of?  As big of keywords as 'virtualization' and 'consolidation' have become, I still get the eerie doubt from a lot of peers about just that.  The minute you say, "Hey, why don't we just VM it!" there's a hush that falls over the room as everyone starts throwing random reasons out why they think it wouldn't work.  And it really comes down to a lack of education (adoption might be a better word..."the norm" so to speak) throughout the industry.  You sales folks have GOT to stop feeding us high-level powerpoint slides with these monotone readings over the top of them.  Put the slides away and just TALK SHOP WITH US!

As much as VMware and other big http://communities.vmware.com/blogs/nickhowell/2009/09/09/oracle-on-vmware-one-mans-battle-against-a-titan/storage players have done for the industry, especially in the last few years, the sales force is still the same as it was10-20 years ago. You're still just using keywords to get our attention, but not really telling us how to use your product to our advantage.  Even when we've had vendors and resellers come INTO our office to give 'demonstrations,' it's still nothing more than them plugging their laptop into our big tv, and showing us the same powerpoint they've shown to 100 other potentials.  Eventually, the techie's get to take over, and the lonely PSE they let out of the closet for a day to come along finally gets to converse with the company's IT team and we get down to business, if we're not nodding off from watching the slides click by while the droning of the sales rep reading them verbatim seems more like a lullaby.

But that's not what I'm writing about here.  This isn't a rant.  This was a precursor into a battle I didn't know I was getting myself into.

A little backstory...

Our current environment involves a couple of Sun 18U refrigerators with a fiber-channel Hitachi brick. Pretty cookie-cutter for a high-end, mission-critical Oracle database. Layer on a bunch of Oracle database, application, and DR software as well as all the manual processes that go along with them, and you've got a rough idea of what we're starting with.

Before this upgrade process started, we had finally gone down the consolidated storage path, joined the NetApp country club (as my boss likes to call it), and were using it for mainly shared network drives, user data, and VMware.  The NetApp came into play because we also wanted to consolidate the Oracle storage onto the NetApp and off of the standalone brick.

There were a few scenarios that were thrown out in the beginning.  There were the new Niagara-based Sun T2000's, which were actually RECOMMENDED to us, and I'm still convinced to this day that this is 100% our fault for not doing the homework we should have done pre-purchase.   In case you're not familiar, the Niagara chipset was never really designed to run databases.  They turned out to be a huge flop during our performance testing until the right engineer/Sun-guru came along and told us about the whole Niagara story.

It was also during this phase that we started parting the seas of IT between the DBA's and the Operations crowd.  I had been doing some serious reading into NFS.  I have had a lot of success using NFS + VMware for my datastores.  We actually started out using iSCSI + VMware, but I added a new large datastore via NFS and the performance was actually better.  And the allure of resizing entire datastores on the fly was enough to make me jump in with both feet.    So, the DBA's would never conceive of running anything but fiber-channel to high-end storage.  So our initial failed phase of testing with the T2000's was largely attributed to using NFS, because it was the likely scapegoat.  It wasn't until we hooked the T2000's up to fiber-channel to the same NetApp configuration that we noticed the performance was still terribad, and only marginally better than our current 5+ year old Sun refrigerators.

After some serious digging, NetApp and Sun engineer involvement, and theorycrafting (more like "ok now what the hell are we supposed to do?"), I started planting the VMware bug.

"No way."

"There's no way VMware can handle the workload."

"Are you kidding me?"

etc.

So, I sat quietly as we went on to the next Sun solution.  This time, we did some serious homework, in looking at the Sun M4000.  Different architecture, built to run high-end databases, etc etc etc.

At the same time, in another corner of the datacenter, I proactively on my own, built a linux VM.  Working with one of the DBA's, we installed a copy of 11gR1, and migrated a copy of our production database to it.  We divi'ed up some volumes on the NetApp to host the data, and mounted those via NFS directly to the VM.   At the same time, we also configured Oracle's new D-NFS client, to make direct connection from the database to the storage, bypassing the kernel layer.

The results....well, they were nothing short of shocking.  The linux VM running 11g completely outworked the M4000 with a 1:1 hardware configuration.  Mind you, this was only on a 1GbE connection, single path, to a NetApp filer that was already heavily taxed hosting tons of CIFS shares, and NFS ops to VMware.  (Testing was conducted using SwingBench and Real Application Testing)

So, where do we go from here?   We had a meeting.  Everyone was excited/shocked/appalled by the results ("How the hell did a little VM run circles around the boxes that launch the space shuttle?!").  We all threw our hands in the table, and said "GO VMWARE!" and off we went, with a purpose in mind to virtualize Oracle.

We are currently in the process of building out a production-level environment, and once I have cleared it to discuss it more, look for a subsequent post related to the final architecture.

We are definitely excited.  Not so much for the consolidation VMware brings to the table, but the agility and easy resilience things like vMotion, HA, and eventually FT, bring to the table, especially for big tier 1 apps such as Oracle OLTP databases.

Stay tuned.

-Nick

In 1999, a friend of mine approached me asking me if I wanted to help him run some cables in an office building for $10/hr.  I said sure.  Being a techy-guy, I started asking questions about what cables were what, where they plugged in, and why they plugged in there...

It was all downhill from there.

Less than 2 years later, I was an MCSE-2000.  For the past 10 years, I've worked in various systems administration positions, primarily focusing on Active Directory & Exchange domains.  We did lots of Exchange migrations and AD 2000-to-2003 migrations earlier in the decade, because well, that's what admins did in those days.

But it wasn't until my current employer and I embraced the idea of virtualization that I truly came into my own. I finally found something I wanted to "specialize" in.

Since then, everything we have done has always had the "should we virtualize it?" tagline added to it, and more recently, it has been, "is there any reason why we can't virtualize it?"

My virtual career path started with ESX 3.0, and NetApp shared storage.  Most of what I discuss here will be our trials and tribulations with migrating a company in the Healthcare sector (non-hospital) to the virtual world of consolidating servers and storage.

I welcome the feedback and any questions anyone and everyone might have, and hope that this not only serves as a track in my progress, but as a way for me to help fellow admins with situations I may have already encountered. 

Nick Howell

Datacenter Engineer