Best way to access 2TB+ volumes

[bluesherpa]

Please tell me if you think this is the best approach. I'm new to ESXi and I'm still learning how to take advantage of it...

I have a server with 8 x 2TB Seagate drives, an Areca 1230 hardware raid controller, and an additional 500GB Seagate hard drive.

I want to get the highest throughput and best performance from the 8 x 2TB drives and the Areca raid controller, RAID 5 support, and the ability to use the drive space within ESXi.

The system is currently configured as followed:

- ESXi is installed on the 500GB hard drive; I'd use a USB flash drive,

but the motherboard doesn't support booting from a USB flash drive.

- The 8 x 2TB drives are assigned to one raid set and, inside the raid set, 8 x 2TB raid 5 volumes have been created.

- Inside of ESXi, the first raid volume created a datastore and then I added the remaining volumes as extents.

- Gigabit ethernet (full duplex) - No jumbo frames

- 6gb of system ram with a core 2 duo processor

Space from the new datastore was assigned to one of the virtual machines and everything tested well. For example, I was able to copy a 2.4GB file from one directory to another in 24 seconds. That seems pretty good to me, since the drives specs show that they can sustain 95MB/s throughput. I'm in the process of figuring out how to install / use IOMeter to get more accurate test results.

I created a SAMBA share and ran two tests. Copying a file to the server over the network sustained between 50MB/s and 54MB/s. Reading a file from the server also sustained between 50MB/s and 54MB/s. This seems odd for two reasons.

First, I tested the system with Ubuntu and SAMBA (no ESXi) and files copied to the server at 66MB/s. The would imply that ESXi encurs overhead that slows network performance by about 12MB/s to 16MB/s. That seems like a lot and gives me the impression that I've done something wrong.

Second, with ESXi installed, network throughput was between 50MB/s and 54MB/s the entire time. There was no initial burst of high performance and throughput didn't slow down over time. That seems strange. Wouldn't buffers / cache come into play? For example, the Areca raid controller has 1gig of ram. At the very least, shouldn't I expect higher throughput until the cache is full? I've seen this behavior on other NAS drives and I'm a little surprised that I'm not running into it here.

Thanks in advance for any thoughts, feedback, or recommendations you have on any aspect of this - whether it's comments about the raid controller, how it's configured, how to best include the drive space into ESXi, or even opinions on what kind of network throughput I should be getting.

[DSTAVERT]

Unless you absolutely need virtual disks that are larger than 2TB you are far better off using the 2TB as individual datastores.

Make sure that write caching is enabled on the controller. If you don't have the battery module it won't be enabled by default. Don't enable it unless you do have the battery.

[bluesherpa]

Are you saying that extents adds a lot of overhead? I was trying to combine the drives into one volume at the OS level for a media server, but perhaps I should approach things differently?

Write cacheing is enabled. Do the performance stats that I listed sound reasonable or even good?

[FredPeterson]

Extents are bad and should never be used unless absolutely necessary.

If one extent goes bad and a VMDIK spans to the other extent...you've just lost that VMDK for good.

You don't necessarily have to have a single huge drive in Windows (or even a *nix) - you can mount other disks as folders. You don't even have to give the other disk a drive letter.

The mountvol command in Windows is pretty spiffy.

[DSTAVERT]

I would absolutely stay away from extents if that is your goal. If you want to have a large block of storage try setting up an separate physical box as a storage server with NFS and mount it as a datastore. There are several software packages you can use. Openfiler, freenas or just a simple Linux install.

[J1mbo · Accepted solution]

For once I disagree with DSTAVERT (sorry ) - since all eight extents are hosted on the same underlying RAID volume, there is no[/b] additional risk in this configuration compared to a single RAID-5 set of 8TB as might be used without ESX. If anything goes bad in this scenario all eight extents would be affected simultaneously.

As for performance, this sounds reasonable to me. Bear in mind that for virtualisation generally concern is much more about random IOPS throughput than block sequential read (or write). Eight spindles will help with that but SATA drives quickly get bogged down unfortunately.

As you have have a Core-2 Duo, I would suggest running guests with 1 vCPU only.

Please award points to any useful answer.

[RParker]

Well why even use extents if it's not ABSOLUTELY necessary?

The BEST way for this scenario is RAID the VOLUME, Make it 20 TB who cares.. if that what works, THEN create LUN's of smaller sizes. I still say, like DSTAVert that extents are a bad idea, and really tyring to force the issue isn't a good idea, there are SO many other ways to make this work better than using extents, so why force using them?

[bluesherpa]

I'm really glad I asked about extents. I'll stay away from using them.

In response to the idea of a separate NAS, honestly, it's more of a budget constraint than anything. Since I can't afford the extra hardware right now, I figured I'd do the next best thing by having one box running everything. And since it's all personal (non-production), it's more for fun, but I still want to learn and make sure I'm doing it the best way possible.

Thanks for advice about 1 vCPU. I'll definitely stick to 1. Meanwhile, I've seen a few quad core processors that would fit in my motherboard going for around $125. I'm trying to save up and eventually get a Xeon box setup too, so I might just wait for that.

Do you have more information on SATA drives and IOPS? I'm not familiar with that and would like to explore and learn more.

Thanks 😃

[bluesherpa]

In response to: The BEST way for this scenario is RAID the VOLUME, Make it 20 TB who

cares.. if that what works, THEN create LUN's of smaller sizes.

Maybe I'm misunderstanding? I created one raid set and then one large raid volume on the raid controller. Then I went into ESXi and tried to create a datastore with the 14TB volume. ESXi created a datastore, but it ended up with a max size of 1.74TB - like the remaining space was unavailable. That's why I ended up deleting the single large raid volume on the raid controller and created a bunch of 2TB raid volumes and then brought them each into ESXi individually. First as one datastore with extents. Now that I know more about extents, I'll delete that datastore and create a datastore for each 2TB volume on the raid controller.

[DSTAVERT]

I hope everyone disagrees at least some time. We all learn something each time we get to put our own ideas out there however bad they are

[J1mbo]

I know others have advised against using extents and indeed extents across physical volumes are EXTREMELY dangerous, but consider what is actually happening here. As stated above, a RAID-5 set across 8 disks, with 8 LUNs thereon presented to ESX and joined into one contiguous unit through extents.

There is no risk or performance issue since it remains a single flat address space, just put together with the slight inconvenience of 2TB 'partitions'.

More the issue is that rebuilding a 14TB volume from SATA drives in the event of a failure will take a very long time indeed and puts up the very real possibility of an unrecoverable read error occuring during the rebuild process, especially if using 'consumer' grade drives which might have uncorrectable error rates of 1 in 10^{15 bytes read (14TB is 1.4 x 10}13), in a sense there is ~10% chance the rebuild could fail.

In this respect it may be wiser to run the hardware as two RAID-5 volumes of four spindles, providing 4 LUNs on each and creating two datastores (via extents) from the two sets of four. However this will impact the IOPS capabilities.

Re IOPS, this is the random workload throughput, perhaps 60% read, 32K blocks. Using IOMeter with a 4GB test file against a Western Digital Caviar-Green 1TB drive, this pulls around 140 IOPS at 6 outstanding IOs (which note is just 4.4MB/s). A 3-drive RAID set running on a Perc-5 array controller in RAID-5 will give around 220 IOPS at 10 outstanding IOs. By comparison, a 5-drive 10k SAS RAID-5 array will deliver 1,200 IOPS at 32 outstanding IOs.

HTH

Message was edited by: J1mbo - corrected my dodgy maths

[J1mbo]

Lol yes that's certainly true dvsavert! :smileygrin: