sslaz
Contributor
Contributor

Networking for ZFS based "virtual storage appliance"

Jump to solution

Hi

I'm setting up an ESXI 5.1 machine to test performance / proof of concept with a "storage appliance" in a VM on a host.

Right now I'm using a version of solaris (nexenta community edition) to test.

I have successfully been able to set up the machine with disk controllers passed through, and to add an initial vswitch with a vmkernel port configured and a port group for virtual machines. - I was able to set up the guest solaris vm with two e1000 adapters in this vswitch, one to be used for management, and one configured with an address in the same private network as the vmkernel port on the management network... and things seem to work fine.

I wanted to test if there were any performance improvement using vmxnet3 adapter, and having a completely "virtual network"  dedicated only to the nfs traffic.

So I have created an additional vswitch with no adapters attached to it, with another vmkernel port and a second virtual machine portgroup, and added a vmxnet3 adapter to my "storage vm" to go on this vswitch.

Before I do this, I was wondering whether VMCI or anything else is already going to do what I want (IE allow ESXi to communicate with the storage vm over nfs with speeds in excess of what the limitation of the e1000 adapter would provide by default)

i say this because my initial testing with iozone gave me numbers which seemed beyond 1 gbs..

0 Kudos
1 Solution

Accepted Solutions
mcowger
Immortal
Immortal

So, to recap,  the e1000 typically would perform better than 1Gbit for "local" (within the same host) networking; and (assuming the guest OS's tcp/ip stack handles the VMXNET3 interface well - IE drivers, configuration, OS efficiency.)  the VMXNET3 adapter would perform even better.

It can even perform better if the uplink to the physical network is faster.  Basically the 1GBit 'link rate' means nothing.

If anyone could point me to a "best practice" guide or discussion on configuring isolated "host private" networks for best performance with virtualized NFS shared storage, I would appreciate it. My plan is to set up  three hosts, with the majority of the datastores local to the same host, with little nfs traffic going off each  box...

Well, theres really only 1 way to setup a 'host-only' network, so I dont think you will find a BP guide.

Also the terminology escapes me somewhat, because "virtual private network", "private network" have other meanings in other contexts. Not sure what to call a network which has no attached physical network adapters and is localized to a single ESXi host...

Most would call this a host-only network.

--Matt VCDX #52 blog.cowger.us

View solution in original post

0 Kudos
13 Replies
mcowger
Immortal
Immortal

What you are seeing is normal - the networking performance of a VM is limited only by the performance of the CPUs and memory bus.

--Matt VCDX #52 blog.cowger.us
sslaz
Contributor
Contributor

Ok, thanks, but that does not quite make sense.  Are you saying that in general, the virtual networking performance between a host and a VM residing on that host is only limited by cpu and memory bus/capacity - and that the choice of configured network adapater (if the communications use tcp/ip) has no effect on max bandwidth between the two hosts? 

Because obviously, the networking performance of  a vm which is communicating with a remote nfs store (or anything else which does not reside on the same host) is limited also by the network adapter and the intervening network (switches, etc)

In that case, what's the point of VMCI  over tcp for machine to machine communication ?  I guess it must just have less overhead?

0 Kudos
rickardnobel
Champion
Champion

sslaz wrote:

Are you saying that in general, the virtual networking performance between a host and a VM residing on that host is only limited by cpu and memory bus/capacity - and that the choice of configured network adapater (if the communications use tcp/ip) has no effect on max bandwidth between the two hosts?

That is correct, since all networking frame forwarding on the vSwitches is really done in ESXi CPU and RAM, the limit is mostly these hardware resources, together with the effectiveness of the guest operating system TCP/IP stack.

The virtual adapter card could improve this even more, with the VMXNET3 adapter being more optimized for running inside ESXi then the E1000 adapter, however you could between two guests on the same host get more than 1 Gbit of bandwidth even with the "1 Gbit" E1000.

My VMware blog: www.rickardnobel.se
sslaz
Contributor
Contributor

Ok, this really clarifies things:

all networking frame forwarding on the vSwitches is really done in ESXi CPU and RAM, the limit is mostly these hardware resources, together with the effectiveness of the guest operating system TCP/IP stack.

The virtual adapter card could improve this even more, with the VMXNET3 adapter being more optimized for running inside ESXi then the E1000 adapter, however you could between two guests on the same host get more than 1 Gbit of bandwidth even with the "1 Gbit" E1000.

Thanks!

So, to recap,  the e1000 typically would perform better than 1Gbit for "local" (within the same host) networking; and (assuming the guest OS's tcp/ip stack handles the VMXNET3 interface well - IE drivers, configuration, OS efficiency.)  the VMXNET3 adapter would perform even better.

If anyone could point me to a "best practice" guide or discussion on configuring isolated "host private" networks for best performance with virtualized NFS shared storage, I would appreciate it. My plan is to set up  three hosts, with the majority of the datastores local to the same host, with little nfs traffic going off each  box...

Also the terminology escapes me somewhat, because "virtual private network", "private network" have other meanings in other contexts. Not sure what to call a network which has no attached physical network adapters and is localized to a single ESXi host...

0 Kudos
mcowger
Immortal
Immortal

So, to recap,  the e1000 typically would perform better than 1Gbit for "local" (within the same host) networking; and (assuming the guest OS's tcp/ip stack handles the VMXNET3 interface well - IE drivers, configuration, OS efficiency.)  the VMXNET3 adapter would perform even better.

It can even perform better if the uplink to the physical network is faster.  Basically the 1GBit 'link rate' means nothing.

If anyone could point me to a "best practice" guide or discussion on configuring isolated "host private" networks for best performance with virtualized NFS shared storage, I would appreciate it. My plan is to set up  three hosts, with the majority of the datastores local to the same host, with little nfs traffic going off each  box...

Well, theres really only 1 way to setup a 'host-only' network, so I dont think you will find a BP guide.

Also the terminology escapes me somewhat, because "virtual private network", "private network" have other meanings in other contexts. Not sure what to call a network which has no attached physical network adapters and is localized to a single ESXi host...

Most would call this a host-only network.

--Matt VCDX #52 blog.cowger.us

View solution in original post

0 Kudos
sslaz
Contributor
Contributor

ah, this is helpful too

The e1000 typically would perform better than 1Gbit for "local" (within the same host) networking; and (assuming the guest OS's tcp/ip stack handles the VMXNET3 interface well - IE drivers, configuration, OS efficiency.)  the VMXNET3 adapter would perform even better.
It can even perform better if the uplink to the physical network is faster.  Basically the 1GBit 'link rate' means nothing.

This makes sense.. in the case of nfs perhaps it gets more tricky?

if the physical network path  was all 10G, it would be ideal and I could have a single e1000 adapter on my host, and then (I believe) a single nfs datastore would be able to achieve faster than 1 Gbit transfer through that single adapter  to a remote vm on a remote host.

In my case, I have  4 1GBit physical adapters on the host so I believe that my aggregate uplink is "faster" but will not be utilized fully for a single nfs connection.

e.g.  if we are talking nfs traffic, I believe that the network speed per  nfs connection will be limited to 1 Gbps .  e.g if the storage VM has only a single e1000 virtual adapter, then it will only use one IP address on that adapter per nfs datastore connection, even if a second IP is configured on that adapter

(eg 10.10.10.1):

Storage vm <-e1000 faster than 1gbps, IP address 10.10.10.1 -> local Esxi host vswitch<- single tcp/ip stream traffic sent to single 1G physical adapter (cant be faster than 1G)  ->  physical switch <-> <- single tcp/ip stream traffic through single 1G adapter remote ESXi)

0 Kudos
mcowger
Immortal
Immortal

if the physical network path  was all 10G, it would be ideal and I could have a single e1000 adapter on my host, and then (I believe) a single nfs datastore would be able to achieve faster than 1 Gbit transfer through that single adapter  to a remote vm on a remote host.

Yes - entirely possible

In my case, I have  4 1GBit physical adapters on the host so I believe that my aggregate uplink is "faster" but will not be utilized fully for a single nfs connection.

For NFS, which is single-connection-based (unless you are using pNFS), then yes, you would likely be limited by the IPhashing algorithms.  But if you had multiple ip tuples involved (multiple mounts, or pNFS, etc), then you could exceed 1Gbit

e.g.  if we are talking nfs traffic, I believe that the network speed per  nfs connection will be limited to 1 Gbps .  e.g if the storage VM has only a single e1000 virtual adapter, then it will only use one IP address on that adapter per nfs datastore connection, even if a second IP is configured on that adapter

Again, yes if you only use a single connection.


--Matt VCDX #52 blog.cowger.us
0 Kudos
sslaz
Contributor
Contributor

pNFS looks interesting for the future... But guess its not supported in vSphere yet.. Though it looks like netapp and redhat support it...

0 Kudos
mcowger
Immortal
Immortal

Well, you were talking about guest performance, and you can do whatever you want in the guest Smiley Happy

But the hypervisor doesn't do it yet....RH, NetApp, EMC all support it.

--Matt VCDX #52 blog.cowger.us
0 Kudos
sslaz
Contributor
Contributor

yes! We will have nfs traffic  between guests, so this is interesting. Smiley Happy

But not so useful for a datastore Smiley Sad

0 Kudos
sslaz
Contributor
Contributor

ok, I just read this in scott lowe's book and now I'm curious again, and want to test this:

Normally, vSphere requires that you mount an NFS datastore using the same IP address or hostname and path on all hosts (.... However,In vSphere 5, you [can]use a DNS hostname that resolves to multiple IP addresses. In this case, the vSphere NFS client will actually use all the different IP addresses behind the hostname. This is the exception to the rule—in this sort of configuration, the NFS client could end up using multiple links in a link aggregate for increased overall throughput between the ESXi host and the NFS datastore.

My plan was to simply add multiple interfaces to my virtual storage server, and give them various different IP addresses, and then statically create multiple datastores on the ESXI host which directly  reference the various IP addresses... The actual exported nfs filesystem on the nfs server would be the same, but with multiple IP addresses, and multiple virtual addresses, I would guarantee that a single connection across the gigabit physical network would not be a limiting factor.  however this seems better - no need for multiple datastore definitions.

0 Kudos
mcowger
Immortal
Immortal

You'd still have to do that (create multiple interfaces), but you'd just rely on the RR effects of DNS having multiple A records....this DOESN'T guarentee a balance though, because most DNS servers just randomly pick which A record to give you when there are multiple...its possible to get the same record on many hosts, and not balance properly.

--Matt VCDX #52 blog.cowger.us
0 Kudos
sslaz
Contributor
Contributor

hmm. ok, the sentence is confusing  because its not clear.  It says "the NFS client could end up using multiple links" which makes me think that instead of only having one client link per host to a nfs server which is serving a datastore, the ESXi host might actually read all returned A records and open up multiple client links to all of the IP addresses (multiple client links for a single vm-> datastore? or multiple only if you have multiple vm->datastore connections?). It's not clear from the description how this actually works... it would be *nice* and more *friendly* if the ESXi server created multiple client links for each client vm. less helpful if it just picks the first returned address each time a vm connects  a specific datastore.

still it would be interesting to test, and easier to implement with a single datastore configured per host instead of 4  per host if it works even "sorta ok".. and worrying about manually balancing vm -> datastore definitions.

0 Kudos