VDP over NFS datastore and network monitoring

[bvil96]

Finally, I decided to start my own thread about this.

Here is the post I added to an other one:

Hi,

I installed VDP for the first time yesterday at 1:00pm. The appliance is sitting on a host in a vCenter that's in another network.

Host has 4 nics, one for corporate network, another one for a factory isolated network, vMotion and another isolated network.

The appliance is connected via NFS to a NAS that's in the same switch over 1GbE links.

Took a backup of a vm (that's sitting on another ESX cluster: same vCenter) yesterday at night and all went well. Took 45 minutes to backup it up the first time.

But since the first minute of VDP installation, I'm looking at my network monitoring system and it shows me bursts of data every 5-10 minutes sending in/out between the nas and esx... Actually vdp appliance sending out and nas receiving in.

It is almost everytime around 257 155 kbit/s every 5-10 minutes in the graph. Nothing else is monitoring (like vmware sensors or wmi of any kind. just snmp traffic from the switch port and a ping to the esx host)

No other ESX shows that kind of traffic. If I look at the nas graph, same thing which leads me in the direction that the case is between these two.

I shuted down the application so I can really see if there is an issue with it or related to nas device directly...

A thing I also noticed is traffic is coming out of vmnic0 for last night backup at 10pm which is fine but immediately dropped after the 45 minutes of backup when done. The strange thing is that the excessive traffic comes out of vmnic2 which is hooked up to the private network and has nothing to do with VDP. But as soon as I close the VDP appliance vm, graphs drop and no problem at all.

So the final question is; where the hell does that traffic comes from and why is it going on the vmnic2 while the vm has nothing to do in that port group!

Actually I think i'll go hook up the nas in the same switch as the vmnic0 as i'm thinking about it.. I guess the traffic is coming from vmnic0 but the nas is hooked up in the same network as vmnic2 so it has to jump accross chips inside the esx host. But I wonder why it keeps polling 200Mbps of data every 5-10 minutes for a duration of 2 minutes though maybe someone could shed some light on that.

That was the reason why I saw traffic on both interfaces. Now, NAS is hooked up in the same switch on the corporate network as the ESX host which the appliance is sitting on.

The vdp vm has 4 datastores:

100GB OS is on the SAN

3x 1TB data on the nas over nfs

Host ESXi 5.5.0 build 2456374

vCenter 5.5.0 build 2442329

NAS is a Lenovo Iomega StorCenter ix4-300d

(I enabled jumbo frames all along the path.. no changes)

What is utilizing that much of bandwidth from the appliance? Is this normal? It's continiously transfering stuff and I can't figure out what that is!

Are these fragments from last night backup? It was not even 80GB total of backup .. shouldn't be that long? Backup said complete and dedup to 32GB from 80...

Please if you could be of any help that would be appreciated.

Thanks.

[bvil96 · Accepted solution]

I think I found something here: VDP CPU Usage

It seems like the appliance is doing Read Checks on the backup storage destination every 5 minutes (300 seconds) and that is normal behavior according to VMware.

Thing is that 1.5GB of data (in my case) every 5 minutes is not acceptable as it might alter bandwidth utilization.

I logged in the console of vdp and applied to command but with interval 0.

I found another post here: VDP Now uses constant bandwidth even during blackout period

They say that the maximum interval available is 3600 seconds which is an hour. That's better than 5 minutes but it looks like the 0 parameter is accepted by the appliance and no error is reported.

I will update in more than an hour to see if it works because at first, I applied the 3600 and my graphs shows the changes applied.

It is 8:20am here so I will be back after 9:30 to make sure the hour passed.

[bvil96]

So far so good now. At 9:12 a (big) amount of data has been sent from the ESX to the NAS.

At first, I thought the 0 parameter didn't work but in the ESX's logs, it appeared to be an Integrity Check from the appliance.

Task: VDP: Integrity Check

info

2015-04-01 9:12:57 AM

** NO DESCRIPTION FOR: com.vmware.vdp2.integritycheck

VPD-Appliance

DOMAIN\username

VDP: Integrity started.

info

2015-04-01 9:12:57 AM

** NO DESCRIPTION FOR: com.vmware.vdp2.integritycheck

VPD-Appliance

DOMAIN\username

Look at the values after that:

Date Time Traffic Total (volume) Traffic Total (speed) Traffic In (volume) Traffic In (speed) Traffic Out (volume) Traffic Out (speed) Downtime Coverage

4/1/2015 9:16:24 AM 5,893,264 KByte 804,761 kbit/s 46,685 KByte 6,375 kbit/s 5,846,579 KByte 798,386 kbit/s 0 % 100 %

4/1/2015 9:15:24 AM 6,188,253 KByte 844,762 kbit/s 45,191 KByte 6,169 kbit/s 6,143,062 KByte 838,593 kbit/s 0 % 100 %

4/1/2015 9:14:24 AM 6,734,451 KByte 919,630 kbit/s 48,734 KByte 6,655 kbit/s 6,685,717 KByte 912,975 kbit/s 0 % 100 %

4/1/2015 9:13:24 AM 4,052,672 KByte 553,140 kbit/s 97,905 KByte 13,363 kbit/s 3,954,767 KByte 539,778 kbit/s 0 % 100%

I also thought there were accumulated amount of data (buffered) since the last scan (an hour ago). But again, it seems the 0 parameter passed correctly.

I'll continue to update this thread with my results. I hope there are not too much Integrity Checks like this during maintenance period (i'm expecting only 1 or a few).

Must say that Backup window is configured from 10pm to 6am. Rest of the time is maintenance window.