VMware Cloud Community
admin
Immortal
Immortal

ESX 3.0.2 - ping latency

Hey all,

we are experiencing trouble after we upgraded to esx 3.0.2. I did one machine last friday and did not notice the problem immediately, today it patched another esx and after the boot sequence was complete i saw this strange behavior..

Normally all our servers have a ping latency around 1 (or smaller) as shown below

Pinging esx1.uz.kuleuven.ac.be \[172.22.3.34] with 32 bytes o

Reply from 172.22.3.34: bytes=32 time=1ms TTL=63

Reply from 172.22.3.34: bytes=32 time<1ms TTL=63

Reply from 172.22.3.34: bytes=32 time<1ms TTL=63

...

Reply from 172.22.3.34: bytes=32 time<1ms TTL=63

Reply from 172.22.3.34: bytes=32 time<1ms TTL=63

Ping statistics for 172.22.3.34:

Packets: Sent = 24, Received = 24, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 1ms, Average = 0ms

the 2 hosts that are running 3.0.2 have these response times:

Pinging playvmware.uz.kuleuven.ac.be \[172.22.1.79] with 32

Reply from 172.22.1.79: bytes=32 time=1ms TTL=63

Reply from 172.22.1.79: bytes=32 time=10ms TTL=63

Reply from 172.22.1.79: bytes=32 time=10ms TTL=63

Reply from 172.22.1.79: bytes=32 time=9ms TTL=63

Reply from 172.22.1.79: bytes=32 time=108ms TTL=63

Reply from 172.22.1.79: bytes=32 time=10ms TTL=63

...

Reply from 172.22.1.79: bytes=32 time<1ms TTL=63

Reply from 172.22.1.79: bytes=32 time=9ms TTL=63

Reply from 172.22.1.79: bytes=32 time=8ms TTL=63

Reply from 172.22.1.79: bytes=32 time=8ms TTL=63

...

Reply from 172.22.1.79: bytes=32 time=2ms TTL=63

Reply from 172.22.1.79: bytes=32 time=33ms TTL=63

Reply from 172.22.1.79: bytes=32 time=2ms TTL=63

Reply from 172.22.1.79: bytes=32 time=11ms TTL=63

Ping statistics for 172.22.1.79:

Packets: Sent = 33, Received = 33, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 108ms, Average = 8ms

can someone verify this? or are we the only one with this issue.. it seems odd to me.. having 2 boxes just upgraded and experiencing the same problem

Reply
0 Kudos
187 Replies
doctormiru
Enthusiast
Enthusiast

Hi

Just installed 3.02 in LAB and expecting the same behavior. 6-7 ms latency and overall slow response times for console operations incl. VI Client.

We just can't wait for a fix Smiley Happy

idle..... and waiting for the 1st 3.02 patch Smiley Happy

Michael

Reply
0 Kudos
bertdb
Virtuoso
Virtuoso

there is a known problem with Samba on the 3.0.2 service console. Search the forums for recent related posts. Probably unrelated to the ping latency thing.

Reply
0 Kudos
DMcCoy
Contributor
Contributor

As I mentioned on the samba thread, its affecting something as my backup time for the service console has increased by 1000% I'm using backup exec v9 agent at the moment and only change was to patch to 3.0.2 from 3.0.1

3.0.1 backup took 4 minutes

3.0.2 backup took 40 minutes

Reply
0 Kudos
bluepenguin
Enthusiast
Enthusiast

Hi

Same problem here, DL360 G5, internal NICs with Broadcom Chipset

Also, smb performance is terrible.

I opened an SR on this.

Reply
0 Kudos
thickclouds
Enthusiast
Enthusiast

This is not encouraging! We are looking to upgrade from 3.0.0 to 3.0.2 - any updates from support?

Charlie Gautreaux vExpert http://www.thickclouds.com
Reply
0 Kudos
MinEZ
Contributor
Contributor

Yesterday I heard from a sales/accountmanager of vmware that esx3.1 is due in september.

Perhaps this will solve the problem?

My SR is still pending ...........

Reply
0 Kudos
bluepenguin
Enthusiast
Enthusiast

No, VMware has not replied to my SR. I have calculated, why smb performance is stuck at a maximum of 0.4 MB/s, see here:

http://www.vmware.com/community/thread.jspa?threadID=97905

CIFS (SMB) on ESX COS transfers 3 ethernet frames (3 x 1500 = 4096 byte) to the windows server with the share, then the COS waits for an acknowledge.

If acknowledge takes 10 ms each 4096 bytes you transfer, you can have a maximum transfer rate of about 400 KB/s

This is very discouraging, really. We are thinking about downgrading to 3.0.1. anyone has any experience in downgrading ESX?

Reply
0 Kudos
MinEZ
Contributor
Contributor

I'm sorry to inform you but I had contact with VMWare yesterday and offered to upgrade to 3.0.2 again to reproduce te problem.

But I only want to do this if they had a rollback.

Because reinstalling, configuring and patching (44x) of our esx server takes about 5 hours.

They where sorry to inform me that there is no rollback that they know of from 3.0.2 to 3.0.1

Reply
0 Kudos
Svante
Enthusiast
Enthusiast

I can confirm same issue here, on HP BL460C. All three servers have the mentioned ping latency now. Luckily it does not seem to affect backup of raw VM files over the network in our case (using NetVault), it still maxes out the transfer speed of the SDLT robot. I'll stick with 3.0.2 since ping times is just fine on the VMs. Transfer speeds from VMs is also as good as before as far as I can tell.

I am having trouble using Veem FastSCP now however; I can log on just fine, see files etc but when I try to transfer something to the ESX it fails. I have not yet investigated this however, could be something else. Maybe someone else could try it though...

Reply
0 Kudos
james_charter
Contributor
Contributor

Hi Svante,

No problems here with Veeam to an upgraded 3.0.2 Host, and transfer times to and from are acceptable. Same ping issues as everyone else - does not seem to be an upgrade stopping situation though. I will test out backups using esxRanger and see if there is any cause for concern.

Cheers,

James

Reply
0 Kudos
Svante
Enthusiast
Enthusiast

Hi James,

Thanks for testing! I tried it again now, same server, and it's fine all of a sudden. Weird since I tried it several times and nothing has changed, no reboot etc. Oh well, guess I should be happy. I would be even happier if I knew what caused it though...

Reply
0 Kudos
depping
Leadership
Leadership

upgrade 8 x 1855 Dell Blades today, no issues with backup or ping latency

Reply
0 Kudos
bertdb
Virtuoso
Virtuoso

minEZ, are you suggesting that a couple of milliseconds extra latency cause a timeout ? What kind of protocol are you talking about ??

This seems unrealistic and even impossible for any TCP/IP protocol I know.

Reply
0 Kudos
bertdb
Virtuoso
Virtuoso

MinEZ, as I've indicated in other posts, I'm convinced that your \_problem_ is different from the latency issue that got this thread started. Several people have stated that they experience the raised latency with 3.0.2, but that this doesn't affect the bandwidth of the connections (backup streams, ...) their service console produces.

Reply
0 Kudos
JoePQA
Contributor
Contributor

We just did a fresh install of 3.0.2 to a Dell PE2950 and these network performance for installing ISO's is abysmal. I will install 3.0.1 tomorrow to see if this fixes the problem but trying to install a virtual machine is painful right now!

I switched from using the broadcom card to some Intel ones and had no increase in performance.

Reply
0 Kudos
zyx100
Enthusiast
Enthusiast

Veeam products are not responsible for this ping increase. One person complained that after upgrading from 3.01 to 3.02 he noticed that grapihcs slowing down significantly and he was not using any Veeam products. He solved the problem by clean install of 3.02. So, if his problem is related to yours than for now the only solution available is clean install. It appears to be a bug in 3.02, but it is not yet known for sure.If clean install does not help than it is more complicated than I thought or we are talking about different problems. Anyway, maybe VMWare will make a patch of fix or anything else usefull on this topic.

Message was edited by:

zyx100

Reply
0 Kudos
bluepenguin
Enthusiast
Enthusiast

I got a reply from VMware Support yesterday, they say that senior engineering are currently investigating this issue.

They also say that transferring files to a linux server works fine.

Reply
0 Kudos
meistermn
Expert
Expert

Maybe the tool freeping is useful for you http://www.tools4ever.com/products/free/freeping/

Some network drivers are newer in ESX 3.0.2

Look from Page 14-20 at the folling I/O Guide

http://www.vmware.com/pdf/vi3_io_guide.pdf

Page 14 network driver table

Broadcom tg3 3.43b tg3 3.43b tg3 3.43b

Broadcom bnx2 1.3.22 bnx2 1.3.22 bnx2 1.4.51b

Intel e100 2.3.40 e100 2.3.40 e100 2.3.40

Intel e1000 7.0.33 e1000 7.0.33 e1000 7.3.15

So broadcom bnx driver version changed

Message was edited by:

meistermn

Reply
0 Kudos
MinEZ
Contributor
Contributor

bertdb, perhaps you are correct that my problem is not caused by the latency. I don't know.

I stated that after upgrading to 3.0.2 I get a timeout when I want to create a datastore on the luns connected to the upgraded servers.

Before the upgrade I did not have that problem and now after a reinstall of 3.0.1 the problem no longer exists.

All i could find was the increased latency and found this post regarding this subject.

In my first post I mentioned that perhaps my problem is not related to the latency problem but it has definitively a relation to 3.0.2

Because the servers had to go into production asap I didnot think about creating logfiles before reinstalling 3.0.1.

My misstake.

Just received an update from VMWare that they couldn't reproduce the problem. And i can understand that.

So for me its Case closed. My servers are running fine now with 3.0.1 + the 44 patches Smiley Wink

On the other hand, DMcCoy stated

As I mentioned on the samba thread, its affecting something as my backup time for the service console has increased by 1000% I'm using backup exec v9 agent at the moment and only change was to patch to 3.0.2 from 3.0.1

3.0.1 backup took 4 minutes

3.0.2 backup took 40 minutes[/i]

Doesn't that have anything to do with bandwith?

Reply
0 Kudos
TTCO
Contributor
Contributor

Dear All,

I have installed 3 brand new IBM x3650 and having same ping latency and in addition, problem to add the ESX hosts to my VC. I'll check this thread to see if any solution will appear.

//TTCO

Reply
0 Kudos