Major slowness with get-vm using 1.0 build

[jbarr]

Hey all,

With great anticipation I installed the new build however get-vm now takes a full 5 minutes longer than with the beta. It seems like there is a timeout that is happening before it finally works. Here are results from beta build vs ga build:

I have 260+ VMs on VC running the release build of 2.5.

Beta

PS C:\> measure-command {$vm=get-vm}

Days : 0

Hours : 0

Minutes : 0

Seconds : 20

Milliseconds : 570

Ticks : 205704576

TotalDays : 0.000238084

TotalHours : 0.005714016

TotalMinutes : 0.34284096

TotalSeconds : 20.5704576

TotalMilliseconds : 20570.4576

GA

PS C:\> measure-command {$vm=get-vm}

Days : 0

Hours : 0

Minutes : 5

Seconds : 27

Milliseconds : 731

Ticks : 3277319374

TotalDays : 0.00379319371990741

TotalHours : 0.0910366492777778

TotalMinutes : 5.46219895666667

TotalSeconds : 327.7319374

TotalMilliseconds : 327731.9374

Thanks

[admin]

A few questions:

First, 32 bit or 64 client? There is an issue on 64 bit clients related to Microsoft KB917507 (the resolution there being to install .NET 2.0 SP1.) (Looking at our release notes this is really not as explicit as it should be.)

If that's not the case, is that happening only the first time you run get-vm or each time?

Third, are you seeing slowness with any other cmdlets, get-vmhost, get-virtualswitch, anything like that?

[jbarr]

I'm seeing this on every 32 bit XP and W2k3 machine I have installed it on; I have not tried 64 bit.

The slowness happens everytime I make the call, not just the first time.

GET-VM in the GA release is also slower when when going directly against an ESX Host.

Other "get-" commandlets run as fast or faster in GA. See data below.

Beta

get-vm against ESX host

PS C:\> measure-command {$v=get-vm -server $v2}

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 383

Ticks : 3839071

TotalDays : 4.44336921296296E-06

TotalHours : 0.000106640861111111

TotalMinutes : 0.00639845166666667

TotalSeconds : 0.3839071

TotalMilliseconds : 383.9071

GA

get-vm against ESX host

PS C:\> measure-command {$v=get-vm -server $v2}

Days : 0

Hours : 0

Minutes : 0

Seconds : 3

Milliseconds : 8

Ticks : 30089854

TotalDays : 3.48262199074074E-05

TotalHours : 0.000835829277777778

TotalMinutes : 0.0501497566666667

TotalSeconds : 3.0089854

TotalMilliseconds : 3008.9854

GA

Other "Get" Cmdlets

PS C:\> $com="cluster","datacenter","datastore","folder","resourcepool","vmhost"

PS C:\> $com | %{"get-$_";measure-command {$var=&get-$_}}

get-cluster

Days : 0

Hours : 0

Minutes : 0

Seconds : 1

Milliseconds : 616

Ticks : 16161299

TotalDays : 1.87052071759259E-05

TotalHours : 0.000448924972222222

TotalMinutes : 0.0269354983333333

TotalSeconds : 1.6161299

TotalMilliseconds : 1616.1299

get-datacenter

Days : 0

Hours : 0

Minutes : 0

Seconds : 1

Milliseconds : 185

Ticks : 11854570

TotalDays : 1.37205671296296E-05

TotalHours : 0.000329293611111111

TotalMinutes : 0.0197576166666667

TotalSeconds : 1.185457

TotalMilliseconds : 1185.457

get-datastore

Days : 0

Hours : 0

Minutes : 0

Seconds : 1

Milliseconds : 370

Ticks : 13700794

TotalDays : 1.5857400462963E-05

TotalHours : 0.000380577611111111

TotalMinutes : 0.0228346566666667

TotalSeconds : 1.3700794

TotalMilliseconds : 1370.0794

get-folder

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 359

Ticks : 3592464

TotalDays : 4.15794444444444E-06

TotalHours : 9.97906666666667E-05

TotalMinutes : 0.00598744

TotalSeconds : 0.3592464

TotalMilliseconds : 359.2464

get-resourcepool

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 298

Ticks : 2981113

TotalDays : 3.45036226851852E-06

TotalHours : 8.28086944444444E-05

TotalMinutes : 0.00496852166666667

TotalSeconds : 0.2981113

TotalMilliseconds : 298.1113

get-vmhost

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 316

Ticks : 3169036

TotalDays : 3.66786574074074E-06

TotalHours : 8.80287777777778E-05

TotalMinutes : 0.00528172666666667

TotalSeconds : 0.3169036

TotalMilliseconds : 316.9036

Beta

Other "Get" Cmdlets

PS C:\> $com="cluster","datacenter","datastore","folder","resourcepool","vmhost"

PS C:\> $com | %{"get-$_";measure-command {$var=&get-$_}}

get-cluster

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 678

Ticks : 6781873

TotalDays : 7.8493900462963E-06

TotalHours : 0.000188385361111111

TotalMinutes : 0.0113031216666667

TotalSeconds : 0.6781873

TotalMilliseconds : 678.1873

get-datacenter

Days : 0

Hours : 0

Minutes : 0

Seconds : 1

Milliseconds : 492

Ticks : 14922066

TotalDays : 1.72709097222222E-05

TotalHours : 0.000414501833333333

TotalMinutes : 0.02487011

TotalSeconds : 1.4922066

TotalMilliseconds : 1492.2066

get-datastore

Days : 0

Hours : 0

Minutes : 0

Seconds : 1

Milliseconds : 733

Ticks : 17331616

TotalDays : 2.00597407407407E-05

TotalHours : 0.000481433777777778

TotalMinutes : 0.0288860266666667

TotalSeconds : 1.7331616

TotalMilliseconds : 1733.1616

get-folder

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 776

Ticks : 7760329

TotalDays : 8.98186226851852E-06

TotalHours : 0.000215564694444444

TotalMinutes : 0.0129338816666667

TotalSeconds : 0.7760329

TotalMilliseconds : 776.0329

get-resourcepool

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 741

Ticks : 7414081

TotalDays : 8.58111226851852E-06

TotalHours : 0.000205946694444444

TotalMinutes : 0.0123568016666667

TotalSeconds : 0.7414081

TotalMilliseconds : 741.4081

get-vmhost

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 767

Ticks : 7675268

TotalDays : 8.88341203703704E-06

TotalHours : 0.000213201888888889

TotalMinutes : 0.0127921133333333

TotalSeconds : 0.7675268

TotalMilliseconds : 767.5268

[jpoling]

I am seeing similar slowness in the GA release. . .

My Client is 32 bit Vista SP1:

Days : 0

Hours : 0

Minutes : 0

Seconds : 22

Milliseconds : 809

Ticks : 228090292

TotalDays : 0.000263993393518519

TotalHours : 0.00633584144444444

TotalMinutes : 0.380150486666667

TotalSeconds : 22.8090292

TotalMilliseconds : 22809.0292

I tried it on my VirtualCenter Server as well (32 bit Server 2003 SP2):

Days : 0

Hours : 0

Minutes : 0

Seconds : 7

Milliseconds : 708

Ticks : 77080025

TotalDays : 8.92129918981481E-05

TotalHours : 0.00214111180555556

TotalMinutes : 0.128466708333333

TotalSeconds : 7.7080025

TotalMilliseconds : 7708.0025

[jguidroz]

I'm seeing the same results. About 3 minutes to run get-vm.

[admin]

We're very actively looking into this.

Can anyone who seems to be having this trouble please post the results of this command:

measure-command {$vm = get-vm}; $vm.length

(For me it was 2.4 seconds and 32 VMs, I'm lucky I guess.)

Thanks, everyone.

[jguidroz]

2 minutes 48 seconds.

175 VMs

[jbarr]

266 VMs

18 seconds vs 5 min. 58 seconds !!

Beta Build

PS C:\WINDOWS> measure-command {$vm = get-vm}; $vm.length

Days : 0

Hours : 0

Minutes : 0

Seconds : 18

Milliseconds : 35

Ticks : 180359908

TotalDays : 0.000208749893518519

TotalHours : 0.00500999744444444

TotalMinutes : 0.300599846666667

TotalSeconds : 18.0359908

TotalMilliseconds : 18035.9908

266

GA Build

PS C:\> measure-command {$vm = get-vm}; $vm.length

Days : 0

Hours : 0

Minutes : 5

Seconds : 58

Milliseconds : 363

Ticks : 3583638845

TotalDays : 0.00414773014467593

TotalHours : 0.0995455234722222

TotalMinutes : 5.97273140833333

TotalSeconds : 358.3638845

TotalMilliseconds : 358363.8845

266

[sepeck]

I am seeing the slowness as well.

Windows XP sp3 (now - last week was sp2, I saw the slowness last week as well) - 32 bit OS

I have installed

MS .NET Framework 1.1 and 1.1 Hotfix (K928366)

MS .NET Framework 2.0 Service Pack 1

MS .NET Framework 3.0 Service Pack 1

PS PS:\VMware> measure-command {$vm = get-vm}; $vm.length

Days : 0
Hours : 0
Minutes : 3
Seconds : 13
Milliseconds : 310
Ticks : 1933106294
TotalDays : 0.00223739154398148
TotalHours : 0.0536973970555556
TotalMinutes : 3.22184382333333
TotalSeconds : 193.3106294
TotalMilliseconds : 193310.6294
106

Steven Peck

[Mark_Weaver]

PS C:\Powershell> Get-VIToolkitVersion

VI Toolkit Version

-

---

VMware VI Toolkit for Windows 1.0 build 103777

PS C:\temp> measure-command {$vm = get-vm}; $vm.length

Days : 0

Hours : 0

Minutes : 11

Seconds : 57

Milliseconds : 13

Ticks : 7170137950

TotalDays : 0.00829877077546296

TotalHours : 0.199170498611111

TotalMinutes : 11.9502299166667

TotalSeconds : 717.013795

TotalMilliseconds : 717013.795

600

The Beta version is as follows:

PS C:\> Measure-Command { $vm = get-vm };$vm.count

Days : 0

Hours : 0

Minutes : 0

Seconds : 28

Milliseconds : 368

Ticks : 283684272

TotalDays : 0.000328338277777778

TotalHours : 0.00788011866666667

TotalMinutes : 0.47280712

TotalSeconds : 28.3684272

TotalMilliseconds : 28368.4272

600

[jbarr]

Everyone with problems - what release and update level of Virtual Center are you using?

I'm at 2.5 Release version. We will be moving to update 2 in the next few weeks.

jb

[jguidroz]

Virtual Center 2.5 Update 1.

[sepeck]

Virtual Center v2.5.0 Build 64192

We will be testing the update 2 'soon'.

[jpoling]

Virtual Center 2.5 Update 1

[jbarr]

C_Shanklin,

Are you guys getting anywhere with this? Is there anything I should do to "formalize" the issue?

Thanks,

jb

[admin]

The development team managed to reproduce the problem this morning, we're still looking to see how wide-reaching the problem is and whether there's a workaround.

[admin]

Hi everyone,

An update from the development team, we've reproduced what we believe to be the problem, the slowness appears to be related to VMs that have "vmxnet" adapters, which are adding a substantial amount of overhead to the process of retreiving VMs, it seems.

I'd like to try to do two things, with help from anyone willing to spend a few moments.

First, for people experiencing slowness, can you confirm that you have a substantial number of VMs using vmxnet adapters? If not, then we've failed to identify the correct cause, or maybe just identified another source of slowness.

Second, can anyone identify a viable way of dealing with the issue until we can get it fixed for real? One thing to note is that if you retreive VMs, then store them in a variable, passing these stored variables to other routines won't experience the slowness. Basically this is just the process of moving the slowness to the first step in a script, then subsequent calls will operate at normal speed. Another, more disruptive, possibility would be converting some adapters to flex or e1000. Anyone have any other ideas?

Thanks in advance.

[hugopeeters]

200 vms

beta: 11 seconds

GA: 51 seconds

All adapters are of type 'Flexible'

[jbarr]

That's probably our issue:

Count Name

-

---

-

---

230 VirtualVmxnet

38 VirtualPCNet32

4 VirtualE1000

Is there a quick way to update the VMs?

[sepeck]

While I look around, is there a quick way to report this? d'oh never mind, here is a way to check for others.

$vm = get-vm | get-networkadapter

$vm | group type

Environment 1

Count Name

150 Vmxnet

55 Flexible

Environment 2

Count Name

224 Flexible

11 e10000

134 Vmxnet

I'll skip the other environments unless you need more information.

Not being all that savvy in regards to VMware, what is the net result of switching adapter types? I am thinking that suggesting a mass change on more then 300 systems, it would be best if I could work remote and not in physical reach of my co-workers for a while so not an immediate option here.

Are there any other reasons/advantages to not using vmxnet adapters I should mention going forward when we deploy new systems?

Message was edited by: sepeck