Updated vCenter, now perl scripts get SOAP error "500 EOF when chunk header expected"

[david_drew]

We've got some reporting scripts that do a number of things, generate alerts and save performance info, etc. vCenter was upgraded this weekend and all of the scripts are failingoccasionallywith this error. At least one of the scripts is pretty light, it doesn't pull that much information, though it does it for hundreds of VMs.

This is using VI Perl 1.6.

Any help would be appreciated.

-David

[lamw]

What version of vCenter? 2.5 or 4.0? If you're using 4.0, you should also update to the latest version of vSphere SDK for Perl or update your vMA 4.0 system as there are new modules for vSphere.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

Twitter: @lamw

If you find this information useful, please award points for "correct" or "helpful".

[jnhall]

Perl Toolkit 1.6 should be forward compatible with VC/ESX 4.0. This is actually a feature of the server. It should detect that it is being contacted by a client that expects VI API 2.5 and behave as if it were a VI API 2.5 server.

You can try upgrading to Perl Toolkit (vSphere Perl SDK, same thing) 4.0 or if you like, post more information about the script(s). It's possible you have found a bug.

[david_drew]

Thanks for both of your responses.

vCenter was upgraded to 4.0. I updated a VI Perl host to the vSphere SDK, but I'm seeing the same issue with the scripts.

One of the scripts is breaking on Vim::find_entity_views(). It prints a huge amount of raw XML to STDERR right before the SOAP error.

1. $entity_type in this case is 'VirtualMachine'

my $entities = Vim::find_entity_views ( view_type => $entity_type);

I'll post more detail tomorrow after I check the other scripts.

[jnhall]

Could you take a look at your vpxd logs while you're at it and see if anything catches your eye.

This is a behavior I haven't seen before and it's my job to see Perl Toolkit misbehave.

[benma]

Hey Guys,

I've updated to vCenter 4.0 and vMA 4

I'm running the following script and recieve SOAP error "500 EOF when chunk header expected" too.

16 Vim::login(service_url => $opts, user_name => $opts, password => $opts);

17 my $vm_views = Vim::find_entity_views((view_type => 'VirtualMachine'));

18 my $vmcounter;

19 foreach my $vm (@$vm_views) {

20 $vmcounter++

21 }

22 print "Amount:, ", $vmcounter,"\n";

From the vpxd log

Unexpected fault reading property: vim.VirtualMachine,GetEnvironmentBrowser: vmodl.fault.NotSupported

[david_drew]

I got suspicous after I saw that the problem originated from Vim::find_entity_views(), and eventually found out that a single VM had corrupt data in vCenter. Filtering out that VM solved everything, and we've since fixed it. It looks like this happened during the upgrade to 4.0.

I won't mark this answered yet, because of benma's problem.

On a different note, here are a few strange API changes I've found so far:

Historical interval values used to be 'Past Day', 'Past Week', etc. Now they're: 'Past day', 'Past week', etc.

Some capitalization has changed in device names, 'Serial Port x' is now 'Serial port x'.

This conflicts with the vSphere docs at: http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.HistoricalInterval.ht...

[benma]

How did you found your corrupt vm ?

[david_drew]

I used the 'filter' tag in find_entity_views() to select groups of VMs by name. I found one group that caused a crash, and went through several iterations of the script narrowing it down until I found it.

Our naming conventions makes it easy, but you should be able to do it.

my $entities = Vim::find_entity_views (

view_type => $entity_type,

filter => { 'name' => qr/(na0|na1|na2|na3)/ }

);

When I found the VM, I confirmed that it was the only one by filtering everything except that one:

filter => { 'name' => qr/^(?!na44_c6126)/ }

[jnhall]

By the way

filter => { '-name' => 'na44_c6126' }

(or the unnecessarily complex filter => { '-name' => qr/^na44_c6126$/ } )

should also work. '-property' means "doesn't match this string/pattern." Hopefully it works!

[jnhall]

"Historical interval values used to be 'Past Day', 'Past Week', etc. Now they're: 'Past day', 'Past week', etc.

Some capitalization has changed in device names, 'Serial Port x' is now 'Serial port x'. "

This device names change broke at least one Perl Toolkit application. I think the change was inadvertent, but I don't think that the localization team plans to change this back (because that would break other stuff). I believe we will be accommodating this change in the next release of Perl Toolkit.

[jnhall]

That behavior (the SOAP error) may be a Perl Toolkit bug and I'll look at it. Perl Toolkit should be coping with whatever the server is coughing back at it.

The actual problem looks like the stubs don't match the server. Can you post the build numbers of vCenter and ESX (if appropriate) that you are using as well as the version of Perl Toolkit? Or in this case, the vMA build number?

[benma]

Hey guys,

thanks for yours replys.

I found the VM through filtering. After remove the VM from inventory and reregister the problem is gone.

I'm running the latest vCenter 4, ESX4 and vMA 4.0 (without the patches from vima-update)

[jnhall]

Do either of you have an idea in what way the VM was corrupted/malfunctioning/etc.?

[david_drew]

I'm waiting on a response from the admin for more info (I tracked the problem down and did a workaround, he made changes to the VM on the ESX server, because it wouldn't come up in the VIC through vCenter).

I'll post details later.

[benma]

@jnhall The Guest OS of the bad VM was still fine. After filtering out the VM i've opended the vSphere Client

and clicked on the VM. In the General box on the summery tab of the VM where you can view some details about your VM

there was everything blank.

Hope that helps

[jnhall]

That's interesting and I have no idea what the problem would be. Do you still have the VM and its associated files? If so it would be good to figure out how to get it here. Do you have a support agreement? If not I can still try to help you.

[david_drew]

In our case, I pointed out the problem VM to our admin. He tried viewing the VM settings in the VIC (connected to vCenter), but got an error. He pointed the VIC at the ESX server, opened up the settings for the VM, then saved it without making any changes, and it fixed the problem in vCenter.

I can get the logs for you if you want. The admin said there wasn't anything in the logs related to that VM.

BTW, sorry, but I've got another question about the interval labels:

Labels for VC 2.5-147633 - 'Past Day', 'Past Week', 'Past Month', ...

Labels for VC 4.0 - 162856 (production): 'Past Day', 'Past week', 'Past month', ...

Labels for VC 4.0 - 162856 (test lab): 'Past day', 'Past week', 'Past month', ...

In our production 4.0 vCenter, "Past Day" is capitalized as stated in the docs, but the other standard intervals have lowercase second words.

In our test lab 4. 0 vCenter, ALL of the intervals have lowercase second words.

Do you have any idea what's causing this, and why the same build would have different labels? Also, since the docs are wrong, what are the correct values for these labels? If I send the wrong label, the script breaks, with a SOAP Fault: Fault string: A specified parameter was not correct. querySpec.interval (Fault detail: InvalidArgumentFault)

Maybe the API could be updated to not be case sensitive?

Thanks, and let me know if you want to see the logs.