Problem while getting performance data for more than 100 VM using queryPerf().

[vijaybhosale]

Hi,

We have two setups, one is our in-house which is having around 1500 VM and we are running java client which collect performance data of VM and Host.

We have modified our java client for getting 200 VM performance data using queryperf() at single call and subsequently get next 200 chunk until collect all entity data(VM's and hosts). This work perfectly fine at in-house,

The client has been designed to get 200 entity data at single call because queryPerf() crashing (giving same error as mentioned in below stack trace) while getting performance data of more than 800 VM in single call.

But same approach (getting performance data of 200 VM at single time.) did not worked at client site and it is throwing exception (See stack trace).

Java client Stack trace.

17/02/2009 17:07:51 VSMVimCollector queryRealtimePerf

WARNING: Query Realtime Permormance

AxisFault

faultCode: Server.userException

faultSubcode:

faultString: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset by peer: socket write error

faultActor:

faultNode:

faultDetail:

stackTrace:javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset by peer: socket write error

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(Unknown Source)

at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)

at java.io.BufferedInputStream.fill(Unknown Source)

at java.io.BufferedInputStream.read(Unknown Source)

at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.queryPerf(VimBindingStub.java:19416)

at VSMVimCollector.queryRealtimePerf(VSMVimCollector.java:315)

Is their any VC side setting to avoid this problem?

Thanks,

Vijay

[Steve_Jin]

Can you check the VC logs to find out more?

%ALLUSERPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs

There should be 10 vpxd-?.log files. Check out the vpxd-index file to find out which one is the current.

Steve JIN, VMware Engineering

VI Java API 2.0: 4+ faster than AXIS with 1/4 of the size. Check out @ http://vijava.sf.net

[vijaybhosale]

Thanks Steve,

I could see the below message in vmxd.log file.

======================================================================

Exception while processing request: An established connection was aborted by the software in your host machine.

==========================================================================

Exactly same time when queryPerf Exception occurred.

============================================================================

4/12/2008 15:43:23 VSMVimCollector queryRealtimePerf

WARNING: Query Realtime Permormance

AxisFault

faultCode: Server.userException

faultSubcode:

faultString: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error

faultActor:

faultNode:

faultDetail:

stackTrace:javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(Unknown Source)

at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)

at java.io.BufferedInputStream.fill(Unknown Source)

at java.io.BufferedInputStream.read(Unknown Source)

at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.queryPerf(VimBindingStub.java:19416)

============================================================================

Does it mean the problematic server has some firewall settings which are not allowing the large data flown between client and server?

Thanks,

Vijay

[Steve_Jin]

It seems there was a problem between your VirtualCenter and the host on which the VMs were running.

"Exception while processing request: An established connection was aborted by the software in your host machine"

Now the number 100 VM is kind of interesting, because it's close to the max number of VMs running on a host. How are your VMs distributed across hosts? Just wonder if it happens when you get VM stats across different hosts.

While investigating this for fun, you can work around this by lowering the number of VMs you retrieve each time.

Steve JIN, VMware Engineering

VI Java API 2.0: 4+ faster than AXIS with 1/4 of the size. Check out @

[vijaybhosale]

Hi Steve,

If I keep number VM to 50 and post request for getting a performance data request processed successfully.

That seems a large data is causing a failure of request processing, requesting a 200 VM perf. data worked other setups. Therefore I am suspecting this is something do with SOAP request/response message size. I read below link from MS which has similar problem communication with web service.

http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/17e3f2c9-c3f8-4531-b251-11fafe46ec1...

They are saying some firewall/Antivirus settings on machine is casing abortion of request processing.

I have few queries about VC web service.

1) Can we limit request/response message size?

2) Is their any configuration defined in VC which is not allowing the large data request not get processed for queryPerf (perf. stat.)?

3) Can we define max Message Size Length in VC configuration?

One more interesting point about this setup is, when we are requesting inventory of from this VC response data (which has around 1000 entities includes VM/Hosts/clusters) the response did not failed. That means when request (inbound message) contains larger size then request may not reach to server/or it get corrupted.

Thanks,

Vijay

[Steve_Jin]

Thanks for sharing the information.

To limit the performance data returned, you can:

1. Limit the amount of managed entities

2. Limit the amount of perf metrics

3. Limit the number of perf samples if real-time data, and period if historical data.

Steve JIN, VMware Engineering

VI Java API 2.0: 4+ faster than AXIS with 1/4 of the size. Check out @ http://vijava.sf.net

[vijaybhosale]

I think all these are the client side settings.

Can we have any settings for VC which is causing this weird result?

[vijaybhosale]

Here are some interesting link, we can configure the input message size.

http://blogs.msdn.com/drnick/archive/2006/03/10/547568.aspx

http://support.microsoft.com/Default.aspx?id=820129

[Steve_Jin]

Thanks for sharing the information. Have you tried the configuration change? I'd like to hear more about your finding there.

Steve JIN, VMware Engineering

Creator of VMware Infrastructure Java API: http://vijava.sf.net

VI Java API 2.0 --- 15 times faster than AXIS in loading, 4+ faster in deserialization; only 1/4 of the size required by AXIS. More importantly freedom to redistribute your applications.

Download, Samples, DocWiki, RSS Feed