Getting "SOAP request error" on all Perl API calls to VC

[piuhapofuhpaosf]

I'm using the VI Perl Toolkit to query a number of VC's that we have. The code runs just fine for all but one of the VC's. However, on one, regardless of what API query I try, I get "SOAP request error - possibly a protocol issue: 500 Server closed connection without sending any data back". Again, this happens for any API query that I attempt. Things of note:

1.) Initially the VMWare Infrastructure Web Access service was not turned on on the VC. I turned it on. I can now access the "Web Access" links for the VC (localhost/ui) and I can browse the managed objects from a web browser using the link at localhost/mob. So it seems that the web services are working on the VC, at least locally. The problem seems to be accessing the web services from another machine.

2.) The VC is remote (i.e. in another city). The network between them should be allowing traffic. I understand that it's certainly possible that the network is blocking traffic, and we're looking into that possibility. However, in the meantime, I'm wondering if anyone has any ideas of what might be going on here. I'm at a loss. I'm not terribly familiar with the VC software, but in the poking around I've done, things seem to be setup ok. The VC itself works just fine for the many hosts that are connected to it. The ONLY problem seems to be making queries to the web services - they just fail.

3.) The connection to the VC seems to be successful. I've perl -d 'd the code and watched the connection happen - it appears to be successful. A SOAP session is created and returned. The code Util::connect("https:\/\/<vc_name>/sdk/vimService","$user","$pass"); doesn't fail. However, after the connection, running something like "my $host_views = Vim::find_entity_views(view_type=>'HostSystem');" fails, as does any other API call I make. The user I'm connecting with (I've tried two different users just to make sure that isn't the problem) exists on the VC and has admin rights.

I can't think of any other information that might be useful.

I will reiterate that the code runs just fine for all of our other VC's, so I know the code itself isn't the problem.

Any ideas would be greatly appreciated.

~d

[stumpr]

Assuming you've tried the basics, I've seen firewalls send reset flags during some requests. It might be the size of the data returned or some other bug related to an intervening packet processor in your network.

What I would recommend if you have tried the usual issues and are just stuck, would be to run a packet trace on your VC server as well as your client. See if the VC is sending a reset that matches what your client is receiving. Or just run your script locally to the VC and see if it works as well. It will tell you if the issue is related to your VC server or to your network.

[piuhapofuhpaosf]

Thanks for the info, and I'll see about giving that a shot. One thing I want to make sure I understand - the Perl SOAP requests are done via LWP's to the https://vc_server_name/sdk/vimService url, so the only port I should be worrying about is 443, right? I can use a web browser and go to https://vc_server_name without issue (the web access link, for example, uses that), so again this seems to me to point to a VC issue, not a network issue.

One other thing - I downloaded VMWare's OVF appliance and loaded it on to one of the hosts controlled by the VC that's not working. I tested the VIPerl code on that appliance and it failed with the message "SOAP request error - possibly a protocol issue: 500 SSL Read Timeout". The SSL Read Timeout is curious. Again, it seems to me the only port that should matter in this discussion is port 443 since these are (as far as I can tell) vanilla requests sent over SSL. That error message made me wonder whether the webservice isn't listening on the right port, but again, port 443 seems to work fine for the web access. And then I also have to keep in mind that the API seems to be fine when queried locally (from the VC itself - when I go to https://vc_server_name/mob, I can see the managed objects just fine, and presumably it is querying the API for that).

Incidentally, I did a netstat on the VC, and the results are:

TCP 0.0.0.0:80 0.0.0.0:0 LISTENING

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING

TCP 0.0.0.0:443 0.0.0.0:0 LISTENING

TCP 0.0.0.0:445 0.0.0.0:0 LISTENING

TCP 0.0.0.0:735 0.0.0.0:0 LISTENING

TCP 0.0.0.0:1025 0.0.0.0:0 LISTENING

TCP 0.0.0.0:1493 0.0.0.0:0 LISTENING

TCP 0.0.0.0:2967 0.0.0.0:0 LISTENING

TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING

TCP 0.0.0.0:8009 0.0.0.0:0 LISTENING

TCP 0.0.0.0:8086 0.0.0.0:0 LISTENING

TCP 0.0.0.0:9084 0.0.0.0:0 LISTENING

TCP 0.0.0.0:9086 0.0.0.0:0 LISTENING

TCP 0.0.0.0:27000 0.0.0.0:0 LISTENING

TCP 0.0.0.0:27010 0.0.0.0:0 LISTENING

TCP :2381 0.0.0.0:0 LISTENING

TCP 127.0.0.1:2301 0.0.0.0:0 LISTENING

TCP 127.0.0.1:2381 0.0.0.0:0 LISTENING

TCP 127.0.0.1:8005 0.0.0.0:0 LISTENING

TCP 127.0.0.1:8084 0.0.0.0:0 LISTENING

TCP 127.0.0.1:8085 0.0.0.0:0 LISTENING

TCP 127.0.0.1:8089 0.0.0.0:0 LISTENING

TCP 127.0.0.1:9085 0.0.0.0:0 LISTENING

I compared it to one of the working VC's and nothing jumped out at me as being wrong.

So basically I'm just a whole bunch of confused. I really don't know where to go from this point. If I can get a packet tracer going I will, but not sure if I'll be able to.

Good times.

[stumpr]

Yes, should only be port 443 (HTTPS).

What I have seen from time to time, is a firewall (sometimes they're transparent, so you may not even be aware of it -- ie, transparent web proxy) that may not be handling your web call. Why I was curious if you have issues connecting with the VC client or your script when you run from a computer local to the network hosting the problematic VC server. I had this problem with a Citrix server once where the SSL connection was being reset suddenly. It turned out to be a firewall mishandling the SSL packets and just sending back RST (reset) flags in the packet, which dropped the connection.

When the error said protocol error 500, that's the error type. I'd try to isolate any network related issues by using your script locally (try running it from the VC server or another server on the same network). Just try to eliminate any intermediates to it. It's sort of out there, but you can 'reset' the TCP/IP stack on Windows 2003. I'd have to dig up the KB article on it. I have done that a few times to combat some strange network issues. I'd try to isolate the cause a bit further though.

[rovalent]

If I am able to run curl -k https://server.domain.tld/sdk/vimService?wsdl and get a response doesnt that mean its not a firewall issue?

(I work with the original poster, so it is the same issue)

[stumpr]

Unless it's just the size of the request/response. You firewall isn't blocking though if you can access the wsdl.

The best way to rule it would be to run your program locally to the VC in question if possible. If you still have the issue while going to the VC locally, then I would say it's likely something in the TCP stack or Virtual Center service on that system. If you don't have the issue, then it's likely something with the TCP stack of the VC Host or something in the intervening network equipment.

I would make sure there isn't a local firewall on the VC Host as well.

It is possible there is a transparent proxy in your network or something similar. I've seen NAT devices send resets (bug). You would likely have to do a packet capture to get some indication of the problem (you'd need one capture on the VC Host, another on your client running your SDK app).

Does this work for a simple script to just get the server version or other small data set? Is there anyway to just create an app and run the query on the VC server directly to quickly eliminate some possibilities?

[rovalent]

I will try and run it locally again. I do not remember the results.

We did capture and found nothing out of the ordinary.

That being said, I will look into the local firewalls on the machine if there are any.

Thanks for the ideas!

--

Ron Valente

[writetonikhil]

I am getting the same error when I try to access view for a 'VirtualMachine' managed object as under:

my $entity_views = Vim::find_entity_views(view_type => 'VirtualMachine');

The result I get is:

Enter the Host name: ps6484 / vcc-esxi1: ps6484

Enter Username:root

Enter Passwd:

SOAP request error - possibly a protocol issue: <?xml version="1.0" encoding="UTF-8"?>

<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"

xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"

xmlns:xsd="http://www.w3.org/2001/XMLSchema"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<soapenv:Body>

Can anybody help?

Regards,

Nikhil