vSphere Profile-Driven Storage Service terminates

[azhskr]

I need some help figuring out what is wrong with this service.  It is set to start automatically, but never has after a reboot.  If I start it manually, it will stay started for 9 minutes, and then enter into a stopped state.  After it enters into a stopped state the following error is shown in the event viewer on the vCenter server:

The VMware vSphere Profile-Driven Storage Service service terminated with service-specific error Incorrect function..

Any ideas on where to start to troubleshoot this issue?  A google search doesn't return anything that is of much use.

[jrmunday]

Did you ever resolve this issue? I have just upgraded to vCenter 5.0 Update 3 and have the same issue.]

If I roll back my snapshots to update 2 the issue is not there.

[Rubeck]

jrmunday... is the web client installed on the VC server too? I've just seen this on a VC5.0U3 install where the Profile-Driven Storage Service would start fine but shut down moments after..

I stopped the web client service and started the Profile-Driven Storage Service again..... now it kept running. Configured the web client service with the delayed start option so that the Profile-Driven Storage Service would have a chance to come up first...

Rebooted a few times to make sure that it worked...all good.

/Rubeck

[jrmunday]

Hi Rubeck,

Thanks very much, this solves the problem. I was pretty close to finding this as I had traced this java process (using port 31000) to the Web Client;

INFO   | jvm 1    | 2013/10/31 12:47:45 | 12:47:45 INFO - Starting SPS HTTP Server

INFO   | jvm 1    | 2013/10/31 12:47:45 | 2013-10-31 12:47:45.675:INFO::Logging to STDERR via org.mortbay.log.StdErrLog

INFO   | jvm 1    | 2013/10/31 12:47:45 | 12:47:45 INFO - Starting server on [HTTP:0.0.0.0:31000, HTTPS:0.0.0.0:31100]

INFO   | jvm 1    | 2013/10/31 12:47:45 | 2013-10-31 12:47:45.706:INFO::jetty-6.1.26.1

INFO   | jvm 1    | 2013/10/31 12:47:45 | 2013-10-31 12:47:45.753:WARN::failed SocketConnector@0.0.0.0:31000: java.net.BindException: Address already in use: JVM_Bind

INFO   | jvm 1    | 2013/10/31 12:47:45 | 2013-10-31 12:47:45.784:INFO::Started VlsiSslSocketConnector@0.0.0.0:31100

INFO   | jvm 1    | 2013/10/31 12:47:45 | 2013-10-31 12:47:45.784:WARN::failed Server@25deb1ad: java.net.BindException: Address already in use: JVM_Bind

INFO   | jvm 1    | 2013/10/31 12:47:45 | 12:47:45 ERROR - Storage Policy Service could not be initialized: java.net.BindException: Address already in use: JVM_Bind

INFO   | jvm 1    | 2013/10/31 12:47:45 | WrapperSimpleApp:

INFO   | jvm 1    | 2013/10/31 12:47:45 | WrapperSimpleApp: Encountered an error running main:

INFO   | jvm 1    | 2013/10/31 12:47:45 | WrapperSimpleApp: com.vmware.sps.fault.SpsInitializedException: Storage Policy Service could not be initialized.

INFO   | jvm 1    | 2013/10/31 12:47:45 | WrapperSimpleApp: Caused by: java.net.BindException: Address already in use: JVM_Bind

Annoyingly, the only error logged in the system event log is the following;

The VMware vSphere Profile-Driven Storage Service service terminated with service-specific error Incorrect function..

Thanks again for your help.

Cheers,

Jon

[Rubeck]

Thanks again for your help.

Actually I'm the one thanking you as your trace output shows why this is happening... I needed to get on with my life quickly so I simply started to disable add-ons which isn't a part of the default vCenter install.. Once it worked, I didn't really have the time to figure out what the reason was.

/Rubeck

[cbfinvent]

just faced the same issue. Thanks Rubeck

solution as described.

set the web client on delayed start (or disable if you don't use it)

and start the profile driven storage....

thank you vmware for making me thing again additionally :smileygrin:

[lakey81]

I also ran into the same problem and an alternate resolution I got from support was to change the config file for the profile service to change the ports it uses.

Open the wrapper.conf file located at C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\bin\service\conf\

Add these entries to the file then restart the vCenter, profile-driven storage, and web client services.

#restrict port range

wrapper.jvm.port=31550

wrapper.jvm.port.min=31550

wrapper.jvm.port.max=31999

wrapper.port=32550

wrapper.port.min=32550

wrapper.port.max=32999

[skywalkr]

Hi, After upgrading to vSphere 5.0 U3, we now have the same port grabbing issue across many vCS systems.  Any restart, such as simple Windows update, is a time consuming re-check and restart of these services as others have reported.

After testing per the instructions above, changing the WEB CLIENT's wrapper.conf control file solves our restart issue.  We did not change the Profile Driven Storage wrapper.conf file.

The PDS wrapper.conf is file's ports are shown below for reference only. Do not change this file. -> C:\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf

# wrapper JVM port

wrapper.jvm.port=31300

wrapper.jvm.port.min=31300

wrapper.jvm.port.max=31999

# wrapper port

wrapper.port=32300

wrapper.port.min=32300

wrapper.port.max=32999

The WEB CLIENT's wrapper.conf needs to have its ports restricted so change this file -> C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\bin\service\conf\

# restrict JVM port range for ESXi 5.0 U3 port race condition

wrapper.jvm.port=31550

wrapper.jvm.port.min=31550

wrapper.jvm.port.max=31999

wrapper.port=32550

wrapper.port.min=32550

wrapper.port.max=32999

# <EOF>

Thanks SO much for the tip.  This really needs to make it into the associated KB article which says to just manually restart the services which is not practical.  G. Mobley

[stacycarter]

Can anyone provide the KB for this issue?  Seeing the same thing in the lab with VC 5.0 after updating to U3.

[skywalkr]

There was no single KB when I posted the above in Oct 2013.  There were several KB dancing around issues but no solution - maybe there is now...I have not searched lately.  G. Mobley

[dmihaescu]

In the vCenter 5.0 U3 release notes (VMware vCenter Server 5.0 Update 3 Release Notes) there is the following paragraph:

• The vSphere Client shows an error message when attempting to view virtual machine Storage Profiles
  When you connect the vSphere Client to vCenter Server and attempt to view a virtual machine's storage profile, you might see the error message, vCenter Server is unable to connect to Profile-driven storage service at http:///sps/sdk.
  A port conflict occurs when the Storage Profile Service and another service are configured to use port 31000. The vSphere Web Client and vCenter Orchestrator can cause this conflict.Workaround: Restart vSphere Web Client service and the vCenter Orchestrator configuration service. The services will pick the next available free port when they are restarted. If only one of the services is present or running on the machine, you need only restart that service.

[paulnmck]

I have been trying to sort Profile-Driven Storage service starting and then stopping after updating to 5.0 U3 and found this discussion. Unfortunately we don't have web client, so that wasn't our issue. I eventually tracked it down to our NetApp VSC using the port 31000 (Used netstat -ao, found 31000, made note of PID then found it in Task Manager) and thus not allowing Profile-Driven Storage service to start. I stopped this service, restart the Profile service and then restarted the VSC service. Both services are now running fine. I have subsequently selected a delayed start to the VSC service but not had a chance to reboot to see if it works.

[stacycarter]

paulnmck - Yep, that did turn out to be the cause of my issue as well (did not realize it was happening even when at 5.0 U2).  The delayed start workaround for the VSC service appeared to work for us.

[conresjkozej]

I experienced this same issue recently and it drove me bonkers! I was able to resolve my issue by identifying which service on my vCenter server was causing a port conflict; ultimately identify which service was using port 31000. The first thing I did was run netstat -o to identify where 31000 was being used. On the far right you have a PID column. Use the PID number to identify which process (i.e. Task Manager) is using that port. In my case, I have vSphere Update Manager installed on my vCenter server in my lab environment. The Update Manager service was ultimately the culprit using port 31000.

To resolve the issue I ensured the Profile-Drive Storage Service was set to Automatic startup. I reset the vSphere Update Manager service and vSphere Web Client service to Automatic (Delayed Startup). I rebooted the system to test my settings and the vSphere Profile-Driven Storage Service started successfully.

[CMitchell3]

I pushed the issue with VMware. Please check KB2074505

VMware KB:    VMware vSphere Profile-Driven Storage service fails to start after rebooting vCenter S...

I am also trying to get them to edit the command to use netstat -aon | find "31000" instead. that provides a cleaner output.

[lucienorrin]

I had the same issue. NetApp VSC was the culprit in my situation.

I tried paulnmck's solution from this thread which worked until a reboot. To get mine working after reboot I had to adjust the sps-config file (KB2001804)

for the profile-driven storage service.

I increased the # attempts from 10 to 20 and decreased the interval 60 to 10 (seconds) between attempts

attemptNumber = 20
sleepInterval = 100000(milliseconds)

So far so good, all services start successfully after a reboot.

[CMitchell3]

In our situation, it kept rolling around. First VSC, then vCO, then vCO config... whichever one of 5 services that use Java and who's port range is set to start with 31000. Every reboot would bring a new service starting before PDS and stealing 31000.

My advice; fix PDS... not the other 5 services. it is the one forced to use 31000 instead of a dynamic range.