VMware Cloud Community
scylla
Enthusiast
Enthusiast
‎06-14-2018 09:39 AM
Jump to solution

Cannot unregister storage provider

Hey folks,

I can't seem to unregister an Hitachi VASA storage provider.   We're using VCenter Appliance version 6.5 (build 8667236).

I've tried a sync, rescan, unregister... but with no success.

Here's what I see in the logs (/var/log/vmware/vmware-sps/sps.log):

2018-06-14T12:36:30.439-04:00 [pool-22-thread-5] DEBUG opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.vim.sms.provider.vasa.VasaProviderImpl - Received API Call: sync

2018-06-14T12:36:30.439-04:00 [pool-22-thread-5] INFO  opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.vim.sms.provider.vasa.VasaProviderImpl - VasaProviderImpl.scheduleSyncTask for vpId=49a83597-d674-4097-b718-85de18b9b8ab, arrayId=null and syncType=PROVIDER

2018-06-14T12:36:30.439-04:00 [pool-22-thread-5] DEBUG opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.vim.sms.provider.vasa.VasaProviderImpl - Validating VP 49a83597-d674-4097-b718-85de18b9b8ab schemas

2018-06-14T12:36:30.439-04:00 [pool-22-thread-5] ERROR opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.vim.sms.policy.SyncPolicyManager - Provider 49a83597-d674-4097-b718-85de18b9b8ab is not active!!!

2018-06-14T12:36:30.439-04:00 [pool-22-thread-5] ERROR opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.sps.vp.VasaProviderValidatorImpl - Unable to retrieve capability schemas from VP 49a83597-d674-4097-b718-85de18b9b8ab

com.vmware.vim.sms.fault.ProviderNotActiveException: Provider 49a83597-d674-4097-b718-85de18b9b8ab is not active!!!

at com.vmware.vim.sms.policy.SyncPolicyManager.getProviderAndCheckIsActive(SyncPolicyManager.java:126)

at com.vmware.vim.sms.policy.SyncPolicyManager.queryCapabilitySchema(SyncPolicyManager.java:86)

at com.vmware.vim.sms.policy.PolicyManagerImpl.queryCapabilitySchema(PolicyManagerImpl.java:270)

at com.vmware.sps.vp.VasaProviderValidatorImpl.retrieveCapabilitySchemas(VasaProviderValidatorImpl.java:90)

at com.vmware.sps.vp.VasaProviderValidatorImpl.validate(VasaProviderValidatorImpl.java:67)

at com.vmware.vim.sms.provider.vasa.validation.VasaProviderValidationManagerImpl.validateVasaProvider(VasaProviderValidationManagerImpl.java:87)

at com.vmware.vim.sms.provider.vasa.VasaProviderImpl.validateSchemas(VasaProviderImpl.java:1883)

at com.vmware.vim.sms.provider.vasa.VasaProviderImpl.scheduleSyncTask(VasaProviderImpl.java:1924)

at com.vmware.vim.sms.provider.vasa.VasaProviderImpl.sync(VasaProviderImpl.java:1700)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65)

at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

2018-06-14T12:36:30.439-04:00 [pool-22-thread-5] ERROR opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.vim.sms.provider.vasa.VasaProviderImpl - Validation for VP 49a83597-d674-4097-b718-85de18b9b8ab failed with error com.vmware.vim.sms.fault.ProviderValidationException: com.vmware.vim.sms.fault.ProviderNotActiveException: Provider 49a83597-d674-4097-b718-85de18b9b8ab is not active!!!, setting status to syncError and also re-scheduling the timer sync

2018-06-14T12:36:30.440-04:00 [pool-22-thread-5] DEBUG opId=StorageProvidersViewMediator-apply-17929-ngc:70002804 com.vmware.vim.sms.provider.vasa.VasaProviderImpl - [setStatus] For provider https://132.207.180.48:50001/version.xml, new status is equal to the current status : syncError

2018-06-14T12:36:30.545-04:00 [pool-24-thread-6] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17930-ngc:70002805 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...

2018-06-14T12:36:30.552-04:00 [pool-22-thread-6] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17930-ngc:70002805 com.vmware.vim.sms.StorageManagerImpl - [queryArray] Received API Call: queryArray

2018-06-14T12:36:30.552-04:00 [pool-22-thread-6] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17930-ngc:70002805 com.vmware.vim.sms.query.QueryManagerImpl - [getStorageArrays] Fetching for providerIds null

2018-06-14T12:36:30.562-04:00 [pool-24-thread-7] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17931-ngc:70002805 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...

2018-06-14T12:36:30.568-04:00 [pool-22-thread-7] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17931-ngc:70002805 com.vmware.vim.storage.common.util.SimpleTimeCounter - TIMER STARTED: queryProvider

2018-06-14T12:36:30.568-04:00 [pool-22-thread-7] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17931-ngc:70002805 com.vmware.vim.sms.provider.ProviderCache - [queryProvider] Returning result of size - 4

2018-06-14T12:36:30.569-04:00 [pool-22-thread-7] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17931-ngc:70002805 com.vmware.vim.storage.common.util.SimpleTimeCounter - TIMER STOPPED: queryProvider, TIME TAKEN: 1 ms.

2018-06-14T12:36:30.579-04:00 [pool-24-thread-8] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17932-ngc:70002805 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...

2018-06-14T12:36:30.588-04:00 [pool-24-thread-9] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17933-ngc:70002805 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...

2018-06-14T12:36:30.598-04:00 [pool-24-thread-10] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17934-ngc:70002805 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...

2018-06-14T12:36:30.608-04:00 [pool-24-thread-1] DEBUG opId=dam-auto-generated: StorageProvidersViewMediator:dr-1311:SmsPropertyProvider:200127:5198-17935-ngc:70002805 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...

2018-06-14T12:36:55.698-04:00 [pool-13-thread-1] DEBUG opId=sps-SHPoller-992235-84 com.vmware.vim.sms.health.ServiceHealthPoller - Certificate expiry check succeeded!

Anyone have any ideas?  I opened a support case with VMWare... but almost days later, I've not yet spoken to a storage engineer.

Thank You!

0 Kudos
1 Solution

Accepted Solutions
scylla
Enthusiast
Enthusiast
‎08-02-2018 12:10 PM
Jump to solution

Hello,

A VMware engineer contacted me this morning with a working solution...  I'm posting it here because this procedure is only available internally at VMware and I'm told that it is unlikely that it will be published for public consumption for some reason.

  • Put your VASA provider in a state where it cannot be reached by the VCenter Appliance.   In my case, I did a shutdown.
  • Edit this file:  /usr/lib/vmware-vpx/sps/conf/sms-config.xml and modify this section:

      <!-- Retry configuration for QS connection -->

      <qsConnection>

         <!-- Retry attempt for QS connection. -->

         <retryAttempt>99999</retryAttempt>

         <!-- Retry interval (in seconds) for QS connection. -->

         <retryInterval>30</retryInterval>

      </qsConnection>

With:

      <!-- Retry configuration for QS connection -->

      <qsConnection>

         <!-- Retry attempt for QS connection. -->

         <retryAttempt>4</retryAttempt>

         <!-- Retry interval (in seconds) for QS connection. -->

         <retryInterval>30</retryInterval>

      </qsConnection>

Then STOP and START the SPS service:

service-control  --stop vmware-sps

service-control  --start vmware-sps

Do a 'tail -f' of the SPS logs and wait until you get a 'GREEN' status that looks like this:

2018-08-02T11:23:51.010-04:00 [http-nio-127.0.0.1-22000-exec-2] DEBUG opId=sps-Main-223274-27 com.vmware.vim.storage.common.health.HealthStatusManagerImpl - Health color of the service : GREEN

  • Go to your VCenter console and remove the Storage Provider
  • Re-edit the above file and replace the value of '4' with the original '99999'
  • STOP and START the SPS service

This procedure worked for me.  I was able to re-add the Service Provider and the VVol's it contained.

I hope that this helps someone else!

View solution in original post

7 Replies
danthrojc
Contributor
Contributor
‎07-04-2018 10:03 AM
Jump to solution

I'm having the same issue.  Were you able to get this resolved?

0 Kudos
scylla
Enthusiast
Enthusiast
‎07-04-2018 10:22 AM
Jump to solution

No, still unresolved...  I have a ticket opened with Hitachi and VMware.   I've been waiting for two weeks now for someone to come up with a Solution.     We may have to re-create from scratch our VCenter Appliance!

0 Kudos
danthrojc
Contributor
Contributor
‎07-10-2018 03:56 PM
Jump to solution

I worked with VMware support and got a solution.  Hopefully this helps you.

- Logged in to the following link https://<VC-IP>/sms/mob
- Clicked on QueryStorageManager link
- Click on Invoke method, then on the "storageManager" link in the Method Invocation Result
- Click on the QueryProvider link and Invoke method
- Find the vasaProvider-XXX link (it should be only one) in the Method Invocation Result for your provider and click on it
- Click on QueryProviderInfo link and Invoke Method
- Search the result for uid field:  Made a note of the result

- Opened again the QueryStorageManager link (see step 1 and step 2)
- Clicked on UnregisterProvider_Task link
- filled in the provider uid from step 6) and click on the invoke method
- Restarted the vCenter services

-  This time click on unregister provider and the storage provider was removed.



0 Kudos
scylla
Enthusiast
Enthusiast
‎07-10-2018 04:54 PM
Jump to solution

Unfortunately this didn't work...

Thanks for trying though!

0 Kudos
Raj1988
Enthusiast
Enthusiast
‎07-11-2018 11:47 AM
Jump to solution

Stop and start the sps service and follow again what danthrojc​ mentioned  .

Also did the invoke method give an output .

0 Kudos
scylla
Enthusiast
Enthusiast
‎07-11-2018 01:04 PM
Jump to solution

I redid it again and then did a stop/start sps service...  I then tried to unregister, again without success.

Here's the result of the 'UnregisterProvider_Task':

Method Invocation Result: ManagedObjectReference

NameTypeValue
dynamicPropertyArrayOfDynamicPropertyUnset
dynamicTypestringUnset
namestringReturn value
valanyTypeSmsTask-52
0 Kudos
scylla
Enthusiast
Enthusiast
‎08-02-2018 12:10 PM
Jump to solution

Hello,

A VMware engineer contacted me this morning with a working solution...  I'm posting it here because this procedure is only available internally at VMware and I'm told that it is unlikely that it will be published for public consumption for some reason.

  • Put your VASA provider in a state where it cannot be reached by the VCenter Appliance.   In my case, I did a shutdown.
  • Edit this file:  /usr/lib/vmware-vpx/sps/conf/sms-config.xml and modify this section:

      <!-- Retry configuration for QS connection -->

      <qsConnection>

         <!-- Retry attempt for QS connection. -->

         <retryAttempt>99999</retryAttempt>

         <!-- Retry interval (in seconds) for QS connection. -->

         <retryInterval>30</retryInterval>

      </qsConnection>

With:

      <!-- Retry configuration for QS connection -->

      <qsConnection>

         <!-- Retry attempt for QS connection. -->

         <retryAttempt>4</retryAttempt>

         <!-- Retry interval (in seconds) for QS connection. -->

         <retryInterval>30</retryInterval>

      </qsConnection>

Then STOP and START the SPS service:

service-control  --stop vmware-sps

service-control  --start vmware-sps

Do a 'tail -f' of the SPS logs and wait until you get a 'GREEN' status that looks like this:

2018-08-02T11:23:51.010-04:00 [http-nio-127.0.0.1-22000-exec-2] DEBUG opId=sps-Main-223274-27 com.vmware.vim.storage.common.health.HealthStatusManagerImpl - Health color of the service : GREEN

  • Go to your VCenter console and remove the Storage Provider
  • Re-edit the above file and replace the value of '4' with the original '99999'
  • STOP and START the SPS service

This procedure worked for me.  I was able to re-add the Service Provider and the VVol's it contained.

I hope that this helps someone else!