VMware Cloud Community
crabanus
Enthusiast
Enthusiast
Jump to solution

vSphere 6u2 and EMC UnityVSA and VVOLs -> Protocol Endpoints are not created -> Certificate problem: ESXi doesn' like UnityVSA (updated)

Hi all,

I've just installed UnityVSA from EMC (Version 4.0.1) in my lab with vSphere 6u2.

I've created an old fashioned VMFS-Datastore on a LUN and an NFS-Datastore on a share. Everything is working fine.

Then I created using Unisphere a Datastore for blockbased VVOLs (via iSCIS) and for filebased VVOLs (via NFS). Everything went fine in the UnityVSA.

Then I added the StorageProvider in vCenter - also this workes without error.

BUT: There are no PEs created on the vSphere.

The PEs are present on the UnityVSA (output from uemcli executed directly on the UnityVSA):

18:25:40 service@VIRT1636W2LHJW-spa spa:~> uemcli -d localhost -u admin -securePassword /stor/prov/vmware/pe show

Password:

Storage system address: localhost

Storage system port: 443

HTTPS connection

1:    ID                   = rfc4122.b835c8dd-5cc2-48d8-9bb6-4e1483c35ecf

      Name                 = nas_pe_3

      Type                 = NAS

      VMware UUID          = rfc4122.b835c8dd-5cc2-48d8-9bb6-4e1483c35ecf

      Export path          = 192.168.4.33:/rfc4122.3c70af02-00ad-4d9e-be70-16fe20fe07e7

      IP address           = 192.168.4.33

      WWN                  =

      Default SP           =

      Current SP           =

      NAS Server           = nas_1

      VMware NAS PE server = PES_1

      VVol datastore       = res_9

      Host                 =

      Health state         = OK (5)

      Health details       = "The protocol endpoint is operating normally. No action is required."

2:    ID                   = rfc4122.60060160-b2ed-2021-a932-d2f0067c0253

      Name                 = scsi_pe_13

      Type                 = SCSI

      VMware UUID          = rfc4122.60060160-b2ed-2021-a932-d2f0067c0253

      Export path          =

      IP address           =

      WWN                  = 60:06:01:60:B2:ED:20:21:A9:32:D2:F0:06:7C:02:53

      Default SP           = SPA

      Current SP           = SPA

      NAS Server           =

      VMware NAS PE server =

      VVol datastore       =

      Host                 = Host_1

      Health state         = OK (5)

      Health details       = "The protocol endpoint is operating normally. No action is required."

3:    ID                   = rfc4122.60060160-b2ed-2021-a2df-e66d9f94b2d1

      Name                 = scsi_pe_14

      Type                 = SCSI

      VMware UUID          = rfc4122.60060160-b2ed-2021-a2df-e66d9f94b2d1

      Export path          =

      IP address           =

      WWN                  = 60:06:01:60:B2:ED:20:21:A2:DF:E6:6D:9F:94:B2:D1

      Default SP           = SPA

      Current SP           = SPA

      NAS Server           =

      VMware NAS PE server =

      VVol datastore       =

      Host                 = Host_2

      Health state         = OK (5)

      Health details       = "The protocol endpoint is operating normally. No action is required."

4:    ID                   = rfc4122.60060160-b2ed-2021-b146-73a102b508cf

      Name                 = scsi_pe_15

      Type                 = SCSI

      VMware UUID          = rfc4122.60060160-b2ed-2021-b146-73a102b508cf

      Export path          =

      IP address           =

      WWN                  = 60:06:01:60:B2:ED:20:21:B1:46:73:A1:02:B5:08:CF

      Default SP           = SPA

      Current SP           = SPA

      NAS Server           =

      VMware NAS PE server =

      VVol datastore       =

      Host                 = Host_3

      Health state         = OK (5)

      Health details       = "The protocol endpoint is operating normally. No action is required."

19:02:21 service@VIRT1636W2LHJW-spa spa:~>

Also the appropriate LUNs for the blockbased VVOLs are created and provisioned (snipplet form vmkernel.log from one of the ESXi; the iSCSI-LUNs are available via two paths):

2016-09-09T18:38:00.810Z cpu3:32925)ScsiPath: 604: Path vmhba33:C0:T3:L1023 is a VVol PE (ver:6)

2016-09-09T18:38:00.811Z cpu1:32924)ScsiPath: 604: Path vmhba33:C0:T2:L1023 is a VVol PE (ver:6)

I can create the VVOL-Datastore from vSphere Web Client, the appropriate Storage Containers (i.e. "Datastores" from UnityVSA) are presented. But as there are no PEs, the Datastores are unaccessible and show a size of 0 bytes (output from PowerCLI):

PowerCLI C:\> get-datastore vvol*

Name                               FreeSpaceGB      CapacityGB

----                               -----------      ----------

vvol-nfs-vsa-1                           0,000           0,000

vvol-iscsi-vsa-1                         0,000           0,000

PowerCLI C:\>

By the way, the vvold on the ESXi ist not running, it says, that there is no VVOL-configuration.

[/etc/init.d/vvold: /etc/init.d/vvold start, called by pid 43665]

[/etc/init.d/vvold: vvold max reserve memory set to 200]

2016-09-09T20:20:15.054Z Section for VMware ESX, pid=43696, version=6.0.0, build=4192238, option=Release

------ Early init logs start --------

2016-09-09T20:20:15.052Z info -[FFD82350] [Originator@6876 sub=Default] Successfully registered SIGHUP handler

2016-09-09T20:20:15.052Z info -[FFD82350] [Originator@6876 sub=Default] Successfully registered SIGPIPE handler

2016-09-09T20:20:15.052Z info -[FFD82350] [Originator@6876 sub=Default] Successfully registered SIGTERM handler

------ Early init logs end   --------

2016-09-09T20:20:15.054Z info vvold[FFD82350] [Originator@6876 sub=Default] Logging uses fast path: true

2016-09-09T20:20:15.054Z info vvold[FFD82350] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore

2016-09-09T20:20:15.054Z info vvold[FFD82350] [Originator@6876 sub=Default] Initialized channel manager

2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=Default] Current working directory: /var/log/vmware

2016-09-09T20:20:15.055Z info vvold[FFDC3B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-09-09T20:20:15.055Z info vvold[FFE04B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-09-09T20:20:15.055Z info vvold[FFE45B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=ThreadPool] Thread pool on asio: Min Io, Max Io, Min Task,

Max Task, Max Concurency: 2, 22, 2, 52, 2147483647

2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-09-09T20:20:15.055Z info vvold[FFE86B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=Default] Syscommand enabled: true

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] ReaperManager Initialized

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] Initalized App with config file:/etc/vmware/vvold/config.xml

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] Listening on port 8090 (NOT using SSL) using version 'vvol.version.version1'.

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] Initializing SOAP tcp adapter

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default.HTTPService] Using default for nonChunkingAgents: 'VMware VI Client|VMware-client|VMware-client/3.*'

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default.HTTPService] Using default for agentsNeedingContentLength: 'VMware-client'

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default.HTTPService] Max buffered response size is 104857600 bytes

2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] enableChunkedResponses: true

2016-09-09T20:20:15.057Z info vvold[FFD82350] [Originator@6876 sub=Libs] UUID: Running in UW, but cannot verify vmk syscall version, giving up.

2016-09-09T20:20:15.057Z info vvold[FFD82350] [Originator@6876 sub=Libs] UUID: Valid gethostid routine. Value = A8C01704.

2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=Default] Creating SOAP body handler for version 'vvol.version.version1'

2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=SOAP-1] Created SOAP body handler for vvol.version.version1 (vvol/1.0)

2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=Default] Creating SOAP body handler for internal version 'vvol.version.version1'

2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=SOAP-2] Created SOAP body handler for vvol.version.version1 (internalvvol/1.0)

2016-09-09T20:20:15.066Z error vvold[FFD82350] [Originator@6876 sub=Default] VVold SI:main, no VVol config available, exiting

[/etc/init.d/vvold: /etc/init.d/vvold stopnomemclear, called by pid 43701]

[/etc/init.d/vvold: vvold stopped.]

[/etc/init.d/vvold: WaitVvoldToComeUp /var/run/vmware/.vmware-vvol.started created]

[/etc/init.d/vvold: vvold stopped after start!]

[/etc/init.d/vvold: /var/run/vmware/.vmware-vvol.started is not created]

[/etc/init.d/vvold: Successfully cleared vvold memory reservation]

With "esxcli storage vvol vasaprovider list" I don't get a Storage Provider listed - although it's registered in vCenter (and listed as online vie the Web Client).

Anything I should be missing, but at the moment, I don't have any idea to go further. Any help very appreciated.

Best regards,

Christian

Reply
0 Kudos
1 Solution

Accepted Solutions
crabanus
Enthusiast
Enthusiast
Jump to solution

Hi all, I know, it's been a while since the last post in this thread has been done - but perhaps I have a solution now and I'd like to share it.

First of all: In the meanwhile I'm working with vSphere 6.5 and VSA 4.1.2. I started over from scatch, faced exactly the same problems as when I originally created this thread. In vSphere 6.5 there is the possibility to provide a certificate from the storage array; it's stored as /EMC/backend/CEM/vasa.ssl/vc1/servercert-1.crt on the VSA and can be pasted into a textfile and the provided to vSphere when the Storage Provider is registered. When you do so, no warning about the certificate appears. But even with this certificate the PE didn't show up.

Then I reviewed the documentation for the 1000th time - and suddenly I realized that the username of a local user has to be provided in the form "local/<username>"!!! I always has been using only "<username>" (and of course, there has been no error message about the credential :-((( ). And this did the trick. The PEs appeared and the VVOL-DSs are not anly longer unaccessible.

Well, at least the iSCSI-VVOL-DS is working now, the NFS-VVOL-DS shows up with correct size, but I cannot write to it. But I think, this is another story.

I haven't doublechecked, whether this also works in the original environment (vSphere 6u2 and VSA 3x), but I'd think so...

Best regards,

Christian

View solution in original post

Reply
0 Kudos
9 Replies
crabanus
Enthusiast
Enthusiast
Jump to solution

Hi all,

after hours and hours I found out, that this seems to be a certificate error (unityvsa-1 is my UnityVSA):

2016-09-12T12:58:48.222Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols called

2016-09-12T12:58:48.222Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols done for 0 VVols

2016-09-12T12:58:59.937Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://unityvsa-1.vs6dc.local:8443/vasa/version.xml

2016-09-12T12:58:59.946Z error vvold[FFB80B70] [Originator@6876 sub=HttpConnectionPool-000000] [ConnectComplete] Connect failed to <cs p:1f0d93e0, TCP:unityvsa-1.vs6dc.local:8443>; cnx: (null), error: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: D9:EE:74:96:AE:83:58:2B:3B:8D:31:5A:0D:DE:BD:69:17:39:BF:11

--> ExpectedThumbprint:

--> ExpectedPeerName: unityvsa-1.vs6dc.local

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.)

2016-09-12T12:58:59.947Z warning vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:

--> PeerThumbprint: D9:EE:74:96:AE:83:58:2B:3B:8D:31:5A:0D:DE:BD:69:17:39:BF:11

--> ExpectedThumbprint:

--> ExpectedPeerName: unityvsa-1.vs6dc.local

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate., using default

2016-09-12T12:58:59.947Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty

2016-09-12T12:58:59.947Z warning vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (UnityVSA-1)!

2016-09-12T12:58:59.947Z info vvold[FF9FBB70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://unityvsa-1.vs6dc.local:8443/vasa/version.xml

2016-09-12T12:58:59.947Z error vvold[FF9FBB70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP UnityVSA-1 state: 0

How can I resolve this? How can I tell my ESXi hosts to talk to the Unity-System?

When I now create a list of the registered VASA-providers on the ESXi host, at least the provider appears, but it's not synched (most likely because of the certificate proboem):

[root@win-esxi-1:~] esxcli storage vvol vasaprovider list

UnityVSA-1

   VP Name: UnityVSA-1

   URL: https://unityvsa-1.vs6dc.local:8443/vasa/version.xml

   Status: syncError

   Arrays:

         Array Id: EMC:VIRT1636W2LHJW

         Is Active: true

         Priority: 0

[root@win-esxi-1:~]

Any ideas? Am I really the only one who has this problem?!?

Best regards,

Christian

Reply
0 Kudos
Bill_Oyler
Hot Shot
Hot Shot
Jump to solution

Hello,

I am having the exact same problem.  I am running the UnityVSA v4.0.1 with ESXi 6.0U2.  I have iSCSI LUNs working great.  I followed all of the VVOL configuration steps documented by EMC and VMware.  My VASA provider for Unity looks great in the Web Client, but I see no Protocol Endpoints showing up in the vSphere Web Client.  (PE's show up perfectly on the Unity side.)  SSH output is the same as yours with "syncError" for the "Status":

[root@labXXXesx301:~] esxcli storage vvol vasaprovider list

emc-unity.XXX.lab

   VP Name: emc-unity.XXX.lab

   URL: https://emc-unity.XXX.lab:8443/vasa/version.xml

  Status: syncError

   Arrays:

         Array Id: EMC:VIRT1619XYV9HW

         Is Active: true

         Priority: 0

Our vCenter is running Windows Server 2012 R2, and we are using an external PSC, also running Windows Server 2012 R2.  I checked the usual troubleshooting steps -- NTP is syncing time identically across hosts and the UnityVSA.  Also, the UnityVSA is only registered to (1) of our (3) vCenters.  Our vCenters are in "Enhanced Linked Mode" so they all show up in the Web Client, but only one of them is showing the UnityVSA VASA provider, which is good.

I also tried to update the certificate using the "Refresh the certificate" option in the Web Client.  This gave me a certificate valid for 365 days, but that doesn't seem to help.  I've tried numerous "Rescan" and "Synchronize" options for the VASA provider, and also rebooted the vCenter, but the "syncError" message seems to persist, and the Protocol Endpoints do not show up.... and vVol datastore remains read-only.

Anyone else able to get vVols to work on Unity with iSCSI???

Bill Oyler Systems Engineer
Reply
0 Kudos
Bill_Oyler
Hot Shot
Hot Shot
Jump to solution

OK, so I got to thinking about the certificate name mismatch.  I looked at the actual SSL certificate issued to the Unity by the VMware PSC, and the Subject Alternative Name field had the IP address of the Unity management interface.  So I tried replacing the FQDN with the IP address of the UnityVSA in the URL field, and performed a few Rescan operations, and BINGO!  The Protocol Endpoints showed up on both hosts, and the "status" is now "online" via SSH:

[root@labXXXesx301:~] esxcli storage vvol vasaprovider list

emc-unity.XXX.lab

   VP Name: emc-unity.XXX.lab

   URL: https://10.201.30.184:8443/vasa/version.xml

   Status: online

   Arrays:

         Array Id: EMC:VIRT1619XYV9HW

         Is Active: true

         Priority: 0

So I guess the moral of the story is... don't use FQDN since the PSC doesn't use FQDN when it creates the certificate.  (I suppose we could alternatively create our own certificate, or export the EMC-generated certificate off the VSA....)

Bill Oyler Systems Engineer
crabanus
Enthusiast
Enthusiast
Jump to solution

Hi Bill,

thank you for your replay - and sorry, that it took so long time for me to respond. I've been busy with other things...

So back to your proposal: I'm glad that using the shortname is working for you - unfortunately not for me. I tried to use the short name, but I still got the same problem. The vasaprovider seems to be installed fine, it's appearing as online in the vCenter, the LUNs for the PEs are available, but not touched. I don't get a PE on the ESXi systems.

Anyone any additional ideas?

Best regards,

Christian

Reply
0 Kudos
Bill_Oyler
Hot Shot
Hot Shot
Jump to solution

For me to get VVols to work, it wasn't the "short name" that I needed to use, but rather the IP address, i.e.:


https://10.201.30.184:8443/vasa/version.xml

VVols and policies were working great for several weeks, now for some reason my storage-policy-based-management (SPBM) isn't working in vCenter but I think it's a vCenter thing and not a Unity thing.  Until I get that figured out, my polices are currently broken, but VVol VMs continue to remain running.  Not sure if it's certificate related again, or something else, but I must say VVol troubleshooting is NOT fun.  Smiley Sad

I do see all Protocol Endpoint LUNs (LUN 1023, etc.) on my ESXi hosts, and the CLI output shows the VASA provider communication working, and that was fixed after I started using IP address for the VASA provider URL.

Hope this helps.

Bill

Bill Oyler Systems Engineer
Reply
0 Kudos
anthonybailey
Contributor
Contributor
Jump to solution

I'm on a Unity 300, and I can confirm that using IP instead of DNS name changed the status from "syncError" to "online". Incidentally, this is the way EMC documentation says you should do it, but I didn't think the reason for that was to get around certificate errors.

Reply
0 Kudos
crabanus
Enthusiast
Enthusiast
Jump to solution

Hi all, I know, it's been a while since the last post in this thread has been done - but perhaps I have a solution now and I'd like to share it.

First of all: In the meanwhile I'm working with vSphere 6.5 and VSA 4.1.2. I started over from scatch, faced exactly the same problems as when I originally created this thread. In vSphere 6.5 there is the possibility to provide a certificate from the storage array; it's stored as /EMC/backend/CEM/vasa.ssl/vc1/servercert-1.crt on the VSA and can be pasted into a textfile and the provided to vSphere when the Storage Provider is registered. When you do so, no warning about the certificate appears. But even with this certificate the PE didn't show up.

Then I reviewed the documentation for the 1000th time - and suddenly I realized that the username of a local user has to be provided in the form "local/<username>"!!! I always has been using only "<username>" (and of course, there has been no error message about the credential :-((( ). And this did the trick. The PEs appeared and the VVOL-DSs are not anly longer unaccessible.

Well, at least the iSCSI-VVOL-DS is working now, the NFS-VVOL-DS shows up with correct size, but I cannot write to it. But I think, this is another story.

I haven't doublechecked, whether this also works in the original environment (vSphere 6u2 and VSA 3x), but I'd think so...

Best regards,

Christian

Reply
0 Kudos
crabanus
Enthusiast
Enthusiast
Jump to solution

Just a quick last update: After waiting about 30 Minutes, also the NFS-VVOL-DS is working. Strange, but finally I'm happy with VVOLs working!

Reply
0 Kudos
rszymczak
Hot Shot
Hot Shot
Jump to solution

For anyone who is faceing this issue and the answer provided (local/<username>) thing didn't help: it's not your fault. It's the lousy EMC bootstrap script that fails to create a self-signed certificate (op mentioned the cert location, you can check it there but UnityVSA won't let you touch it without root permissions, which we don't have).

So, what did the trick in my case:

1. Create a new, VALID PEM formated certificate. I used a RSA 2048 key

Note: the "generated" self-signed cert of the UnityVSA was outdated (2015) and so vCenter wouldn't accept it EVEN if you tell vCenter to trust the cert. Which is just what should happen if a cert is outdated.

2. Enable SSH acces on UnityVSA: from the HTML UI go to service --> service tasks and --> enable ssh

3. Login to UnityVSA using the account service (you set the password for it at install time (default: same as the admin password)

4. Upload your PEM formated certificate named something like mycert.crt (crt extention is important) to UnityVSA. Make sure to contain the certificate chain in the certificate if it's not self-signed.

5. Upload your PEM formated private key for the certificate name something like mycert.pk (pk extention is important) to UnityVSA

6. On the UnityVSA run the command  svc_custom_cert mycert

This will update the UnityVSA certificate and restart the UnityVSA webserver. The Webcertificate is also used for the VASA provider (since the initial VASA connection in UnityVSA is just a connection to http:// on port 8443, which is using the same cert as on port 443).

After I updated the certificate I was able to register the UnifyVSA VASA provider. Note that this was the latest version of UnityVSA as of today (4.2.0.9392909) on vCenter 6.5U1 and with ESXi 6.5U1.

Reply
0 Kudos