Hi all,
I've just installed UnityVSA from EMC (Version 4.0.1) in my lab with vSphere 6u2.
I've created an old fashioned VMFS-Datastore on a LUN and an NFS-Datastore on a share. Everything is working fine.
Then I created using Unisphere a Datastore for blockbased VVOLs (via iSCIS) and for filebased VVOLs (via NFS). Everything went fine in the UnityVSA.
Then I added the StorageProvider in vCenter - also this workes without error.
BUT: There are no PEs created on the vSphere.
The PEs are present on the UnityVSA (output from uemcli executed directly on the UnityVSA):
18:25:40 service@VIRT1636W2LHJW-spa spa:~> uemcli -d localhost -u admin -securePassword /stor/prov/vmware/pe show
Password:
Storage system address: localhost
Storage system port: 443
HTTPS connection
1: ID = rfc4122.b835c8dd-5cc2-48d8-9bb6-4e1483c35ecf
Name = nas_pe_3
Type = NAS
VMware UUID = rfc4122.b835c8dd-5cc2-48d8-9bb6-4e1483c35ecf
Export path = 192.168.4.33:/rfc4122.3c70af02-00ad-4d9e-be70-16fe20fe07e7
IP address = 192.168.4.33
WWN =
Default SP =
Current SP =
NAS Server = nas_1
VMware NAS PE server = PES_1
VVol datastore = res_9
Host =
Health state = OK (5)
Health details = "The protocol endpoint is operating normally. No action is required."
2: ID = rfc4122.60060160-b2ed-2021-a932-d2f0067c0253
Name = scsi_pe_13
Type = SCSI
VMware UUID = rfc4122.60060160-b2ed-2021-a932-d2f0067c0253
Export path =
IP address =
WWN = 60:06:01:60:B2:ED:20:21:A9:32:D2:F0:06:7C:02:53
Default SP = SPA
Current SP = SPA
NAS Server =
VMware NAS PE server =
VVol datastore =
Host = Host_1
Health state = OK (5)
Health details = "The protocol endpoint is operating normally. No action is required."
3: ID = rfc4122.60060160-b2ed-2021-a2df-e66d9f94b2d1
Name = scsi_pe_14
Type = SCSI
VMware UUID = rfc4122.60060160-b2ed-2021-a2df-e66d9f94b2d1
Export path =
IP address =
WWN = 60:06:01:60:B2:ED:20:21:A2:DF:E6:6D:9F:94:B2:D1
Default SP = SPA
Current SP = SPA
NAS Server =
VMware NAS PE server =
VVol datastore =
Host = Host_2
Health state = OK (5)
Health details = "The protocol endpoint is operating normally. No action is required."
4: ID = rfc4122.60060160-b2ed-2021-b146-73a102b508cf
Name = scsi_pe_15
Type = SCSI
VMware UUID = rfc4122.60060160-b2ed-2021-b146-73a102b508cf
Export path =
IP address =
WWN = 60:06:01:60:B2:ED:20:21:B1:46:73:A1:02:B5:08:CF
Default SP = SPA
Current SP = SPA
NAS Server =
VMware NAS PE server =
VVol datastore =
Host = Host_3
Health state = OK (5)
Health details = "The protocol endpoint is operating normally. No action is required."
19:02:21 service@VIRT1636W2LHJW-spa spa:~>
Also the appropriate LUNs for the blockbased VVOLs are created and provisioned (snipplet form vmkernel.log from one of the ESXi; the iSCSI-LUNs are available via two paths):
2016-09-09T18:38:00.810Z cpu3:32925)ScsiPath: 604: Path vmhba33:C0:T3:L1023 is a VVol PE (ver:6)
2016-09-09T18:38:00.811Z cpu1:32924)ScsiPath: 604: Path vmhba33:C0:T2:L1023 is a VVol PE (ver:6)
I can create the VVOL-Datastore from vSphere Web Client, the appropriate Storage Containers (i.e. "Datastores" from UnityVSA) are presented. But as there are no PEs, the Datastores are unaccessible and show a size of 0 bytes (output from PowerCLI):
PowerCLI C:\> get-datastore vvol*
Name FreeSpaceGB CapacityGB
---- ----------- ----------
vvol-nfs-vsa-1 0,000 0,000
vvol-iscsi-vsa-1 0,000 0,000
PowerCLI C:\>
By the way, the vvold on the ESXi ist not running, it says, that there is no VVOL-configuration.
[/etc/init.d/vvold: /etc/init.d/vvold start, called by pid 43665]
[/etc/init.d/vvold: vvold max reserve memory set to 200]
2016-09-09T20:20:15.054Z Section for VMware ESX, pid=43696, version=6.0.0, build=4192238, option=Release
------ Early init logs start --------
2016-09-09T20:20:15.052Z info -[FFD82350] [Originator@6876 sub=Default] Successfully registered SIGHUP handler
2016-09-09T20:20:15.052Z info -[FFD82350] [Originator@6876 sub=Default] Successfully registered SIGPIPE handler
2016-09-09T20:20:15.052Z info -[FFD82350] [Originator@6876 sub=Default] Successfully registered SIGTERM handler
------ Early init logs end --------
2016-09-09T20:20:15.054Z info vvold[FFD82350] [Originator@6876 sub=Default] Logging uses fast path: true
2016-09-09T20:20:15.054Z info vvold[FFD82350] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore
2016-09-09T20:20:15.054Z info vvold[FFD82350] [Originator@6876 sub=Default] Initialized channel manager
2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=Default] Current working directory: /var/log/vmware
2016-09-09T20:20:15.055Z info vvold[FFDC3B70] [Originator@6876 sub=ThreadPool] Thread enlisted
2016-09-09T20:20:15.055Z info vvold[FFE04B70] [Originator@6876 sub=ThreadPool] Thread enlisted
2016-09-09T20:20:15.055Z info vvold[FFE45B70] [Originator@6876 sub=ThreadPool] Thread enlisted
2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=ThreadPool] Thread pool on asio: Min Io, Max Io, Min Task,
Max Task, Max Concurency: 2, 22, 2, 52, 2147483647
2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=ThreadPool] Thread enlisted
2016-09-09T20:20:15.055Z info vvold[FFE86B70] [Originator@6876 sub=ThreadPool] Thread enlisted
2016-09-09T20:20:15.055Z info vvold[FFD82350] [Originator@6876 sub=Default] Syscommand enabled: true
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] ReaperManager Initialized
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] Initalized App with config file:/etc/vmware/vvold/config.xml
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] Listening on port 8090 (NOT using SSL) using version 'vvol.version.version1'.
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] Initializing SOAP tcp adapter
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default.HTTPService] Using default for nonChunkingAgents: 'VMware VI Client|VMware-client|VMware-client/3.*'
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default.HTTPService] Using default for agentsNeedingContentLength: 'VMware-client'
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default.HTTPService] Max buffered response size is 104857600 bytes
2016-09-09T20:20:15.056Z info vvold[FFD82350] [Originator@6876 sub=Default] enableChunkedResponses: true
2016-09-09T20:20:15.057Z info vvold[FFD82350] [Originator@6876 sub=Libs] UUID: Running in UW, but cannot verify vmk syscall version, giving up.
2016-09-09T20:20:15.057Z info vvold[FFD82350] [Originator@6876 sub=Libs] UUID: Valid gethostid routine. Value = A8C01704.
2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=Default] Creating SOAP body handler for version 'vvol.version.version1'
2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=SOAP-1] Created SOAP body handler for vvol.version.version1 (vvol/1.0)
2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=Default] Creating SOAP body handler for internal version 'vvol.version.version1'
2016-09-09T20:20:15.058Z info vvold[FFD82350] [Originator@6876 sub=SOAP-2] Created SOAP body handler for vvol.version.version1 (internalvvol/1.0)
2016-09-09T20:20:15.066Z error vvold[FFD82350] [Originator@6876 sub=Default] VVold SI:main, no VVol config available, exiting
[/etc/init.d/vvold: /etc/init.d/vvold stopnomemclear, called by pid 43701]
[/etc/init.d/vvold: vvold stopped.]
[/etc/init.d/vvold: WaitVvoldToComeUp /var/run/vmware/.vmware-vvol.started created]
[/etc/init.d/vvold: vvold stopped after start!]
[/etc/init.d/vvold: /var/run/vmware/.vmware-vvol.started is not created]
[/etc/init.d/vvold: Successfully cleared vvold memory reservation]
With "esxcli storage vvol vasaprovider list" I don't get a Storage Provider listed - although it's registered in vCenter (and listed as online vie the Web Client).
Anything I should be missing, but at the moment, I don't have any idea to go further. Any help very appreciated.
Best regards,
Christian
Hi all, I know, it's been a while since the last post in this thread has been done - but perhaps I have a solution now and I'd like to share it.
First of all: In the meanwhile I'm working with vSphere 6.5 and VSA 4.1.2. I started over from scatch, faced exactly the same problems as when I originally created this thread. In vSphere 6.5 there is the possibility to provide a certificate from the storage array; it's stored as /EMC/backend/CEM/vasa.ssl/vc1/servercert-1.crt on the VSA and can be pasted into a textfile and the provided to vSphere when the Storage Provider is registered. When you do so, no warning about the certificate appears. But even with this certificate the PE didn't show up.
Then I reviewed the documentation for the 1000th time - and suddenly I realized that the username of a local user has to be provided in the form "local/<username>"!!! I always has been using only "<username>" (and of course, there has been no error message about the credential :-((( ). And this did the trick. The PEs appeared and the VVOL-DSs are not anly longer unaccessible.
Well, at least the iSCSI-VVOL-DS is working now, the NFS-VVOL-DS shows up with correct size, but I cannot write to it. But I think, this is another story.
I haven't doublechecked, whether this also works in the original environment (vSphere 6u2 and VSA 3x), but I'd think so...
Best regards,
Christian
Hi all,
after hours and hours I found out, that this seems to be a certificate error (unityvsa-1 is my UnityVSA):
2016-09-12T12:58:48.222Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols called
2016-09-12T12:58:48.222Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VVolUnbindManager::UnbindIdleVVols done for 0 VVols
2016-09-12T12:58:59.937Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://unityvsa-1.vs6dc.local:8443/vasa/version.xml
2016-09-12T12:58:59.946Z error vvold[FFB80B70] [Originator@6876 sub=HttpConnectionPool-000000] [ConnectComplete] Connect failed to <cs p:1f0d93e0, TCP:unityvsa-1.vs6dc.local:8443>; cnx: (null), error: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: D9:EE:74:96:AE:83:58:2B:3B:8D:31:5A:0D:DE:BD:69:17:39:BF:11
--> ExpectedThumbprint:
--> ExpectedPeerName: unityvsa-1.vs6dc.local
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
2016-09-12T12:58:59.947Z warning vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:
--> PeerThumbprint: D9:EE:74:96:AE:83:58:2B:3B:8D:31:5A:0D:DE:BD:69:17:39:BF:11
--> ExpectedThumbprint:
--> ExpectedPeerName: unityvsa-1.vs6dc.local
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate., using default
2016-09-12T12:58:59.947Z info vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty
2016-09-12T12:58:59.947Z warning vvold[FF9FBB70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (UnityVSA-1)!
2016-09-12T12:58:59.947Z info vvold[FF9FBB70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://unityvsa-1.vs6dc.local:8443/vasa/version.xml
2016-09-12T12:58:59.947Z error vvold[FF9FBB70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP UnityVSA-1 state: 0
How can I resolve this? How can I tell my ESXi hosts to talk to the Unity-System?
When I now create a list of the registered VASA-providers on the ESXi host, at least the provider appears, but it's not synched (most likely because of the certificate proboem):
[root@win-esxi-1:~] esxcli storage vvol vasaprovider list
UnityVSA-1
VP Name: UnityVSA-1
URL: https://unityvsa-1.vs6dc.local:8443/vasa/version.xml
Status: syncError
Arrays:
Array Id: EMC:VIRT1636W2LHJW
Is Active: true
Priority: 0
[root@win-esxi-1:~]
Any ideas? Am I really the only one who has this problem?!?
Best regards,
Christian
Hello,
I am having the exact same problem. I am running the UnityVSA v4.0.1 with ESXi 6.0U2. I have iSCSI LUNs working great. I followed all of the VVOL configuration steps documented by EMC and VMware. My VASA provider for Unity looks great in the Web Client, but I see no Protocol Endpoints showing up in the vSphere Web Client. (PE's show up perfectly on the Unity side.) SSH output is the same as yours with "syncError" for the "Status":
[root@labXXXesx301:~] esxcli storage vvol vasaprovider list
emc-unity.XXX.lab
VP Name: emc-unity.XXX.lab
URL: https://emc-unity.XXX.lab:8443/vasa/version.xml
Status: syncError
Arrays:
Array Id: EMC:VIRT1619XYV9HW
Is Active: true
Priority: 0
Our vCenter is running Windows Server 2012 R2, and we are using an external PSC, also running Windows Server 2012 R2. I checked the usual troubleshooting steps -- NTP is syncing time identically across hosts and the UnityVSA. Also, the UnityVSA is only registered to (1) of our (3) vCenters. Our vCenters are in "Enhanced Linked Mode" so they all show up in the Web Client, but only one of them is showing the UnityVSA VASA provider, which is good.
I also tried to update the certificate using the "Refresh the certificate" option in the Web Client. This gave me a certificate valid for 365 days, but that doesn't seem to help. I've tried numerous "Rescan" and "Synchronize" options for the VASA provider, and also rebooted the vCenter, but the "syncError" message seems to persist, and the Protocol Endpoints do not show up.... and vVol datastore remains read-only.
Anyone else able to get vVols to work on Unity with iSCSI???
OK, so I got to thinking about the certificate name mismatch. I looked at the actual SSL certificate issued to the Unity by the VMware PSC, and the Subject Alternative Name field had the IP address of the Unity management interface. So I tried replacing the FQDN with the IP address of the UnityVSA in the URL field, and performed a few Rescan operations, and BINGO! The Protocol Endpoints showed up on both hosts, and the "status" is now "online" via SSH:
[root@labXXXesx301:~] esxcli storage vvol vasaprovider list
emc-unity.XXX.lab
VP Name: emc-unity.XXX.lab
URL: https://10.201.30.184:8443/vasa/version.xml
Status: online
Arrays:
Array Id: EMC:VIRT1619XYV9HW
Is Active: true
Priority: 0
So I guess the moral of the story is... don't use FQDN since the PSC doesn't use FQDN when it creates the certificate. (I suppose we could alternatively create our own certificate, or export the EMC-generated certificate off the VSA....)
Hi Bill,
thank you for your replay - and sorry, that it took so long time for me to respond. I've been busy with other things...
So back to your proposal: I'm glad that using the shortname is working for you - unfortunately not for me. I tried to use the short name, but I still got the same problem. The vasaprovider seems to be installed fine, it's appearing as online in the vCenter, the LUNs for the PEs are available, but not touched. I don't get a PE on the ESXi systems.
Anyone any additional ideas?
Best regards,
Christian
For me to get VVols to work, it wasn't the "short name" that I needed to use, but rather the IP address, i.e.:
https://10.201.30.184:8443/vasa/version.xml
VVols and policies were working great for several weeks, now for some reason my storage-policy-based-management (SPBM) isn't working in vCenter but I think it's a vCenter thing and not a Unity thing. Until I get that figured out, my polices are currently broken, but VVol VMs continue to remain running. Not sure if it's certificate related again, or something else, but I must say VVol troubleshooting is NOT fun.
I do see all Protocol Endpoint LUNs (LUN 1023, etc.) on my ESXi hosts, and the CLI output shows the VASA provider communication working, and that was fixed after I started using IP address for the VASA provider URL.
Hope this helps.
Bill
I'm on a Unity 300, and I can confirm that using IP instead of DNS name changed the status from "syncError" to "online". Incidentally, this is the way EMC documentation says you should do it, but I didn't think the reason for that was to get around certificate errors.
Hi all, I know, it's been a while since the last post in this thread has been done - but perhaps I have a solution now and I'd like to share it.
First of all: In the meanwhile I'm working with vSphere 6.5 and VSA 4.1.2. I started over from scatch, faced exactly the same problems as when I originally created this thread. In vSphere 6.5 there is the possibility to provide a certificate from the storage array; it's stored as /EMC/backend/CEM/vasa.ssl/vc1/servercert-1.crt on the VSA and can be pasted into a textfile and the provided to vSphere when the Storage Provider is registered. When you do so, no warning about the certificate appears. But even with this certificate the PE didn't show up.
Then I reviewed the documentation for the 1000th time - and suddenly I realized that the username of a local user has to be provided in the form "local/<username>"!!! I always has been using only "<username>" (and of course, there has been no error message about the credential :-((( ). And this did the trick. The PEs appeared and the VVOL-DSs are not anly longer unaccessible.
Well, at least the iSCSI-VVOL-DS is working now, the NFS-VVOL-DS shows up with correct size, but I cannot write to it. But I think, this is another story.
I haven't doublechecked, whether this also works in the original environment (vSphere 6u2 and VSA 3x), but I'd think so...
Best regards,
Christian
Just a quick last update: After waiting about 30 Minutes, also the NFS-VVOL-DS is working. Strange, but finally I'm happy with VVOLs working!
For anyone who is faceing this issue and the answer provided (local/<username>) thing didn't help: it's not your fault. It's the lousy EMC bootstrap script that fails to create a self-signed certificate (op mentioned the cert location, you can check it there but UnityVSA won't let you touch it without root permissions, which we don't have).
So, what did the trick in my case:
1. Create a new, VALID PEM formated certificate. I used a RSA 2048 key
Note: the "generated" self-signed cert of the UnityVSA was outdated (2015) and so vCenter wouldn't accept it EVEN if you tell vCenter to trust the cert. Which is just what should happen if a cert is outdated.
2. Enable SSH acces on UnityVSA: from the HTML UI go to service --> service tasks and --> enable ssh
3. Login to UnityVSA using the account service (you set the password for it at install time (default: same as the admin password)
4. Upload your PEM formated certificate named something like mycert.crt (crt extention is important) to UnityVSA. Make sure to contain the certificate chain in the certificate if it's not self-signed.
5. Upload your PEM formated private key for the certificate name something like mycert.pk (pk extention is important) to UnityVSA
6. On the UnityVSA run the command svc_custom_cert mycert
This will update the UnityVSA certificate and restart the UnityVSA webserver. The Webcertificate is also used for the VASA provider (since the initial VASA connection in UnityVSA is just a connection to http:// on port 8443, which is using the same cert as on port 443).
After I updated the certificate I was able to register the UnifyVSA VASA provider. Note that this was the latest version of UnityVSA as of today (4.2.0.9392909) on vCenter 6.5U1 and with ESXi 6.5U1.