VMware Horizon Community
fredr3k
Contributor
Contributor
Jump to solution

Nvidia Quadro <3 vSphere 5.1?

I assume that many of you with exictement have read through all news in vSphere 5.1

What cought my eye in View point of view was this:

"

Improved 3D Graphics Support – (View Only) –  hardware acceleration with possiblity to leverage NVIDIA’s hardware  cards installed in ESXi server, where those graphics cards are  virtualized and used in View Desktops. It’s targeted for graphics  intensive workloads, CAD designers, medical imaging etc…

NVIDIA Quadro 4000/5000/6000 and NVIDIA Tesla M2070Q are supported graphics cards.  Note that the ESXi Image profile must be installed with the NVIDIA GPU VIB file."

It certainly raises some questions for me:

a) The Nvidia GPU VIB-files is that free of charge or is it licensed from Nvidia or Vmware?

b) Have anyone tested 5.1 with Quadro-cards (and VMware Techie out there?) Pros? Cons?

c) Is the GPU fully accessible from the View guest or is there some kind of abstraction layer in between? After some research it seems like the abstraction layer is utilizing Xorg in a clever way.

202 Replies
LMUISHuntsville
Enthusiast
Enthusiast
Jump to solution

drxzeeh,

Good to hear it is working.  By the way, do you know what wasnt working?  How did you get it to start working?  I'm curious to hear what the issue was.

Regards,

Nathan

Reply
0 Kudos
drxzeeh
Contributor
Contributor
Jump to solution

Well, first of all that xorg service, i started it using the vsphere client but that didnt seem to cut it. after you gave me that init.d command it finally started.

after starting that the existing vm's didnt work.  i made a new template with a clean windows 7 installation. ( also for the teradici client software )

that did seem to cut it.   for the rest its pretty simple but just a few mistakes such as:

No dual gpu. ( got no gpu on chip )

the vib driver installation was fine, i used the update manager in vSphere

but the most important was the dual gpu, the xorg service and a clean template.

the cli commands Linjo gave me gave me more insight in what was working and what wasnt. really helped me in troubleshooting the problems.

Reply
0 Kudos
winjet1
Contributor
Contributor
Jump to solution

For me it seems that starting the XORG service did it.  I kept my VM (v9) that I had used and it cut the available pool by 256Mb.

Odd thing is I set the pool to use 512Mb.  I did see my Windows performance index go from 3 to 6 under 3D performance.  I will try 3D Mark next for a more accurate reading.

Anyone have any thoughts as to why only 256Mb is being reserved?  Also, anyone else get this to work on a card besides NVIDIA GRID i.e. maybe we can create a "it works" list of video cards.

(Full disclosure - I am running this on a modified GTX 680 into a GRID K2 with 1x GPU and 2Gb of vid RAM soooo.........it's a little MacGyver)

LMUISHuntsville
Enthusiast
Enthusiast
Jump to solution

That is interesting about your windows performance index being a "6" with an emulated K2.  I'm using a Quadro 4000 and have the same WPI.  I was thinking that if I move to a more capable card (more gpu cores) that score may go up some.

I was wondering if anyone else found that nvidia modding thread and was going to post results here.  I came across it two weeks ago, but havent gotten a chance to play with a modded 670 or 680 yet.

Technically speaking any Geforce/Quadro/Tesla card will work.  It is only restricted by nVidia's current driver to only load when it sees a "supported" device ID.  If you fool the driver into seeing what it thinks to be a supported device ID from its list it will work.  I was told that the very first driver nVidia created for vSGA worked with GeForce cards as well, but then once they released an update it restricted support for only pro level cards. 

Let us know what kind of 3D Mark 06 score you get from the first test.  I could only get the first two tests to run.  It would always through Direct3D device lost errors and quit before it could run any of the other tests.  The highest score I've gotten with the first test was 4200 frames in the alotted test time.  That was with the software view client.

Regards,

Nathan

Reply
0 Kudos
winjet1
Contributor
Contributor
Jump to solution

Well I got accidental damage protection for the 680 so I figured "why not".  I started with that modding thread and eventually arrived here once I couldn't figure out the VMware side.

[soap box] I really wish NVIDIA had supported other lower end cards with their drivers natively.  The GRID cards cost $2k+ and that's really hard to choke down when you're just tinkering around at home.  $500 I can swing.  The price of a used Honda, I can't swing. [end soap box]

They are modding new cards all the time on that forum so some cheaper cards should be available soon.  The 680 was just the easiest to mod (until the find the mod for the second GPU on the 690).

With the 4000 are you doing a shared GPU or pass-through?  NVIDIA's site lists a few other cards than the K2 included in that driver but I didn't know if those other cards supported virtualization.

Reply
0 Kudos
LMUISHuntsville
Enthusiast
Enthusiast
Jump to solution

I'm using the Quadro 4000 with vSGA; not doing passthrough at the moment.  Yes the other cards it lists are all supported with shared graphics (vSGA) Quadro 4000/5000/6000, Tesla M2070Q, and GRID K1/K2.

If you didint know already there is also a reg setting that opens up the frame rate lock on the devTap driver.  By default it is locked to 30 FPS.  To change it add this reg setting..

[HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware SVGA DevTap]
"MaxAppFrameRate"=dword:00000000

This is only useful if you are trying the get the maximum performance out of a certain application and not too concerned with sharing out the GPU resources efficiently to all your other VMs.  The VM that has this setting will use more GPU resources than the other VMs that don't have this setting.  This was put in place to set a cap on how much GPU resources a particular VM could use.

Reply
0 Kudos
winjet1
Contributor
Contributor
Jump to solution

In theory 5.2 is supposed to remove the need for the registry tweak.  Although I am guessing that means when a view session is established.

Here is some odd findings re 3DMark 06 v1.02.

When I run the benchmark through the console I can complete all of the tests and I get a 3DMark score of 11580 w/ the registry key set (it was like 7580 w/o the reg key).

When I try to run the benchmark through a view session, I get the same device lost error.  I'm guessing it is due to a crappy video drive by VMware.

Reply
0 Kudos
LMUISHuntsville
Enthusiast
Enthusiast
Jump to solution

The registry tweak that 5.2 takes care of is not the devtap frame rate lock.  The reg tweak that you are talking about is a network tweak, which modifies the FastSendDatagramThreshold value.  It can be found here - HKLM\System\CurrentControlSet\Services\Afd\Parameters
FastSendDatagramThreshold  (set it to 1500).

What is your 3DMark score for the first test?  I was able to get past the initial errors when using the view client by hitting "ok" on the error dialog box and trying to run the benchmark again.  The first test was the only one that would "eventually" run.  Yes, I'm not sure what is going on with the view client not being able to run through the entire set of tests.  Would be nice if it would.

Google Earth will also not show 3D at all when using DirectX.  You have to move over to OpenGL mode to even get 3D objects to show and even then the colors are off (they are on some type of grey scale only... no colors).

Reply
0 Kudos
Linjo
Leadership
Leadership
Jump to solution

The probable reason for the the device lost is that 3D Mark is changing resolution quite rapidly and PCoIP can't keep up with or might not support that resolution.

// Linjo

Best regards, Linjo Please follow me on twitter: @viewgeek If you find this information useful, please award points for "correct" or "helpful".
Reply
0 Kudos
siggebasse
Contributor
Contributor
Jump to solution

I got my Quadro 4000 installed and the Nivida driver installed. But I can´t  start the xorg service that everyone is talking about. It jus says "Xorg is not running"  I tried the "/etc/init.d/xorg start" command but same result. After reading this thread I put the console display at the onborad GPU.  Same result.

I have no vm running on the host.  Can anyone give me a clue what to do?

when i run the "nvidia-smi" command it returns:
Failed to initialize NVML: Unknown Error

Reply
0 Kudos
Linjo
Leadership
Leadership
Jump to solution

No problem, we will figure this out.

Could you post the output from these commands?

gpuvm    

Show what VMs are using gpu(s)

Esxcli software vib install –v /path-to-vib/name-of-vib.vib

Loads the NVIDIA VIB

esxcli software vib list | grep NVIDIA

Verify NVIDIA vib is installed

Esxcli system module load –module nvidia

Verify NVIDIA module load

Esxcli hardware pci list –c 0x300 –m 0xff

Verify devices are present

nvidia-smi

General status of the GPU / driver version

// Linjo

Best regards, Linjo Please follow me on twitter: @viewgeek If you find this information useful, please award points for "correct" or "helpful".
Reply
0 Kudos
siggebasse
Contributor
Contributor
Jump to solution

Thanks for quick respons.  Dont you never sleep? Smiley Happy

gpuvm   -> no result

Esxcli software vib install –v /path-to-vib/name-of-vib.vib -> 

Installation Result
   Message: Host is not changed.
   Reboot Required: false
   VIBs Installed:
   VIBs Removed:
   VIBs Skipped: NVIDIA_bootbank_NVIDIA-VMware_ESXi_5.1_Host_Driver_304.76-1OEM.510.0.0.802205


esxcli software vib list | grep NVIDIA ->

NVIDIA-VMware_ESXi_5.1_Host_Driver  304.76-1OEM.510.0.0.802205          NVIDIA  VMwareAccepted    2013-04-03

Esxcli system module load –module nvidia

# esxcli system module load -m nvidia (module, does not work for me!?) ->
Unable to load module /usr/lib/vmware/vmkmod/nvidia: Busy

Esxcli hardware pci list –c 0x300 –m 0xff ->


000:004:00.0
   Address: 000:004:00.0
   Segment: 0x0000
   Bus: 0x04
   Slot: 0x00
   Function: 0x00
   VMkernel Name:
   Vendor Name: Matrox Graphics, Inc.
   Device Name: G200eR2
   Configured Owner: Unknown
   Current Owner: VMkernel
   Vendor ID: 0x102b
   Device ID: 0x0534
   SubVendor ID: 0x1014
   SubDevice ID: 0x0405
   Device Class: 0x0300
   Device Class Name: VGA compatible controller
   Programming Interface: 0x00
   Revision ID: 0x00
   Interrupt Line: 0x0b
   IRQ: 11
   Interrupt Vector: 0x21
   PCI Pin: 0x00
   Spawned Bus: 0x00
   Flags: 0x0221
   Module ID: -1
   Module Name: None
   Chassis: 0
   Physical Slot: 0
   Slot Description: Matrox G200
   Passthru Capable: true
   Parent Device: PCI 0:3:0:0
   Dependent Device: PCI 0:3:0:0
   Reset Method: Bridge reset
   FPT Sharable: true

000:08b:00.0
   Address: 000:08b:00.0
   Segment: 0x0000
   Bus: 0x8b
   Slot: 0x00
   Function: 0x00
   VMkernel Name:
   Vendor Name: NVIDIA Corporation
   Device Name: NVIDIAQuadro 4000
   Configured Owner: Unknown
   Current Owner: VMkernel
   Vendor ID: 0x10de
   Device ID: 0x06dd
   SubVendor ID: 0x10de
   SubDevice ID: 0x0780
   Device Class: 0x0300
   Device Class Name: VGA compatible controller
   Programming Interface: 0x00
   Revision ID: 0xa3
   Interrupt Line: 0xff
   IRQ: 255
   Interrupt Vector: 0x61
   PCI Pin: 0x70
   Spawned Bus: 0x00
   Flags: 0x0201
   Module ID: -1
   Module Name: None
   Chassis: 0
   Physical Slot: 5
   Slot Description: PCI-Express Slot 5
   Passthru Capable: true
   Parent Device: PCI 0:128:2:2
   Dependent Device: PCI 0:128:2:2
   Reset Method: Bridge reset
   FPT Sharable: true

nvidia-smi ->


Failed to initialize NVML: Unknown Error

Reply
0 Kudos
LMUISHuntsville
Enthusiast
Enthusiast
Jump to solution

Is the Quadro 4000 configured for pass-through?  If it is, unconfigure it for pass-through.

Is the onboard video adapter configured as the primary output device in the BIOS?  If it is not, set it as primary output device.

Reply
0 Kudos
siggebasse
Contributor
Contributor
Jump to solution

Tjek!

Pass-through is not enabled and the active display is the onboard gpu device.

Reply
0 Kudos
siggebasse
Contributor
Contributor
Jump to solution

The host is running ESXi 5.1.0, 799733.  IBM 3650 M4. Vcenter: 5.1.0 7999731 could that some how be a issue?

I will make sure that I have the lastest firmware..brb!

Reply
0 Kudos
winjet1
Contributor
Contributor
Jump to solution

If you are using View 5.2 (which supports graphics virtualization) you should be on at least ESXi build 838463 (5.1.0a).

https://my.vmware.com/web/vmware/details?downloadGroup=VCL-VSP510-ESXI-510A-EN&productId=320&rPId=35...

Reply
0 Kudos
siggebasse
Contributor
Contributor
Jump to solution

It is now upgraded to ESXi build 838463.  I still cant start xorg with this command "/etc/init.d/xorg start"

gpuvm -> still no result 

Esxcli software vib install –v /path-to-vib/name-of-vib.vib ->

Installation Result
   Message: Host is not changed.
   Reboot Required: false
   VIBs Installed:
   VIBs Removed:
   VIBs Skipped: NVIDIA_bootbank_NVIDIA-VMware_ESXi_5.1_Host_Driver_304.76-1OEM.510.0.0.802205

esxcli software vib list | grep NVIDIA->

NVIDIA-VMware_ESXi_5.1_Host_Driver  304.76-1OEM.510.0.0.802205          NVIDIA  VMwareAccepted    2013-04-04

Esxcli system module load –module nvidia -> "esxcli system module load –m nvidia"  i use "-m" instead

Unable to load module /usr/lib/vmware/vmkmod/nvidia: Busy

Esxcli hardware pci list –c 0x300 –m 0xff ->

000:004:00.0
   Address: 000:004:00.0
   Segment: 0x0000
   Bus: 0x04
   Slot: 0x00
   Function: 0x00
   VMkernel Name:
   Vendor Name: Matrox Graphics, Inc.
   Device Name: G200eR2
   Configured Owner: Unknown
   Current Owner: VMkernel
   Vendor ID: 0x102b
   Device ID: 0x0534
   SubVendor ID: 0x1014
   SubDevice ID: 0x0405
   Device Class: 0x0300
   Device Class Name: VGA compatible controller
   Programming Interface: 0x00
   Revision ID: 0x00
   Interrupt Line: 0x0b
   IRQ: 11
   Interrupt Vector: 0x21
   PCI Pin: 0x78
   Spawned Bus: 0x00
   Flags: 0x0221
   Module ID: -1
   Module Name: None
   Chassis: 0
   Physical Slot: 0
   Slot Description: Matrox G200
   Passthru Capable: true
   Parent Device: PCI 0:3:0:0
   Dependent Device: PCI 0:3:0:0
   Reset Method: Bridge reset
   FPT Sharable: true

000:08b:00.0
   Address: 000:08b:00.0
   Segment: 0x0000
   Bus: 0x8b
   Slot: 0x00
   Function: 0x00
   VMkernel Name:
   Vendor Name: NVIDIA Corporation
   Device Name: NVIDIAQuadro 4000
   Configured Owner: Unknown
   Current Owner: VMkernel
   Vendor ID: 0x10de
   Device ID: 0x06dd
   SubVendor ID: 0x10de
   SubDevice ID: 0x0780
   Device Class: 0x0300
   Device Class Name: VGA compatible controller
   Programming Interface: 0x00
   Revision ID: 0xa3
   Interrupt Line: 0xff
   IRQ: 255
   Interrupt Vector: 0x61
   PCI Pin: 0x28
   Spawned Bus: 0x00
   Flags: 0x0201
   Module ID: -1
   Module Name: None
   Chassis: 0
   Physical Slot: 5
   Slot Description: PCI-Express Slot 5
   Passthru Capable: true
   Parent Device: PCI 0:128:2:2
   Dependent Device: PCI 0:128:2:2
   Reset Method: Bridge reset
   FPT Sharable: true

nvidia-smi->

Failed to initialize NVML: Unknown Error

conclusion = no differents. I really can´t figure it out. 

Reply
0 Kudos
winjet1
Contributor
Contributor
Jump to solution

Try to start it from vSphere.  It took about 5 minutes for mine to start up.

Also check to make sure you put the VIB in as an extension and not a patch.

Reply
0 Kudos
siggebasse
Contributor
Contributor
Jump to solution

What do you mean by "put the VIB in as an extension and not a patch"?

Reply
0 Kudos
winjet1
Contributor
Contributor
Jump to solution

If you used UpdateManager you can create a baseline to install the nvidia drivers (.zip file).  You want to make sure you select the option of extension and not patch when asked.

With XORG not starting that usually is a driver installation problem (maybe indicated by the "skipped" category?).

Reply
0 Kudos