14 Replies Latest reply on Dec 6, 2009 3:13 PM by sebek

    2TB LUN Limit

    sebek Enthusiast

       

      Hi,

       

       

      I am migrating my VI 3.5 into vSphere 4.0. I am trying to migrate one of my VMs with RDM configured. My RDM is 8TB large. Unfortunately, I cannot power on that VM on vSphere. The error msg I receive is "Disks bigger than 2TB - 512B are not supported. Unable to create virtual SCSI device for scsi1:xx". I am aware of 2TB limit, however, I can break the barrier on 3.5, so why would not I on ESX 4? Any ideas are more than welcome.

       

       

        • 1. Re: 2TB LUN Limit
          mcowger Champion

          It was a 'bug' that let you do it on 3.5, which has been 'corrected'

           

           

           






          --Matt

          VCP, vExpert, Unix Geek

          1 person found this helpful
          • 2. Re: 2TB LUN Limit
            AntonVZhbankov Guru
            User Moderators

            The only way you can use single LUN bigger than 2TB in vSphere:

            1) It has to be iSCSI storage

            2) You connect this iSCSI LUN with software iSCSI initiator from inside guest OS.


            ---

            MCSA, MCTS, VCP, VMware vExpert '2009

            http://blog.vadmin.ru

            1 person found this helpful
            • 3. Re: 2TB LUN Limit
              sebek Enthusiast

              Just like I supposed. Dissapointing Thanks, guys, anyway.

              • 4. Re: 2TB LUN Limit
                sebek Enthusiast

                 

                Update. According to Configuration Maximums, max rdm size is 2TB - 512B. In addition, vmdk size is limited to 2TB - 512B, too, beacause of VMFS file size limit, which is 2TB - 512B.  It comes, because esx formats volumes using max block size of 8MB. IMHO, it makes 64TB (- 16K Volume Size useless, because it is not possible to place large vmdk files (over 2TB). Of course there  might be some rumour about placing many vmdks on the same large datastore (built with extents) and using dynamic disks to combine vmdks into large volume. Imho it's not good idea, but it's possible. But what if I don't want to use dynamic disks? I think it is possible to use software iSCSI initiator inside the vm. Quite poor alternative from performance perspective. I was very surprised of vsphere behaviour, because esx 3.5 handles large rdms pretty fine. Does 2TB LUN Size Limit make 64TB Volume Size useless? It is a big step backward, imho. I have just figured out the answer.. Placing vswp files on large datastores can be a good solution in large environments. Any different propositions?

                 

                 

                thanks,

                 

                 

                 

                 

                 

                • 5. Re: 2TB LUN Limit
                  AntonVZhbankov Guru
                  User Moderators

                  I was very surprised of vsphere behaviour, because esx 3.5 handles large rdms pretty fine.

                   

                  It was clearly stated in 3.5 docs that maximum LUN size is 2 TB.

                   

                  Configuration Maximums

                  VMwareInfrastructure 3: Update 2 and later for ESX Server 3.5, ESX Server 3i version 3.5, VirtualCenter 2.5

                   

                  Table2.  Storage Maximums

                  Item

                  Maximum

                  Raw Device Mapping size (TB)

                  2


                  ---

                  MCSA, MCTS, VCP, VMware vExpert '2009

                  http://blog.vadmin.ru

                  • 6. Re: 2TB LUN Limit
                    merovingianA51 Novice

                    Wow, you guys just ruined my day.  Right now on my ESX 3.5 farm I am running VMs with 24TB RAW device mappings.  I just went to add a 2TB RAW LUN Mapping to my ESX4 environment and it failed!  I suspect that the LUN is just over 2TB which may be causing the problem.  But now I'm left with a big problem - what to do with all my 24TB LUN Mappings.  I do not have ISCSI setup and have no idea how it works.  Our environments are all FC!!

                     






                    ----


                    "Crippling Microsoft is the geek equivalent of taking down the Death Star"

                    • 7. Re: 2TB LUN Limit
                      AntonVZhbankov Guru
                      User Moderators

                      Get back to 3.5 until you figure out what to do with storage.


                      ---

                      MCSA, MCTS, VCP, VMware vExpert '2009

                      http://blog.vadmin.ru

                      • 8. Re: 2TB LUN Limit
                        s1xth Expert
                        VMware Employees

                         

                        Anton..

                         

                         

                        Just curious...In the near future I am looking at moving one of my current SANs (dell pe 850 and connected to a SAS MD1000) to a iSCSI based setup, along this migration I want to make the host a VM, (no reason it shouldnt be). I am currently using about 1.8tb and expect that to grow to 3TB by the end of next year. What would you recommend for the best setup for the VM? I would be using a EQL PS4000 with sata disks.

                         

                         

                        Thanks!!

                         

                         

                        • 9. Re: 2TB LUN Limit
                          larstr Virtuoso
                          vExpert
                          s1xth wrote:

                          along this migration I want to make the host a VM, (no reason it shouldnt be). I am currently using about 1.8tb and expect that to grow to 3TB by the end of next year. What would you recommend for the best setup for the VM? I would be using a EQL PS4000 with sata disks.

                          In that case you will need to use the iSCSI initiator inside the guest to map the disk once it goes over the 2 TB limit.

                           

                          Lars

                          • 10. Re: 2TB LUN Limit
                            s1xth Expert
                            VMware Employees

                             

                            How would I set that up with the iSCSI initiator inside the guest? Do I need to create a new vSwitch for iSCSI for just this VM or can i use the same vSwitch that I currently have configured for my current iSCSI connections? What is the best practice in this type of setup? Use a dedicated pNic for the Vm iSCSI intiator?

                             

                             

                            Thanks!!

                             

                             

                            • 11. Re: 2TB LUN Limit
                              sebek Enthusiast

                              I assume you use iSCSI for connecting storage to ESX via VMkernel port. In that case you place your vmdks on iSCSI datastore, thus, all limitations still apply.

                               

                              To connect iSCSI to your VM, you need to obtain iSCSI initiator for guest OS, which are avaiable for free or are built in OS. I recommend to use dedicated pnic (or two), for redundancy and according to IO workload. I would connect each pnic to separate physical switch, too. and consider playing with frame size inside guest OS to achieve better performance.

                              • 12. Re: 2TB LUN Limit
                                larstr Virtuoso
                                vExpert

                                You can create a VM network on the same uplink as your vmkernel storage network.

                                 

                                To get the full picture of vswitches and best practises please have a read here:

                                http://kensvirtualreality.files.wordpress.com/2009/12/the-great-vswitch-debate-combined.pdf

                                 

                                Ken Cline knows his stuff

                                 

                                Lars

                                • 13. Re: 2TB LUN Limit
                                  DSTAVERT Guru
                                  User Moderators

                                  I would think this might be a good candidate for VMDirectPath.

                                  • 14. Re: 2TB LUN Limit
                                    sebek Enthusiast

                                    If VMotion is not necessary.