VMware Cloud Community
devzero
Expert
Expert

Reason why you often can`t have RDM with local SCSI/Raid

Hello,

if you ever wondered why adding some RDM was greyed out and you could not map a local SCSI Volume, i think i found out what`s the reason why:

It`s because ESX doesn`t see a Serial Number for that Volume.

I read, that this was a limitation of some SCSI/Raid Controllers, but it seems it isn`t.

It must be a matter of the ESX driver, as i can read the Serial Number when booting into antoher Linux (grml 1.1)

Look here:

On ESX 3.5 (need sg3_utils-1.06-2.el3.1.i386.rpm from RHEL/CentOS to have sginfo)

sginfo -a /dev/sdb

INQUIRY reponse (cmd: 0x12)

-


Device Type 0

Vendor: ServeRA

Product: Vol2

Revision level: V1.0

doesn't understand request for serial number

modePageOffset: raw_curr too small, offset=8 resp_len=8 bd_len=0

mode page=0x3f has bad page format

perhaps '-z' switch may help

on grml (Linux Live-CD)

sginfo -a /dev/sdb

INQUIRY response (cmd: 0x12)

-


Device Type 0

Vendor: ServeRA

Product: Vol2

Revision level: V1.0

Serial Number 'D7F92229'

Caching mode page (0x8)

-


Initiator Control 0

ABPF 0

CAP 0

DISC 0

SIZE 0

Write Cache Enabled 1

MF 0

Read Cache Disabled 0

Demand Read Retention Priority 3

Demand Write Retention Priority 9

Disable Pre-fetch Transfer Length 82

Minimum Pre-fetch 16672

Maximum Pre-fetch 22127

Maximum Pre-fetch Ceiling 27698

FSW 0

LBCSS 0

DRA 1

NV_DIS 0

Number of Cache Segments 32

Cache Segment size 8224

Non-Cache Segment size 2105376

So this is either a bug or a missing feature of the driver.

0 Kudos
10 Replies
RParker
Immortal
Immortal

Sounds plausible, but I think you are overlooking the obvious. RDM is a SAN terminology not a local disk storage. It's meant to be shared, and it's therefore a restriction but may also be a limitation.

So VM Ware (in my way of thinking) would prevent creating RDM's on local storage, because it's not something they can probably do since local storage devices aren't appearing as LUN's. They can probably get around it, but why? More often then not things like this just aren't done locally because local storage is designed to boot the system's not host large storage volumes.

0 Kudos
devzero
Expert
Expert

why is RDM a SAN terminology?

ist`s a RAW DEVICE MAPPING.

what does that matter if a raw device is SAN or DAS and what does that matter if it`s shared or not ?

i can think of several scenarious where this would be useful.

think of a TB sized disk you have data on and you want to attach it to a virtual environment.

so, you plug it into your sata/sas slot, create a rdm and attach it to a VM - and you have that data online.

without RDM you would need to have another TB of storage and lot`s of time to move it....

piece of cake with vmware hosted products....i just did it again with that.

More often then not things like this just aren't done locally because local storage is designed to boot the system's not host large storage volumes.

why isn`t local storage suitable for hosting large storage volumes?

I can have Terabytes with that. for much less than a 1000 bucks.

you mean, because EMC is a SAN company, they want me to have a SAN for large storage?

come on !

0 Kudos
RParker
Immortal
Immortal

why isn`t local storage suitable for hosting large storage volumes?

Uh... because it's local, and RAID controllers in a physical box aren't designed for MULTI user? They are smaller machines, lets not forget, and they are NOT really designed for Virtual Machines, they are getting better, but SAN devices are ideal for LARGE, Multi-Access data points. That's what they were made for. There is a reason that a physical box is 1% of the price of a typical SAN solution, that's not an accident, otherwise everyone would simply put their data on physical boxes and skip SAN altogether. Sure you can do it, but that storage you are talking about SATA.. I can tell you you have never even tried to host many VM's on a SATA RAID, go ahead give it a shot... When you start the 4th VM and the rest of the VM's come to a COMPLETE crawl, don't say I didn't warn you. Yeah, it's cheap... but not good performance, and it's worthless for anything but long term backup storage.

can have Terabytes with that. for much less than a 1000 bucks.

go for it. Don't complain when that $1000.00 4TB local storage isn't good for much more than MP3's and Photos.. good luck with that.

you mean, because EMC is a SAN company, they want me to have a SAN for large storage?

NO, that's that way it's ALWAYS been. Read up on Raw Device Mapping. It's for LUN's.it was intended for LARGE SAN's (that's who designed it, those big EMC / Netapp Companies), and since RDM's usually are 300-500G or bigger, when they came up with the idea it was at a time when local storage was 9GB SCSI drives... Think back 10 years, how far have we come? consider technology we use, and HOW it was implemented.. nothing has changed -- EXCEPT -- things are cheaper now, there haven't been any real big advances in hardware technology, not really.. things got faster, cheaper, and more of them (cores aren't really that fast, there are just more of them on a single die which comprises the speed). So SAN technology is like mainframes it existed LONG before anyone thought of Virtualizing.

Now when SSDD drives come down, NOW you will see changes to local machines, because with Dunnington (6 core) or 12 core by then, these "small" machines will become self sustaining portable SAN solutions...

come on !

Calm down, it will be ok. I didn't say I liked it, I am saying that's why. RDM is a LUN element, when or where have you EVER seen RDM in a RAID configuration, hmmmm? Yeah Show me a local SCSI RAID that has this option, you won't find it.. because as I said at the outset it's a SAN Terminology.

0 Kudos
devzero
Expert
Expert

thanks for sharing your thoughts.

but i think we think very different here.

you`re speaking of storage performance and of san vs. local storage.

i`m speaking of attaching a local scsi disk to a vm instead of formatting it with vmfs first.

it`s not about performance, it`s about access to data. it`s about skipping a filesystem layer. something which can be done with hosted products since the early beginnings and something i still don`t understand why it`s an issue with esx.

if it``s all up to a scsi id, then maybe it shouldn`t be too hard to fix.

as we can have local storage with esx, and if that is supported, then some additional flexibility could not hurt.

it`s an option.

you don`t have to use it that way.

0 Kudos
kjb007
Immortal
Immortal

You can add local storage, but not as an RDM. You have to add in the device as a generic SCSI device. See here for some help: http://communities.vmware.com/thread/155874

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
0 Kudos
Texiwill
Leadership
Leadership

Hello,

There is 'Raw Disk Map' or RDM and Raw Device Map or RAW modes. They are a bit different. In general you can not use RDM with Local RAID controllers but you may be able to use RAW with local RAID types using the SCSI Generic Mode. Actually I have found that ESX v3.5 U3 has issues with the RAW mode/SCSI Generic Modes these days.

I used to have it working quite well on v3.0.x but it is on the list to test with v3.5.x once more. Good luck!


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll

Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
BUGCHK
Commander
Commander

There are several SCSI pages which can contain a 'serial number'. It's my understanding that ESX uses 83(16).

0 Kudos
dljones
Contributor
Contributor

So you are saying if I have a 10TB MSA60 tray on my HP DL380 Server I have to copy it over and then copy all the data back to my vmdisk? What a load!! This means my upgrade could take days if not weeks to complete that is if we have a temp location to copy all this data over usin.wndows to copy all this over a network then back. Not a very good way to handle large storage with DAS. What EMC is doing is forceing all vendors that support DAS to buy a MAS2000 contoller for the HP Server at a cost our customer will not be happy with. So their is no win - win for anyone!!!

dj

0 Kudos
dljones
Contributor
Contributor

Update

Once we upgraded to ESXi 5.0 or higher adding RDM to an internal Raid with a DAS (direct attached tray) is easy now.

I don’t have any issues creating a large lun in VMware. example a HP D2600 tray I'm running 18.6TB in size and can be used in Windows 2008 Server as one large lun without dynamic disk (thanks to windows 2008 Server getting past the limit of expanding a basic lun).

dj

0 Kudos
doubleH
Expert
Expert

Thank you for your message. I am currently out of the office and will return Apr 9, 2013. I will respond upon my return.

For support related issues please contact the IT Service Desk servicedesk@camhydro.com or x2700.

Thank you

Heath

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
0 Kudos