I figured this out. Much searching locally and across the internet turned up little info but with what I found and some experimentation I found this to work for migrating a Solaris 10 Update 3 IDE-based VM to ESX, so am posting it so others don't have to go through the same pain I did (maybe a KB article would be appropriate?).
1) Run VMware Converter
2) Boot the Failsafe Kernel under ESX
3) Allow it to mount the root disk (under /a, the default)
4) Run "mount" and note the /devices/... path used for the root disk.
In my case this was /devices/pci@0,0/pci1000,30@10/sd@0,0:a
I don't know if this will vary, it may always be this path after Converter runs.
I'll use XXX to represent your pathname after /devices/ below just in case
(e.g. XXX=/pci@0,0/pci1000,30@10/sd@0,0:a in my case)
5) TERM=sun-color; export TERM
This will make using vi easier.
6) vi /a/boot/solaris/bootenv.rc
update property "bootpath" (from above):
setprop bootpath XXX
in my case it was:
setprop bootpath /pci@0,0/pci1000,30@10/sd@0,0:a
7) cp /etc/path_to_inst /a/etc/path_to_inst
8) rm /a/etc/devices/*
9) update bootarchive:
bootadm update-archive -R /a
10) Edit /etc/vfstab
Find the line for your root disk. Replace the /dev/dsk/??? path with /devices/XXX, and replace the /dev/rdsk/??? path with /devices/XXX,raw (e.g. /devices/pci@0,0/pci1000,30@10/sd@0,0:a,raw)
This is necessary because before device reconfiguration occurs the kernel will try to see if the / filesystem needs checking, by reading /etc/vfstab, but until reconfiguration occurs there is no /dev entry for the / filesystem. So you need to use the absolute /devices path until reconfiguration can be performed and a /dev alias path is created.
11) Reboot
12) At the Grub screen where you choose the kernel you want, select the normal kernel, but press 'e' to edit it.
13) Move down to the 2nd (kernel) line, and type 'e' to edit it
Append "-r -s" to force device reconfiguration and boot single-user, then press Return
to return to the previous screen
14) Type 'b' to boot this temporarily-modified configuration
15) When you enter single-user, do an ls -l /dev/dsk to find the c?txdxsx disk that corresponds to the /devices path you are using. It's probably c2t0d0s0
16) Touch up /etc/vfstab to use the /dev/* paths which correspond to your disk (don't forget to update the swap entry as well, using the same txdxsx with the correct c? value for your disk).
17) Type ^D to go to multi-user
Done! Be careful typing those long /devices path names!
