VMware Cloud Community
LesykM
Contributor
Contributor
Jump to solution

ESXi inside KVM

Hello!

I wonder, if it is possible to run ESXi 5.0/5.1 inside KVM (proxmox) ?

I need to run few for testing purposes. I cannot setup any other virtualization software.

I need only ESXi itself, even without virtual machines inside it:)

Thanks!

Tags (3)
180 Replies
zamf
Contributor
Contributor
Jump to solution

Thanks a lot for your help.  Should I consider this an ESXi bug and report it?  We have been able to run many flavors of operating systems in our version of QEMU, so it is strange ESXi crashes. Perhaps this could be detected by an assert instead of generating a PSOD. Moreover, perhaps by generating a coredump and reporting the bug, I could get more insights into the root cause of the problem. What do you think?


Please let me know what is the best way to report the probelm. Should I use the FTP instructions here? http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...


Thanks!

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

I don't think there's enough evidence to ascertain whether the issue is in ESXi or in the emulator. A beta build of ESXi may have an assertion to cover this, but if there is an emulation bug, it's unlikely.  We typically don't add assertions to verify hardware behavior.

If you can't reproduce the problem on supported hardware, you probably won't get much traction with a bug report.  Send me the information on how to reproduce the problem, and I'll try to take a look when I get some time.

Reply
0 Kudos
brendanpg
Contributor
Contributor
Jump to solution

Hi,

I got everything but networking working with 5.5rollup1 so far on KVM (kernel 3.12.6, libvirt 1.2.0, QEMU 1.7.0, plus the Qemu piix patch from above), plus the following kvm flags:


modprobe.d]# cat kvm.conf

options kvm ignore_msrs=1

modprobe.d]# cat kvm-intel.conf

options kvm-intel nested=y ept=y

The last piece I am stuck on is the networking.

I tried Realtek, e1000 and vmxnet on the kvm side.

None of these work with ESXi 5.5, and I cant figure out why. I see some packets, ping will work in some cases - I see truncated packets and larger packets simply dont work. Any ideas on what I can do to get any basic networking - the test harness doesnt need super-high performance.

Reply
0 Kudos
mathiasew
Contributor
Contributor
Jump to solution

Did anyone manage to get it to work? I am desperate for a solution Smiley Sad

Reply
0 Kudos
mathiase
Enthusiast
Enthusiast
Jump to solution

nothing yet? who can I bribe to make to work? :smileygrin:

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

brendanpg wrote:

None of these work with ESXi 5.5, and I cant figure out why. I see some packets, ping will work in some cases - I see truncated packets and larger packets simply dont work. Any ideas on what I can do to get any basic networking - the test harness doesnt need super-high performance.

Can you provide more details on the missing packets?  Let's start with ping.  If you put a sniffer on the virtual network, what sort of traffic do you see?

Reply
0 Kudos
brendanpg
Contributor
Contributor
Jump to solution

They were truncated packets. I have to revive the setup for getting the actual packet data. What would happen would be a few successful pings (sometimes it would ping, sometimes it would not) and then the packets would be truncated and anything larger than a minimum sized packet would be truncated. .

With ping it would say on the ESXi side "20 bytes missing" or something of this nature when dumping vmk.

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

Did the truncated packets come from the ESXi guest or from the kvm virtual switch or both?

Reply
0 Kudos
mathiase
Enthusiast
Enthusiast
Jump to solution

Hi, I am just trying to catch up with the so far made progress in this matter. I downloaded QEMU source, edited hw/i386/pc_piix.c according to your earlier post, compiled and booted from ISO with

qemu-img create -f raw esxi00.img 10G

qemu-system-x86_64 -cpu SandyBridge -smp 2 -boot dc -m 2048 -hda esxi00.img -cdrom VMware-VMvisor-Installer-5.5.0-1331820.x86_64.iso

Now it is booting from the ISO and half way through the boot process making constant progress. As of writing this the screen shows "vmkibft loaded successfully". But to get there it took around 30min :smileygrin: any idea why it is that slow?

I would like to get to the point where the frames are truncated and then work from there.

cheers

Mathias

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

mathiase wrote:

Now it is booting from the ISO and half way through the boot process making constant progress. As of writing this the screen shows "vmkibft loaded successfully". But to get there it took around 30min :smileygrin: any idea why it is that slow?

That sounds like a good question for the KVM developers. Smiley Wink

Reply
0 Kudos
mathiase
Enthusiast
Enthusiast
Jump to solution

Yeah right :smileygrin: I was just wondering maybe you spit something out like "set flag XYZ starting qemu and you'll be good" Smiley Wink So I am going to wait for it and then never shut my machine down again :smileygrin: You will probably hear from me when the installer completed. So long!

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
mathiasew
Contributor
Contributor
Jump to solution

Hi, this is really strange: during installation I type 1234567 as the password and after installation the password doesnt work ... Can hardly be a layout issue, can it?

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

That is strange.  I have no explanation.

Reply
0 Kudos
brendanpg
Contributor
Contributor
Jump to solution

Hi jmattson ;

I ran into the same problem on a physical Realtek with an injected VIB for ESXi 5.5.U1. Basically link is up, ethtool works on vmnic0 and all offloads are disabled, tcpdump shows truncated IP, and DHCP addresses dont work and static addresses dont work.

I get an error 49 cant unset IP, and then I get a no-router warning from ipv6.

So whats interesting is the virtual e1000 from qemu, the realtek from qemu and the 8168 realtek in physical are having the same issues - truncated IP, no DHCP, etc.

Reply
0 Kudos
alexmoore83
Contributor
Contributor
Jump to solution

I've just attempted this too, and have had much the same experience... I am able to install and boot ESXi 5.5U1 inside KVM (qemu-1.7.0) after:

- Applying the patch to hw/i386/pc_piix.c from page 1 of this thread, and recompiling qemu

- Running "echo 1 > /sys/module/kvm/parameters/ignore_msrs" on the KVM host

...and have now hit the issue whereby networking doesn't seem to work (I'm presenting an e1000 nic to ESXi).

Although I did initially see "truncated-ip 246 bytes missing!" in the output from tcpdump-uw when run within ESXi, it looks like that is misleading and not the real problem. Running "tcpdump-uw -s0" so that it captures the entirety of the packets makes the "truncated-ip" messages go away.

What actually seems to be happening is that ESXi does not believe it is receiving any traffic from the network at all. It is successfully sending DHCP discovers (I have it configured to use DHCP), and they reach my DHCP server, which responds with DHCP offers. A tcpdump on the KVM host (on the virtual interface to which the ESXi guest is attached) shows that it is sending those DHCP offers towards the ESXi guest, but ESXi never sees them. Indeed the packet capture from within ESXi implies that it is not receiving any traffic whatsoever, including ARPs etc.

FWIW I'm also seeing the following show up regularly in /var/log/messages on the KVM host, at a rate of 1 per vCPU per second:

Apr 18 23:56:31 audi kernel: kvm [21198]: vcpu1 ignored rdmsr: 0x34

Apr 18 23:56:32 audi kernel: kvm [21198]: vcpu0 ignored rdmsr: 0x34

Reply
0 Kudos
alexmoore83
Contributor
Contributor
Jump to solution

Actually, running "ethtool -S vmnic0" inside ESXi tells me that it is receiving packets. Just tcpdump-uw (ie capturing on vmk0) does see any of them arrive.

Edit: Although I'm not quite sure whether I believe ethtool. The first 5 lines of output actually say:

NIC statistics:

    rx_packets: 755

    tx_packets: 121

    rx_bytes: 0

    tx_bytes: 0

And the figures aren't incrementing despite continued traffic.

Message was edited by: alexmoore83

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

alexmoore83 wrote:

FWIW I'm also seeing the following show up regularly in /var/log/messages on the KVM host, at a rate of 1 per vCPU per second:

Apr 18 23:56:31 audi kernel: kvm [21198]: vcpu1 ignored rdmsr: 0x34

Apr 18 23:56:32 audi kernel: kvm [21198]: vcpu0 ignored rdmsr: 0x34

That's a query of the SMI count, which is probably just used for reporting purposes.

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

alexmoore83 wrote:

Actually, running "ethtool -S vmnic0" inside ESXi tells me that it is receiving packets. Just tcpdump-uw (ie capturing on vmk0) does see any of them arrive.

Edit: Although I'm not quite sure whether I believe ethtool. The first 5 lines of output actually say:

NIC statistics:

    rx_packets: 755

    tx_packets: 121

    rx_bytes: 0

    tx_bytes: 0

And the figures aren't incrementing despite continued traffic.

Message was edited by: alexmoore83

What MAC address is reported by esxcfg-vmknic -l?  Is this the same as the MAC address that KVM has assigned to the ESXi VM?

Reply
0 Kudos
alexmoore83
Contributor
Contributor
Jump to solution

jmattson wrote:

What MAC address is reported by esxcfg-vmknic -l?  Is this the same as the MAC address that KVM has assigned to the ESXi VM?

Yes that reports the same MAC address as the one assigned by KVM (52:54:00:e1:fd:40 in both cases, which I believe was auto-generated by libvirt when I first defined the VM).

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

I don't know if this will be helpful, but can you post the output of

tcpdump -e ether src 52:54:00:e1:fd:40 or ether dst 52:54:00:e1:fd:40 (on the host)

while ESXi is negotiating for its DHCP address?

Reply
0 Kudos