VMware
1 2 Previous Next 28 Replies Last post: Apr 4, 2006 5:43 PM by petr   Go to original post

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

15. Sep 7, 2005 3:53 PM in response to: petr
Click to view JPV's profile Novice
JPV
24 posts since
Jun 16, 2005
OK, I've added the following to /etc/rc.local:
echo 5120 > /proc/sys/vm/min_free_kbytes

I have not made the change yet, as I want to only change one thing at a time between crashes so we can hopefully have a reasonably definitive answer to this thread.

BTW, it looks like you can do it via /etc/sysctl.conf (via vm.swappiness and vm.min_free_kbytes) or more brute force in /etc/rc.local as I have been. What is the preferred method?

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

16. Sep 7, 2005 4:12 PM in response to: JPV
Click to view petr's profile Champion
petr
7,218 posts since
Jul 10, 2003
I'm using /etc/sysctl.conf as I do not have any /etc/rc.local... For some options it may be more important (or even vital) to get them set earlier during boot, when /etc/sysctl.conf is read, but for this option it should not matter - unless you run out of memory immediately when host boots, which apparently is not your problem (it could be problem with GSX setup to automatically start your virtual machine on host boot).

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

17. Sep 7, 2005 9:53 PM in response to: JPV
Click to view JPV's profile Novice
JPV
24 posts since
Jun 16, 2005
Crashed again. That was quick. The only difference is that in the middle of the usual lines in dmesg I found this:

/dev/vmmon[3839]: host clock rate change request 200 -> 0
vmmon: Had to deallocate locked 282 pages from vm driver e7edc000
vmmon: Had to deallocate AWE 2741 pages from vm driver e7edc000
device eth0 left promiscuous mode
bridge-eth0: disabled promiscuous mode

I was using Snort, so the promisc stuff is expected. No clue about the other stuff, which I've never seen before.

Rebooted and now using:

[ringo:root /dev/pts/1]
/root# cat /proc/sys/vm/swappiness
90

[ringo:root /dev/pts/1]
/root# cat /proc/sys/vm/min_free_kbytes
5120

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

18. Sep 8, 2005 10:22 AM in response to: JPV
Click to view petr's profile Champion
petr
7,218 posts since
Jul 10, 2003
Those messages are normal when vmware exits/is killed. It stops /dev/rtc timer, deallocates memory used by virtual machine, and closes vmnet0, so eth0 exits promisc mode.

Earlier these messages were probably lost in ext3 complaints - you should see most of them even on "normal" poweroff.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

19. Sep 8, 2005 4:23 PM in response to: JPV
Click to view JPV's profile Novice
JPV
24 posts since
Jun 16, 2005
I'm not ready to call this fixed just yet, but it's looking good so far! (Hope I didn't just jinx myself.)

I'm running 2 Windows (384M RAM each), Linspire and cAos-2 (256M RAM each) on top of the CentOS-4.1 with 1G physical RAM. I just tried to start MEPIS, and was told I don't have enough physical memory. I've never seen that before, but it's a good sign as it a) didn't let me do something stupid and b) hasn't crashed yet.

"echo 90 > /proc/sys/vm/swappiness" did not solve the problem, but either "echo 5120 > /proc/sys/vm/min_free_kbytes" or the combination of both *may* have.

I'm off to go defrag a Windows hard drive, that should thrash it a bit. :-)

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

20. Sep 9, 2005 7:33 AM in response to: JPV
Click to view petr's profile Champion
petr
7,218 posts since
Jul 10, 2003
Some more time passed, and as you did not come back yet, I would suggest switching swappiness back to 60.

If you'll switch host configuration (run vmware under root account, and go into global preferences) to allow most of memory to be swapped, ~80MB + size of videoram must be allocated in physical memory (unswappable). That is, ~100MB for each VM. For 4VM 400MB. Let's say that your X needs 100MB to be happy (you may want more, my X server says that 160MB is what it needs). So 500MB is left for guest's memory. And 'allow most of guest's memory to be swapped' says that 25% of memory must fit there - so you should be able to run your configuration with this setting - 2GB >= 384 + 384 + 256 + 256. But it will be probably rather slow, especially after resume.

You can also select 'allow some memory to be swapped' - then 50% of memory must fit in, and so you are limited by ~1GB - and your 2*384 + 2*256 does not fit there. 4*256 will probably fit in.

And if you'll select 'no swapping', you'll be limited by 500MB of memory - 4*128MB VM, or 2*384MB VM.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

21. Sep 12, 2005 3:05 PM in response to: petr
Click to view JPV's profile Novice
JPV
24 posts since
Jun 16, 2005
Some more time passed, and as you did not come back
yet, I would suggest switching swappiness back to 60.

So far so good. Extensive use over the weekend installing and running various BSDs resulted in no problems even with 2 Windows VMs running (but doing nothing).

Swappiness is set back to 60, and I'll let it run for a while longer to make sure.

RE: all the memory info, thank you, that's great info, but perhaps overkill for me. While the host machine is the best I have for the purpose, it's still not great, so limiting the number of machines I run at once (read, the damage I can do to myself) is probably best. :-)

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

22. Sep 14, 2005 8:15 PM in response to: JPV
Click to view ceturc's profile Lurker
ceturc
2 posts since
Sep 14, 2005
JPV: I have experienced the exact same problem with a stock RHEL4 host doing lots of I/O (it caught me off guard, as my RHEL3 hosts had been rock stable for months and months).

I now understand this problem from this thread and have entered a significantly higher number:

vm.min_free_kbytes = 10240

Has this setting worked out for you over the past few days? From following the discussion here, it ~appears~ to have solved your problem - have you had any crashes in the meantime? Is this single sysctl.conf setting the answer to our woes?

Thanks so much,
Chuck

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

23. Sep 21, 2005 9:25 PM in response to: ceturc
Click to view JPV's profile Novice
JPV
24 posts since
Jun 16, 2005
It looks like upping 'min_free_kbytes' did fix the problem--I've been stable for 14+ days now, where before I'd crash in 1 hour to 2 days. Thank you Petr!

To recap, the fix is to pick one of the methods below, where <nnnn> is some number much greater than the default. 5120 worked for me, YMMV:

1) Recommended. Edit /etc/sysctl.conf and set "vm.min_free_kbytes = <nnnn>"
2) "echo <nnnn> > /proc/sys/vm/min_free_kbytes" in /etc/rc.local or some other place in the boot process.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

24. Mar 1, 2006 1:22 PM in response to: JPV
Click to view Dave Augustus's profile Novice
Dave Augustus
8 posts since
Jan 27, 2005
Awesome!!!

I was looking at hardware problems for weeks.

Thanks to everyone who worked to solve this problem.

BTW, I am running VMWS 5 on Centos 4.2.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

25. Apr 4, 2006 10:15 AM in response to: JPV
Click to view Ahnjoan's profile Lurker
Ahnjoan
2 posts since
Apr 4, 2006
I believe we are having similar issues however I have tried the prescribed remedy and it isn't stopping my problem. I have changed vm.min_free_kbytes to "5120" in /etc/sysctl.conf. I have configured my machine to write to a remote syslog server. Here are the messages that get sent during the issue.

Apr 4 16:28:01 somehost.evening.com kernel: kswapd0: page allocation failure. order:0, mode:0x850
Apr 4 16:28:01 somehost.evening.com kernel:
Apr 4 16:28:01 somehost.evening.com kernel: Call Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff80171b47>{alloc_page_interleave+61}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8015c8d9>{__get_free_pages+11} <ffffffff8015f850>{kmem_getpages+36}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8015ffe5>{cache_alloc_refill+609} <ffffffff8015fcb3>{__kmalloc+123}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa004ba8c>{:jbd:__jbd_kmalloc+21} <ffffffffa00477b0>{:jbd:journal_get_undo_access+96}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa0058aa9>{:ext3:ext3_try_to_allocate_with_rsv+84}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa00591df>{:ext3:ext3_new_block+680} <ffffffffa005b3b6>{:ext3:ext3_alloc_block+7}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa00463c4>{:jbd:start_this_handle+964} <ffffffff8017a58f>{__block_write_full_page+198}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa005d40c>{:ext3:ext3_get_block+0} <ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff801638d7>{shrink_zone+3095} <ffffffff803037b4>{thread_return+42}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8013474a>{autoremove_wake_function+0} <ffffffff801641ef>{balance_pgdat+506}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff80164439>{kswapd+252} <ffffffff8013474a>{autoremove_wake_function+0}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff80131c95>{finish_task_switch+55} <ffffffff8013474a>{autoremove_wake_function+0}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff80131ce4>{schedule_tail+11} <ffffffff80110ce3>{child_rip+8}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8016433d>{kswapd+0} <ffffffff80110cdb>{child_rip+0}
Apr 4 16:28:01 somehost.evening.com kernel:
Apr 4 16:28:01 somehost.evening.com kernel: Mem-info:
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 DMA per-cpu:
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 hot: low 2, high 6, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 cold: low 0, high 2, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 Normal per-cpu:
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 hot: low 32, high 96, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 cold: low 0, high 32, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 HighMem per-cpu: empty
Apr 4 16:28:01 somehost.evening.com kernel:
Apr 4 16:28:01 somehost.evening.com kernel: Free pages: 11920kB (0kB HighMem)
Apr 4 16:28:01 somehost.evening.com kernel: Active:203733 inactive:21400 dirty:0 writeback:0 unstable:0 free:2980 slab:13408 mapped:198999 pagetables:3049
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 DMA free:11920kB min:80kB low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:1186 all_unreclaimable? yes
Apr 4 16:28:01 somehost.evening.com kernel: protections[]: 0 0 0
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 Normal free:0kB min:5036kB low:10072kB high:15108kB active:814932kB inactive:85600kB present:1030696kB pages_scanned:99 all_unreclaimable? no
Apr 4 16:28:01 somehost.evening.com kernel: protections[]: 0 0 0
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 4 16:28:01 somehost.evening.com kernel: protections[]: 0 0 0
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 DMA: 0*4kB 6*8kB 2*16kB 2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11920kB
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 HighMem: empty
Apr 4 16:28:01 somehost.evening.com kernel: Swap cache: add 163758, delete 137114, find 105445/114390, race 0+0
Apr 4 16:28:01 somehost.evening.com kernel: Free swap: 3854764kB
Apr 4 16:28:01 somehost.evening.com kernel: 261770 pages of RAM
Apr 4 16:28:01 somehost.evening.com kernel: 6670 reserved pages
Apr 4 16:28:01 somehost.evening.com kernel: 215081 pages shared
Apr 4 16:28:01 somehost.evening.com kernel: 26644 pages swap cached
Apr 4 16:28:01 somehost.evening.com kernel: ext3_try_to_allocate_with_rsv: aborting transaction: Out of memory in __ext3_journal_get_undo_access
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Out of memory
Apr 4 16:28:01 somehost.evening.com kernel: Aborting journal on device md2.
Apr 4 16:28:01 somehost.evening.com kernel: ext3_abort called.
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2): ext3_journal_start_sb: Detected aborted journal
Apr 4 16:28:01 somehost.evening.com kernel: Remounting filesystem read-only
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Out of memory
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

26. Apr 4, 2006 10:27 AM in response to: Ahnjoan
Click to view petr's profile Champion
petr
7,218 posts since
Jul 10, 2003
What settings do you use for VMs? How big guests you run on your box? You should probably allow some/most of memory to get swapped...

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

27. Apr 4, 2006 11:42 AM in response to: petr
Click to view Ahnjoan's profile Lurker
Ahnjoan
2 posts since
Apr 4, 2006
I really only use a single XP guest VM, it is set to use 256M of memory and the maximum that VMware "says" I should use is 722M. Hope that is what you were looking for.

FYI - There seem to be two other folks with the exact same problem, just no resolution, if you use google and search for "kswapd0 mode:0x850".

Thanks
Jared

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

28. Apr 4, 2006 5:43 PM in response to: Ahnjoan
Click to view petr's profile Champion
petr
7,218 posts since
Jul 10, 2003
... and to my knowledge they all use RHEL4 or CentOS4. In that case I have no idea, for 256MB VM even 512MB host should be sufficient, and 900MB host should not even notice that there is some VM running...
1 2 Previous Next Go to original post

Actions

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities