VMware Communities: Message List - Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal error

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Wed, 05 Apr 2006 00:43:35 GMT

... and to my knowledge they all use RHEL4 or CentOS4. In that case I have no idea, for 256MB VM even 512MB host should be sufficient, and 900MB host should not even notice that there is some VM running...

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

Ahnjoan — Tue, 04 Apr 2006 18:42:19 GMT

I really only use a single XP guest VM, it is set to use 256M of memory and the maximum that VMware "says" I should use is 722M. Hope that is what you were looking for.

FYI - There seem to be two other folks with the exact same problem, just no resolution, if you use google and search for "kswapd0 mode:0x850".

Thanks
Jared

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Tue, 04 Apr 2006 17:27:50 GMT

What settings do you use for VMs? How big guests you run on your box? You should probably allow some/most of memory to get swapped...

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

Ahnjoan — Tue, 04 Apr 2006 17:15:33 GMT

I believe we are having similar issues however I have tried the prescribed remedy and it isn't stopping my problem. I have changed vm.min_free_kbytes to "5120" in /etc/sysctl.conf. I have configured my machine to write to a remote syslog server. Here are the messages that get sent during the issue.

Apr 4 16:28:01 somehost.evening.com kernel: kswapd0: page allocation failure. order:0, mode:0x850
Apr 4 16:28:01 somehost.evening.com kernel:
Apr 4 16:28:01 somehost.evening.com kernel: Call Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff80171b47>{alloc_page_interleave+61}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8015c8d9>{__get_free_pages+11} <ffffffff8015f850>{kmem_getpages+36}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8015ffe5>{cache_alloc_refill+609} <ffffffff8015fcb3>{__kmalloc+123}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa004ba8c>{:jbd:__jbd_kmalloc+21} <ffffffffa00477b0>{:jbd:journal_get_undo_access+96}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa0058aa9>{:ext3:ext3_try_to_allocate_with_rsv+84}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa00591df>{:ext3:ext3_new_block+680} <ffffffffa005b3b6>{:ext3:ext3_alloc_block+7}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa00463c4>{:jbd:start_this_handle+964} <ffffffff8017a58f>{__block_write_full_page+198}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffffa005d40c>{:ext3:ext3_get_block+0} <ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff801638d7>{shrink_zone+3095} <ffffffff803037b4>{thread_return+42}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8013474a>{autoremove_wake_function+0} <ffffffff801641ef>{balance_pgdat+506}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff80164439>{kswapd+252} <ffffffff8013474a>{autoremove_wake_function+0}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff80131c95>{finish_task_switch+55} <ffffffff8013474a>{autoremove_wake_function+0}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff80131ce4>{schedule_tail+11} <ffffffff80110ce3>{child_rip+8}
Apr 4 16:28:01 somehost.evening.com kernel: <ffffffff8016433d>{kswapd+0} <ffffffff80110cdb>{child_rip+0}
Apr 4 16:28:01 somehost.evening.com kernel:
Apr 4 16:28:01 somehost.evening.com kernel: Mem-info:
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 DMA per-cpu:
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 hot: low 2, high 6, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 cold: low 0, high 2, batch 1
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 Normal per-cpu:
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 hot: low 32, high 96, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: cpu 1 cold: low 0, high 32, batch 16
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 HighMem per-cpu: empty
Apr 4 16:28:01 somehost.evening.com kernel:
Apr 4 16:28:01 somehost.evening.com kernel: Free pages: 11920kB (0kB HighMem)
Apr 4 16:28:01 somehost.evening.com kernel: Active:203733 inactive:21400 dirty:0 writeback:0 unstable:0 free:2980 slab:13408 mapped:198999 pagetables:3049
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 DMA free:11920kB min:80kB low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:1186 all_unreclaimable? yes
Apr 4 16:28:01 somehost.evening.com kernel: protections[]: 0 0 0
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 Normal free:0kB min:5036kB low:10072kB high:15108kB active:814932kB inactive:85600kB present:1030696kB pages_scanned:99 all_unreclaimable? no
Apr 4 16:28:01 somehost.evening.com kernel: protections[]: 0 0 0
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 4 16:28:01 somehost.evening.com kernel: protections[]: 0 0 0
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 DMA: 0*4kB 6*8kB 2*16kB 2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11920kB
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Apr 4 16:28:01 somehost.evening.com kernel: Node 0 HighMem: empty
Apr 4 16:28:01 somehost.evening.com kernel: Swap cache: add 163758, delete 137114, find 105445/114390, race 0+0
Apr 4 16:28:01 somehost.evening.com kernel: Free swap: 3854764kB
Apr 4 16:28:01 somehost.evening.com kernel: 261770 pages of RAM
Apr 4 16:28:01 somehost.evening.com kernel: 6670 reserved pages
Apr 4 16:28:01 somehost.evening.com kernel: 215081 pages shared
Apr 4 16:28:01 somehost.evening.com kernel: 26644 pages swap cached
Apr 4 16:28:01 somehost.evening.com kernel: ext3_try_to_allocate_with_rsv: aborting transaction: Out of memory in __ext3_journal_get_undo_access
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Out of memory
Apr 4 16:28:01 somehost.evening.com kernel: Aborting journal on device md2.
Apr 4 16:28:01 somehost.evening.com kernel: ext3_abort called.
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2): ext3_journal_start_sb: Detected aborted journal
Apr 4 16:28:01 somehost.evening.com kernel: Remounting filesystem read-only
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Out of memory
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted
Apr 4 16:28:01 somehost.evening.com kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

Dave Augustus — Wed, 01 Mar 2006 21:22:58 GMT

Awesome!!!

I was looking at hardware problems for weeks.

Thanks to everyone who worked to solve this problem.

BTW, I am running VMWS 5 on Centos 4.2.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Thu, 22 Sep 2005 04:25:48 GMT

It looks like upping 'min_free_kbytes' did fix the problem--I've been stable for 14+ days now, where before I'd crash in 1 hour to 2 days. Thank you Petr!

To recap, the fix is to pick one of the methods below, where <nnnn> is some number much greater than the default. 5120 worked for me, YMMV:

1) Recommended. Edit /etc/sysctl.conf and set "vm.min_free_kbytes = <nnnn>"
2) "echo <nnnn> > /proc/sys/vm/min_free_kbytes" in /etc/rc.local or some other place in the boot process.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

ceturc — Thu, 15 Sep 2005 03:15:51 GMT

JPV: I have experienced the exact same problem with a stock RHEL4 host doing lots of I/O (it caught me off guard, as my RHEL3 hosts had been rock stable for months and months).

I now understand this problem from this thread and have entered a significantly higher number:

vm.min_free_kbytes = 10240

Has this setting worked out for you over the past few days? From following the discussion here, it ~appears~ to have solved your problem - have you had any crashes in the meantime? Is this single sysctl.conf setting the answer to our woes?

Thanks so much,
Chuck

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Mon, 12 Sep 2005 22:05:05 GMT

Some more time passed, and as you did not come back
yet, I would suggest switching swappiness back to 60.

So far so good. Extensive use over the weekend installing and running various BSDs resulted in no problems even with 2 Windows VMs running (but doing nothing).

Swappiness is set back to 60, and I'll let it run for a while longer to make sure.

RE: all the memory info, thank you, that's great info, but perhaps overkill for me. While the host machine is the best I have for the purpose, it's still not great, so limiting the number of machines I run at once (read, the damage I can do to myself) is probably best.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Fri, 09 Sep 2005 14:33:51 GMT

Some more time passed, and as you did not come back yet, I would suggest switching swappiness back to 60.

If you'll switch host configuration (run vmware under root account, and go into global preferences) to allow most of memory to be swapped, ~80MB + size of videoram must be allocated in physical memory (unswappable). That is, ~100MB for each VM. For 4VM 400MB. Let's say that your X needs 100MB to be happy (you may want more, my X server says that 160MB is what it needs). So 500MB is left for guest's memory. And 'allow most of guest's memory to be swapped' says that 25% of memory must fit there - so you should be able to run your configuration with this setting - 2GB >= 384 + 384 + 256 + 256. But it will be probably rather slow, especially after resume.

You can also select 'allow some memory to be swapped' - then 50% of memory must fit in, and so you are limited by ~1GB - and your 2*384 + 2*256 does not fit there. 4*256 will probably fit in.

And if you'll select 'no swapping', you'll be limited by 500MB of memory - 4*128MB VM, or 2*384MB VM.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Thu, 08 Sep 2005 23:23:36 GMT

I'm not ready to call this fixed just yet, but it's looking good so far! (Hope I didn't just jinx myself.)

I'm running 2 Windows (384M RAM each), Linspire and cAos-2 (256M RAM each) on top of the CentOS-4.1 with 1G physical RAM. I just tried to start MEPIS, and was told I don't have enough physical memory. I've never seen that before, but it's a good sign as it a) didn't let me do something stupid and b) hasn't crashed yet.

"echo 90 > /proc/sys/vm/swappiness" did not solve the problem, but either "echo 5120 > /proc/sys/vm/min_free_kbytes" or the combination of both *may* have.

I'm off to go defrag a Windows hard drive, that should thrash it a bit.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Thu, 08 Sep 2005 17:22:57 GMT

Those messages are normal when vmware exits/is killed. It stops /dev/rtc timer, deallocates memory used by virtual machine, and closes vmnet0, so eth0 exits promisc mode.

Earlier these messages were probably lost in ext3 complaints - you should see most of them even on "normal" poweroff.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Thu, 08 Sep 2005 04:53:30 GMT

Crashed again. That was quick. The only difference is that in the middle of the usual lines in dmesg I found this:

/dev/vmmon[3839]: host clock rate change request 200 -> 0
vmmon: Had to deallocate locked 282 pages from vm driver e7edc000
vmmon: Had to deallocate AWE 2741 pages from vm driver e7edc000
device eth0 left promiscuous mode
bridge-eth0: disabled promiscuous mode

I was using Snort, so the promisc stuff is expected. No clue about the other stuff, which I've never seen before.

Rebooted and now using:

[ringo:root /dev/pts/1]
/root# cat /proc/sys/vm/swappiness
90

[ringo:root /dev/pts/1]
/root# cat /proc/sys/vm/min_free_kbytes
5120

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Wed, 07 Sep 2005 23:12:13 GMT

I'm using /etc/sysctl.conf as I do not have any /etc/rc.local... For some options it may be more important (or even vital) to get them set earlier during boot, when /etc/sysctl.conf is read, but for this option it should not matter - unless you run out of memory immediately when host boots, which apparently is not your problem (it could be problem with GSX setup to automatically start your virtual machine on host boot).

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Wed, 07 Sep 2005 22:53:18 GMT

OK, I've added the following to /etc/rc.local:
echo 5120 > /proc/sys/vm/min_free_kbytes

I have not made the change yet, as I want to only change one thing at a time between crashes so we can hopefully have a reasonably definitive answer to this thread.

BTW, it looks like you can do it via /etc/sysctl.conf (via vm.swappiness and vm.min_free_kbytes) or more brute force in /etc/rc.local as I have been. What is the preferred method?

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Wed, 07 Sep 2005 22:21:35 GMT

min_free_kbytes = 957 is definitely too low for 1GB system. You should have something around 2500 (2.5MB) in that file for 1GB system, more if you have slow disks and/or fast processor.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Wed, 07 Sep 2005 06:52:34 GMT

OK, it crashed again, running one W2K and two Linux (cAos-2 and Debian Sarge) VMs! I also have a Linspire, cAos-2 and W2K VM paused. All the same symptoms as above. But one point, I lied about assigned memory--the Windows boxes have 384M RAM each (though I don't remember doing that). I did confirm that each of the Linux VMs has 256M (recall 1G physical RAM and about 200M swap, not sure what I was thinking there). And this time, I am sure I was doing some disk IO (apt-get install samba rsync apache).

I was and am logged into the host via SSH, which was unaffected since it's only /data that is toast, / is fine. I got the following console message, then did the checks as requested (I think).

In the interests of changing only one thing at a time, I've added "echo 90 > /proc/sys/vm/swappiness" to /etc/rc.local. I frankly picked that one as it's easier than messing with tune2fs. Here we go again

Diagnostics
=========
Message from syslogd@ringo at Wed Sep 7 02:18:51 2005 ...
ringo kernel: journal_get_undo_access: No memory for committed data

[ringo:root /dev/pts/0]
/tmp# top
top - 02:23:55 up 18 days, 22:50, 3 users, load average: 0.13, 0.52, 0.87
Tasks: 74 total, 2 running, 72 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7% us, 9.0% sy, 0.0% ni, 87.3% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1034344k total, 1025488k used, 8856k free, 3428k buffers
Swap: 200804k total, 68316k used, 132488k free, 725288k cached
<snip>

[ringo:root /dev/pts/0]
/tmp# dmesg | head -5
start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted

[ringo:root /dev/pts/0]
/tmp# dmesg | tail -5
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted
EXT3-fs error (device hda4) in start_transaction: Journal has aborted

[ringo:root /dev/pts/0]
/tmp# dmesg > dmesg-2005-09-07

[ringo:root /dev/pts/0]
/tmp# wc -l dmesg-2005-09-07
225 dmesg-2005-09-07

[ringo:root /dev/pts/0]
/tmp# smartctl -a /dev/hda
smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
<snip>
Local Time is: Wed Sep 7 02:32:36 2005 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
<snip>
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
9 Power_On_Hours 0x0012 098 098 000 Old_age Always - 16971
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 76
194 Temperature_Celsius 0x0002 125 125 000 Old_age Always - 44 (Lifetime Min/Max 16/53)
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged
<snip>

[ringo:root /dev/pts/0]
/tmp# fdisk -l

Disk /dev/hda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 10 80293+ 83 Linux
/dev/hda2 11 201 1534207+ 83 Linux
/dev/hda3 202 226 200812+ 82 Linux swap
/dev/hda4 227 9726 76308750 83 Linux

[ringo:root /dev/pts/0]
/tmp# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
LABEL=/data /data ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
LABEL=SWAP-hda3 swap swap defaults 0 0
<snip>

[ringo:root /dev/pts/0]
tmp# cat /proc/sys/vm/swappiness
60

[ringo:root /dev/pts/0]
tmp# cat /proc/sys/vm/min_free_kbytes
957

[ringo:root /dev/pts/0]
tmp# cat /proc/sys/vm/lowmem_reserve_ratio
cat: /proc/sys/vm/lowmem_reserve_ratio: No such file or directory

[ringo:root /dev/pts/0]
tmp# echo 90 > /proc/sys/vm/swappiness

[ringo:root /dev/pts/0]
tmp# cat /proc/sys/vm/swappiness
90

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Sat, 03 Sep 2005 00:17:05 GMT

I see I should have mentioned I have 3 partitions, /boot, / and /data with all the VMW stuff in /data. /var is symlinked into /data/var (I chose sizes poorly). In all crashes, only /data was affected and it does have a 32M journal. System itself was fine, but VMW was very unhappy to be suddenly living in read-only land.

I will try your tune2fs suggestions after next crash. Thanks for all your help on this!

[ringo:root /dev/pts/0]
/root# echo 'stat <8>' | /sbin/debugfs -f - /dev/hda4
debugfs 1.35 (28-Feb-2004)
debugfs: stat <8>
Inode: 8 Type: regular Mode: 0600 Flags: 0x0 Generation: 0
User: 0 Group: 0 Size: 33554432
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 65616
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x42b1af7d -- Thu Jun 16 12:57:33 2005
atime: 0x00000000 -- Wed Dec 31 19:00:00 1969
mtime: 0x42b1af7d -- Thu Jun 16 12:57:33 2005
BLOCKS:
(0-11):1550-1561, (IND):1562, (12-1035):1563-2586, (DIND):2587, (IND):2588, (1036-2059):2589-3612, (IND)
:3613, (2060-3083):3614-4637, (IND):4638, (3084-4107):4639-5662, (IND):5663, (4108-5131):5664-6687, (IND
):6688, (5132-6155):6689-7712, (IND):7713, (6156-7179):7714-8737, (IND):8738, (7180-8192):8739-9751
TOTAL: 8202

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Fri, 02 Sep 2005 23:16:30 GMT

Memory is used also by disk caches, and it can happen that ext3 needs free memory when memory is full of dirty pages with VM & their vmdk files, so no reasonable amount can be freeed at this moment, without doing I/O. And I/O cannot be done as all this lives on one filesystem Increasing swappiness should force host to start writting dirty pages out sooner.

No application should be able to cause problem you see on correct kernel - kernel should start swapping long ago, and if you run out of both physical memory and swap, it should have killed offending application long ago, but not destroy/dismount/damage filesystem. But from your messages it seems like that you did not run out of virtual memory, just kernel tried to be smart about writting data to disk, and it was so smart that when it decided to write them out, it was too late...

You might want to determine what's your device with problem (say /dev/sda1), run 'tune2fs -l /dev/sda1' to find journal inode number (usually 8), and then run [code]echo 'stat <8>' | /sbin/debugfs -f - /dev/sda1[/code] to find journal size (default is 32MB).

If it will look too small, you may want to use tune2fs to remove journal from ext3 (it becomes ext2... 'tune2fs -O ^has_journal /dev/sda1') and add bigger one to the filesystem ('tune2fs -J size=<size you want> -j /dev/sda1'). Keep rescue CD near computer as tune2fs -O ^has_journal will probably require reboot, and it is possible that you have kernel without ext2 support, so maybe you'll have to run second tune2fs from installer's rescue session.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Fri, 02 Sep 2005 20:25:13 GMT

I've had a couple more crashes since I last posted, but have not had one in 11+ days, while running only 1 VM (W2K). I've only crashed when running 2 or more Windows VMs, so I'll fire up another and wait for a crash. Not sure about heavy disk IO though, VMs were mostly sitting there doing not much of anything during previous crashes.

Re: SMART, it was not running. I checked out the excellent article on this topic at http://www.linuxjournal.com/article/6983 and fired it up. Great tip!

Re: Memory, have 1G phys, running 2 VMs with 256M each, see top output above in http://www.vmware.com/community/thread.jspa?threadID=20690&messageID=248349#248349. Also running a bare minimum Xfce4 with virtually nothing else, expressly to save resources for VMs instead of host.

Was not familiar with /proc/sys/vm/swappiness, it is at the default of 60. But given that I am being quite conservative with memory (IMOHO), should this really matter? Could there be some kind of memory leak when VMware has to handle >1 Windows VM, with associated auto mouse-grab?

FYI, according to VMW Help, Check for Updates, there aren't any. I'm running Workstation 5.0.0 build 13124 on (CentOS/RHEL) kernel 2.6.9-5.0.5.EL.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Mon, 22 Aug 2005 23:54:45 GMT

Yes, it probably is original error source. You've run out of physical memory. Add more memory, run smaller VMs, or configure system for more agressive swapping and releasing memory. First two are obvious, third one can be configured by /proc/sys/vm/swappiness, /proc/sys/vm/min_free_kbytes (and partially by /proc/sys/vm/lowmem_reserve_ratio). Bigger value you put to 'min_free_kbytes', the better. For start I would try double current value. 'swappiness' is by default 60, you may try 70 or maybe even bigger value (100 is max).

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

sacolcor — Mon, 22 Aug 2005 18:15:13 GMT

Will try that the next time this happens.

I suspect that this console message might be the original error, or close to it:
"<hostname> kernel: journal_get_undo_access: No memory for committed data".

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Mon, 22 Aug 2005 17:53:43 GMT

Real error is before this message... You need 'dmesg -s131072' or something like that to find real cause. First message is truncated at the beginning because 'EXT3-fs error (device hda4) in' did not fit to the dmesg buffer size you are using.

Search on google for 'in start_transaction: journal has aborted' returns over 600 hits... I have no idea which particular instance you are hitting until you can provide all error messages. If you cannot find it with 'dmesg -s131072', you may have to use serial console or rebuild your kernel with bigger dmesg buffer or run netconsole to log messages over network to some other system.

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

sacolcor — Mon, 22 Aug 2005 17:44:53 GMT

The line I was referring to was:

Dmesg begins " start_transaction: Journal has aborted" then has 224: "EXT3-fs error (device hda4) in start_transaction: Journal has aborted"

You indicate that this is a known bug for the Linux kernel...can you provide a link to a bug entry or mailing list record where we can get more information?

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Mon, 22 Aug 2005 12:42:43 GMT

It is not correct behavior, but it is bug of your *host*, VMware only triggers it because it needs huge I/O bandwidth for guest OS.

BTW, I have no idea what dmesg JPV noted you talk about. I see no 'dmesg' output anywhere. /var/log/messages of course won't show anything - if you remount disk read-only, there is no way how to update /var/log/messages. You must do 'dmesg' when disk is remounted read-only, and write text by hand down to the paper...

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

sacolcor — Fri, 19 Aug 2005 18:39:19 GMT

I have also had this crash multiple times, running CentOS-4.1 hosting WinXPSP2. It seems to be most common during heavy disk I/O, particularly when installing large service packs. The dmesg is the same as the one JPV noted.

This would seem to be a serious problem; it's causing filesystem corruption on the host (and probably the guest, too). And if it's happening on CentOS4, RHEL4 is probably similarly affected.

petr: You indicated that the host "remounts harddisk read-only when it sees someone really wants to read & write files a lot..." Could you elaborate a bit? I don't see how that could be considered correct behavior.

Thanks!

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

mattlav — Fri, 19 Aug 2005 10:33:39 GMT

Another thing to check would be the SMART status of the drive. The only time that I have had the system remount RO has been when the disk is about to fail. If that drive has about had it then you will see the failures in the SMART output and can grab the data fast.

Matthew

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

petr — Thu, 18 Aug 2005 18:02:53 GMT

Post 'dmesg' from host when this happens. It has nothing to do with guest, VMware just wants stable I/O subsystem, and your host remounts harddisk read-only when it sees someone really wants to read & write files a lot...

Re: Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal er

JPV — Tue, 16 Aug 2005 22:09:45 GMT

The same thing just happened again. The only difference this time is that I thought to grab output from top as well:

top - 18:09:39 up 3 days, 21:56, 3 users, load average: 0.25, 0.37, 0.36
Tasks: 72 total, 2 running, 70 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.7% us, 11.7% sy, 0.0% ni, 85.7% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1034344k total, 1025912k used, 8432k free, 4752k buffers
Swap: 200804k total, 66304k used, 134500k free, 723048k cached

Any clues anyone!?!

Disk turns read-only: Workstation 5 crash on RHEL4.1 w/ EXT3 journal error

JPV — Sat, 13 Aug 2005 00:21:52 GMT

I have VMware Workstation 5 running on a host OS of CentOS-4.1 (RHEL 4.1 rebuild, see www.centos.org). Host OS is a *minimal* install, using Xfce4 and as little else as I could manage. SELinux is disabled.

For the second time now, I've crashed as follows. The first time was a few weeks ago and I ignored it and rebooted, so no info there. I *think* I was running a Windows WM that time too though. (Damn Windows, can even get it stable when running it on Linux! )

This time I was running 2 Win2K server VMs (linked clones) with 3 other (various Linux) VMs paused. The box is a plain Dell Optiplex GX-260 P4-500 with 1G RAM and 80G IDE. It is company owned and I haven't messed with it (i.e. no overclocking). The machine is on an old Compaq PS/2 KVM, so no USB BS. The Windows VMs are fully patched (ironically I was testing a patch management tool in them).
VMware tools are installed on the 2 Windows VMs and they are using auto mouse-grab. I am not running full-screen.

A soft reboot of the host machine fixed the problem, for now. Naturally, on reboot an fsck was forced on that partition, and it completed with a couple of minor inod complaints.

I can't try another host as I don't have anything laying around with the hosepower needed. But up until recently that box was a Win2KPro workstation that worked fine except that it had Windows on it.

Possibly related to the following, though I have NOT been able to tie it to any mouse movement (haven't tried very hard either):
http://www.vmware.com/community/thread.jspa?threadID=17535
http://www.vmware.com/community/thread.jspa?threadID=7598
http://www.linuxquestions.org/questions/history/342887
http://www.webservertalk.com/archive235-2004-6-285876.html
http://ist.uwaterloo.ca/~kscully/crow.html

Symptom:
Dialog box: 'Operation on file "foo.vmdk" failed (Read-only files system.) Choose retry to attempt again [Abort] [Continue] [Retry]'

Dmesg begins " start_transaction: Journal has aborted" then has 224: "EXT3-fs error (device hda4) in start_transaction: Journal has aborted"

Console had: "hostname kernel: journal_get_undo_access: No memory for committed data"

/var/log/messages has nothing relevant.

Running everything under non-root user. VMware was only running for 2 days or so, nothing in the vmware.log that looks relevant, head/tail as follows:

/home/jp/vmware/Windows_2000_Server# head vmware.log
Aug 11 00:41:22: vmx| Log for VMware Workstation pid=31004 version=5.0.0 build=build-13124 option=Release
Aug 11 00:41:22: vmx| Command line: "/usr/lib/vmware/bin/vmware-vmx" "-@" "pipe=/tmp/vmware-jp/vmx0e8fab5e213e2683;vm0e8fab5e213e2683" "/home/jp/vmware/Windows_2000_Server/Windows_2000_Server.vmx"
Aug 11 00:41:22: vmx| UI Connecting to pipe '/tmp/vmware-jp/vmx0e8fab5e213e2683' with user '(null)'
Aug 11 00:41:22: vmx| pcpu #0 CPUID numEntries=2 GenuntelineI
Aug 11 00:41:22: vmx| pcpu #0 CPUID version=0xf27 id1.edx=0xbfebfbff id1.ecx=0x400 id1.ebx=0x20809
Aug 11 00:41:22: vmx| pcpu #0 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0
Aug 11 00:41:22: vmx| CPUID id1.edx: 0xbfebfbff id1.ecx: 0x400 id81.edx: 0 id81.ecx: 0
Aug 11 00:41:22: vmx| changing directory to /home/jp/vmware/Windows_2000_Server/.
Aug 11 00:41:22: vmx| Config file: /home/jp/vmware/Windows_2000_Server/Windows_2000_Server.vmx
Aug 11 00:41:22: vmx| VMXVmdbCbVmVmxExecState: Exec state change requested to state poweredOn without reset

/home/jp/vmware/Windows_2000_Server# tail vmware.log
Aug 11 01:54:23: vmx| SCSI0:0: Command WRITE(10) took 6.118 seconds (ok)
Aug 11 01:54:31: vmx| SCSI0:0: Command WRITE(10) took 2.540 seconds (ok)
Aug 11 01:54:31: vmx| SCSI0:0: Command WRITE(10) took 2.606 seconds (ok)
Aug 11 01:56:47: vmx| SCSI0:0: Command WRITE(10) took 1.219 seconds (ok)
Aug 11 01:56:47: vmx| SCSI0:0: Command WRITE(10) took 1.379 seconds (ok)
Aug 11 01:56:47: vmx| SCSI0:0: Command WRITE(10) took 1.501 seconds (ok)
Aug 11 06:59:12: vmx| DISKLIB-LIB :numIOs = 50000 numMergedIOs = 2560 numSplitIOs = 1565
Aug 11 23:18:39: vmx| SCSI0:0: Command WRITE(10) took 1.221 seconds (ok)
Aug 12 00:41:21: vmx| LICENSE using: '/home/jp/.vmware/license.ws.5.0'
Aug 12 01:16:22: vmx| DISKLIB-LIB :numIOs = 100000 numMergedIOs = 4150 numSplitIOs = 3215