File-system corruption caused by expanding file-size

[ashish_narmen1]

I experienced this problem today while using VMWare Workstation 6.0 on Suse 10.2. The exact details of the software are as follows

VMWare Workstation : 6.0.1 build-55017

Host: Suse Linux: 2.6.18.2-34-default #1 SMP Mon Nov 27 11:46:27 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux

I started off to create a virtual machine for the Solaris Express Developer Edition. The default option for the virtual hard disk was "Split disk into 2 GB files". However, I unchecked that option such the virtual hard disk would dynamically expand.

Guest: Solaris Express Developer Edition

During the course of installation (which failed), the virtual machine rebooted in a semi-installed state and the vm displayed the grub. Also, the entire GNOME desktop was painfully slow and simple actions were taking a lot of time. (The size of the vmdisk had expanded to about 4 GB by this time)

Upon rebooting, I was not able to log in to my user account. After a couple of failed attempts, I had to login using root. However, the default location of the virtual machine /home/username/vmware was accessible using read-only mode and I was unable to delete the Solaris Express virtual machine.

Another attempt to reboot and login in FailSafe mode did not mount that partition and repeated attempts to load the partition using the mount command were unsuccessful. After that I had to initiate a clean up of the file system using fsck which was able to detect and correct errors. Then I was able to mount the partition and delete the error causing file after which I was able to

I concur that the problem was caused by the file size that had grown beyond 4 gigs. But according to what I have read, ext3 supports greater file sizes too. Also, if this was the cause of the issue why did the kernel not shutdown the errant process (vmware) which was trying to write a file greater than the max allowed size. If not, does vmware run under the context of root due to which it was able to continue writing to the vmdk file causing the file-system to crash.

Looking forward to a response which would help me in drilling down to the root cause of this issue.

/etc/fstab listing

/dev/hda1 / ext3 acl,user_xattr 1 1

/dev/hda3 /home ext3 acl,user_xattr 1 2

/dev/hda5 swap swap defaults 0 0

proc /proc proc defaults 0 0

sysfs /sys sysfs noauto 0 0

debugfs /sys/kernel/debug debugfs noauto 0 0

usbfs /proc/bus/usb usbfs noauto 0 0

devpts /dev/pts devpts mode=0620,gid=5 0 0

/dev/fd0 /media/floppy auto noauto,user,sync 0 0

/dev/VolGroup00/LogVol00 /home/ashin/data ext3 defaults 1 2

Log file snippet

Sep 30 12:09:59 ashin-suse kernel: EXT3-fs error (device hda3): ext3_free_blocks: Freeing blocks in system zones - Block = 33992, count = 1

Sep 30 12:09:59 ashin-suse kernel: Aborting journal on device hda3.

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_free_blocks_sb: Journal has aborted

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3): ext3_free_blocks: Freeing blocks in system zones - Block = 1180160, count = 1

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_free_blocks_sb: Journal has aborted

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_free_blocks_sb: Journal has aborted

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3): ext3_free_blocks: Freeing blocks not in datazone - block = 368853200, count = 1

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3): ext3_free_blocks: Freeing blocks not in datazone - block = 4294934784, count = 1

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_free_blocks_sb: Journal has aborted

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_free_blocks_sb: Journal has aborted

Sep 30 12:10:00 ashin-suse kernel: ext3_abort called.

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3): ext3_journal_start_sb: Detected aborted journal

Sep 30 12:10:00 ashin-suse kernel: Remounting filesystem read-only

Sep 30 12:10:00 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_free_blocks_sb: Journal has aborted

.

.

.

Sep 30 12:10:05 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_reserve_inode_write: Journal has aborted

Sep 30 12:10:05 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_truncate: Journal has aborted

Sep 30 12:10:05 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_reserve_inode_write: Journal has aborted

Sep 30 12:10:05 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_orphan_del: Journal has aborted

Sep 30 12:10:05 ashin-suse kernel: EXT3-fs error (device hda3) in ext3_reserve_inode_write: Journal has aborted

Sep 30 12:10:05 ashin-suse kernel: /dev/vmmon[5125]: host clock rate change request 19 -> 0

Sep 30 12:10:05 ashin-suse kernel: vmmon: Had to deallocate locked 127763 pages from vm driver ffff810007e38000

Sep 30 12:10:05 ashin-suse kernel: vmmon: Had to deallocate AWE 4937 pages from vm driver ffff810007e38000

Sep 30 12:10:05 ashin-suse kernel: __journal_remove_journal_head: freeing b_committed_data

Sep 30 12:10:05 ashin-suse kernel: __journal_remove_journal_head: freeing b_committed_data

....

.....

.........

Sep 30 12:10:10 ashin-suse kernel: vmmon: Had to deallocate AWE 16 pages from vm driver ffff810007e38000

Sep 30 12:11:44 ashin-suse gconfd (ashin-4251): Failed to write "/home/ashin/.gconf/apps/nautilus/preferences/%gconf.xml": Failed to open "/home/ashin/.gconf/apps/nautilus/preferences/%gconf.xml.new": Read-only file system

Sep 30 12:11:44 ashin-suse gconfd (ashin-4251): Failed to sync one or more sources: Failed to write some configuration data to disk

Sep 30 12:12:00 ashin-suse gconfd (ashin-4251): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:12:00 ashin-suse gconfd (ashin-4251): Could not open saved state file '/home/ashin/.gconfd/saved_state.tmp' for writing: Read-only file system

Sep 30 12:12:00 ashin-suse gconfd (ashin-4251): Failed to write "/home/ashin/.gconf/apps/gnome-session/options/%gconf.xml": Failed to open "/home/ashin/.gconf/apps/gnome-session/options/%gconf.xml.new": Read-only file system

Sep 30 12:12:00 ashin-suse gconfd (ashin-4251): Failed to write "/home/ashin/.gconf/apps/nautilus/preferences/%gconf.xml": Failed to open "/home/ashin/.gconf/apps/nautilus/preferences/%gconf.xml.new": Read-only file system

Sep 30 12:12:00 ashin-suse gconfd (ashin-4251): Failed to sync one or more sources: Failed to write some configuration data to disk

Sep 30 12:12:04 ashin-suse gconfd (ashin-4251): Exiting

Sep 30 12:12:16 ashin-suse gdm[3924]: Couldn't authenticate user

Sep 30 12:12:49 ashin-suse gdm[3924]: run_session_child: Could not open ~/.xsession-errors

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): starting (version 2.14.0), pid 5569 user 'ashin'

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Resolved address "xml:readonly:/etc/opt/gnome/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Resolved address "xml:readwrite:/home/ashin/.gconf" to a read-only configuration source at position 1

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Resolved address "xml:readonly:/etc/opt/gnome/gconf/gconf.xml.defaults" to a read-only configuration source at position 2

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Resolved address "xml:readonly:/etc/opt/gnome/gconf/gconf.xml.schemas" to a read-only configuration source at position 3

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): None of the resolved addresses are writable; saving configuration settings will not be possible

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): No writable config sources successfully resolved, may not be able to save some configuration changes

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:13:21 ashin-suse gconfd (ashin-5569): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:13:22 ashin-suse gconfd (ashin-5569): Failed to open saved state file: Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system)

Sep 30 12:13:22 ashin-suse gconfd (ashin-5569): Failed to log addition of listener gnome-session (Failed: Failed to open gconfd logfile; won't be able to restore listeners after gconfd shutdown (Read-only file system));will not be able to restore this listener on gconfd restart, resulting in unreliable notification of configuration changes.

[Liz]

I believe ext3does support files over 2gb, however, there is a lot of posts around the internet stating its not reliable.

If your disk is still valid, what you can do is run the vmware diskmanager utility to convert it into 2gb chunks. But using Rh3 on ext3 we have files over 10g which are still working fine.