After cloning a very large VM (5 VMDKs totaling 4.6TB) from Server 1 to Server 2 (see config below), my LVM data disk was mounted read-only. I enabled the fsck flag in the /etc/fstab for the mount point and rebooted. fsck dropped me into filesystem repair mode. I ran fsck I began getting a seemingly endless string of errors such as:
Inode 188104589 has illegal block(s). Clear? yes
Illegal block #0 (3662660783) in inode 188104589. CLEARED
Inode 188104589 is too big. Truncate? yes
Block #3 (441281053) causes symlink to be too big. CLEARED.
Too many illegal blocks in inode 188104589.
Inode 188104481, i_size is 2723283268511623476, should be 0. Fix? Yes.
Inode 188104481, i_blocks is 3404183318, should be 0. Fix? yes.
Inode 188103418 has a bad extended attribute block 1069320015. Clear? Yes.
Inode 188104482 has compression flag set on filesystem without compression support. Clear? yes
Inode 188104482 has INDEX_FL flag set but it is not a directory. Clear HTree index? yes.
I manually accepted each time a fix was suggested for several minutes but the process would not end. I canceled the process and re-ran fsck -p (automatic repair) which said I could not run in that mode because UNEXPECTED ???something (I forget) had been encountered. I re-ran with fsck -y (respond automatically with 'yes' to all prompts) for over 10 minutes, and the stream of errors continue on. A screen recording of these errors is attached.
I checked the original VM and the filesystem is not reporting any errors - I appears I haven't cloned over an existing problem.
Running e2fsck version 1.39. vCenter Server v4.1 initiated the clone.
When I cloned the VMs, I cloned from thick provisioned disks to thin provisioned disks.
Could data have been corrupted during the clone over our gigabit network? The process took 104 hours to complete (4.6TB thick provisioned disks to 1.75TB thin provisioned disks). I checked another small VM clone (40GB) from server 1 to server 2 - no file system errors being reported.
I have another VolGroup and Logical Volume configured for the operating system disk (40GB). It is not reporting any errors, although I haven't forced a check.
Can anyone recommend and avenues to isolate the problem?
Do the fsck errors I reported seem serious? Can I trust the data on that file system? It is not a production server, it was intended as a backup to our production file server.
Am I going too far down the rabbit hole? I can just format the file system and now rSync data from the production to the backup.
Server 1: Dell T610
Direct Attached Storage - PERC6i connected to internal storage
8x 1TB disks in RAID 5 array
3x 1.9TB logical disks + 1x 600GB logical disk
ESXi 4.0 installed on flash drive
VMFS v3.33 - 6.36TB datastore by combining extents from logical RAID disks
2x 2TB VMDK connected to FileServer VM & 1x 300GB VMDK connected to File Server VM as /dev/sdc, /dev/sdd, /dev/sde
LVM is used to combine disks into a single large logical volume (/dev/VG/VGDATA - 4.6TB)
LVM logical disk is formated in ext3
Server 2: Dell Poweredge 2950
Direct Attach Storage - PERC5e connected to MD1000
8x 1TB disks in RAID 5 array
3x 1.9TB logical disks + 1x 600GB logical disk
ESXi 4.1 installed on flash drive
VMFS v3.46 - 6.36TB datastore by combining extents from logical RAID disks
2x 2TB VMDK connected to FileServer VM & 1x 300GB VMDK connected to File Server VM as /dev/sdc, /dev/sdd, /dev/sde
LVM is used to combine disks into a single large logical volume (/dev/VG/VGDATA - 4.6TB)
LVM logical disk is formated in ext3
