MattBri
Contributor
Contributor

e2fsck errors after cloning from ESXi 4.0 to ESXi 4.1

After cloning a very large VM (5 VMDKs totaling 4.6TB) from Server 1 to Server 2 (see config below), my LVM data disk was mounted read-only. I enabled the fsck flag in the /etc/fstab for the mount point and rebooted. fsck dropped me into filesystem repair mode. I ran fsck I began getting a seemingly endless string of errors such as:

Inode 188104589 has illegal block(s). Clear? yes

Illegal block #0 (3662660783) in inode 188104589. CLEARED

Inode 188104589 is too big. Truncate? yes

Block #3 (441281053) causes symlink to be too big. CLEARED.

Too many illegal blocks in inode 188104589.

Inode 188104481, i_size is 2723283268511623476, should be 0. Fix? Yes.

Inode 188104481, i_blocks is 3404183318, should be 0. Fix? yes.

Inode 188103418 has a bad extended attribute block 1069320015. Clear? Yes.

Inode 188104482 has compression flag set on filesystem without compression support. Clear? yes

Inode 188104482 has INDEX_FL flag set but it is not a directory. Clear HTree index? yes.

I manually accepted each time a fix was suggested for several minutes but the process would not end. I canceled the process and re-ran fsck -p (automatic repair) which said I could not run in that mode because UNEXPECTED ???something (I forget) had been encountered. I re-ran with fsck -y (respond automatically with 'yes' to all prompts) for over 10 minutes, and the stream of errors continue on. A screen recording of these errors is attached.

I checked the original VM and the filesystem is not reporting any errors - I appears I haven't cloned over an existing problem.

Running e2fsck version 1.39. vCenter Server v4.1 initiated the clone.

When I cloned the VMs, I cloned from thick provisioned disks to thin provisioned disks.

Could data have been corrupted during the clone over our gigabit network? The process took 104 hours to complete (4.6TB thick provisioned disks to 1.75TB thin provisioned disks). I checked another small VM clone (40GB) from server 1 to server 2 - no file system errors being reported.

I have another VolGroup and Logical Volume configured for the operating system disk (40GB). It is not reporting any errors, although I haven't forced a check.

Can anyone recommend and avenues to isolate the problem?

Do the fsck errors I reported seem serious? Can I trust the data on that file system? It is not a production server, it was intended as a backup to our production file server.

Am I going too far down the rabbit hole? I can just format the file system and now rSync data from the production to the backup.

Server 1: Dell T610

Direct Attached Storage - PERC6i connected to internal storage

8x 1TB disks in RAID 5 array

3x 1.9TB logical disks + 1x 600GB logical disk

ESXi 4.0 installed on flash drive

VMFS v3.33 - 6.36TB datastore by combining extents from logical RAID disks

2x 2TB VMDK connected to FileServer VM & 1x 300GB VMDK connected to File Server VM as /dev/sdc, /dev/sdd, /dev/sde

LVM is used to combine disks into a single large logical volume (/dev/VG/VGDATA - 4.6TB)

LVM logical disk is formated in ext3

Server 2: Dell Poweredge 2950

Direct Attach Storage - PERC5e connected to MD1000

8x 1TB disks in RAID 5 array

3x 1.9TB logical disks + 1x 600GB logical disk

ESXi 4.1 installed on flash drive

VMFS v3.46 - 6.36TB datastore by combining extents from logical RAID disks

2x 2TB VMDK connected to FileServer VM & 1x 300GB VMDK connected to File Server VM as /dev/sdc, /dev/sdd, /dev/sde

LVM is used to combine disks into a single large logical volume (/dev/VG/VGDATA - 4.6TB)

LVM logical disk is formated in ext3

0 Kudos
4 Replies
sabya1232003
Enthusiast
Enthusiast

Looks you have a very complex Volume configuration using multiple LUNs and different storage types.But that should not be a problem.The only issue i doubt is cloning from 4.0 to 4.1 ESXi environment.may be there is some incompatibility in storage while recognizing the new version of kernel.

you can try remove-add of the vmdk for the VM and check if that helps

0 Kudos
AndreTheGiant
Immortal
Immortal

In which way you have clone your VM?

Using vCenter Server?

Andre

Andre | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
0 Kudos
MattBri
Contributor
Contributor

Yes, vCenter Server v4.1 was used.

0 Kudos
AndreTheGiant
Immortal
Immortal

Are you making an hot clone?

In this case could be that filesystem is "dirty".

Andre

Andre | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
0 Kudos