Corrupt File System in Fedora Core 6 (2.6.18) Guest

[SonyVaioTeam]

We have experienced two instances where this Linux file system become corrupt. We are looking at this problem in 4 areas.

1. Vmware

2. EMC

3. PHD technologies backup software

4. Linux OS

I have openned a support ticket with Vmware support, EMC, PHD and they found no trouble in their systems. I also asked to query their support database for any issues causing a guest file system to corrupt and nothing was found from these vendors.

One of the theory's is that the verison of ESX and the vesion of the kernel caused the Linux file system corruption. The guest ran on a ESX3.5 update 2 for two months OK then I moved it to a ESX3.5 update 3 host and the file system become corrupt after 2 days on the update[coincodence|http://www.google.com/search?hl=en&rlz=1B3GGGL_enUS297US298&ei=HX73SfiyDZDhtge_r8m6Dw&sa=X&oi=spell&resnum=1&ct=result&cd=1&q=coincodence&spell=1] or root cause?

Also our Linux consultant has found some specific kernel bugs causing ext3 systems to become corrupt in the FC6 kernel we are running. Not exactly in our application environment but stil corruption.

Any experiences with this issue?

thanks in advance.

Grant

[kjb007]

As far as the Update difference on ESX, I would find that improbable. What type of storage is the vm running on? Is it SAN or local storage?

-KjB

VMware vExpert

[SonyVaioTeam]

Fiber SAN

Grant Sabesky

Manager - Systems and Networks - Sony Vaio

San Diego, CA

[kjb007]

Any other difference between the two hosts? Did you move the vm to a new host, or upgrade the U2 host to U3, and then ran into corruption? What about the switching? Is that the same, maybe different paths that could contribute to SAN connectivity issues? Are all your filesystems corrupted, or a specific one? Does the vm boot?

-KjB

VMware vExpert

[SonyVaioTeam]

Hp dl585G5 update3, dell 2970 update2. Upgrades were months before the vm move. Same san switches, different paths to shared LUN...verified by EMC. All other vm's windows and linux are fine. The vm boots to shell indicating file system problems and need to run fsck.

Thanks for all your help.

Grant Sabesky

Manager - Systems and Networks - Sony Vaio

San Diego, CA

[kjb007]

Are the other Linux vm's at the same kernel? Were there snapshots on the vm you moved? Did you have vmware tools installed?

-KjB

VMware vExpert

[SonyVaioTeam]

Are the other Linux vm's at the same kernel? - NO

Were there snapshots on the vm you moved? - NO

Did you have vmware tools installed? - YES

-KjB

VMware vExpert

[kjb007]

So without snapshots, there isn't a direct link between ESX and the vm filesystems, per se. Did you get any further with the kernel issues you mentioned previously?

-KjB

VMware vExpert

[SonyVaioTeam]

We have not been able to recreate the corruption with our testing. One interesting fact from today....We are doing manual file system checks (fsck) each morning to verify for a few weeks. When we rebooted in the VI client the vm prompted us with this message. I checked the vmx file and it was set to lsilogic....why? There are two hard drives on this host…I only see on entry for a scsi virtual adapter = lsi. Should there be one entry for each drive?

[SonyVaioTeam]

Vmx file

# more Gforge_Dev_Server.vmx

#!/usr/bin/vmware

config.version = "8"

virtualHW.version = "4"

floppy0.present = "false"

nvram = "Gforge_Dev_Server.nvram"

deploymentPlatform = "windows"

virtualHW.productCompatibility = "hosted"

tools.upgrade.policy = "manual"

powerType.powerOff = "default"

powerType.powerOn = "default"

powerType.suspend = "default"

powerType.reset = "default"

displayName = "Gforge_Dev_Server_Restore2- active production server"

extendedConfigFile = "Gforge_Dev_Server.vmxf"

numvcpus = "2"

scsi0.present = "true"

scsi0.sharedBus = "none"

scsi0.virtualDev = "lsilogic"

memsize = "3072"

scsi0:0.present = "true"

scsi0:0.fileName = "Gforge_Dev_Server-000001.vmdk"

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:1.present = "true"

scsi0:1.fileName = "Gforge_Dev_Server_1-000001.vmdk"

scsi0:1.deviceType = "scsi-hardDisk"

ide0:0.present = "true"

ide0:0.fileName = "/dev/hda"

ide0:0.deviceType = "atapi-cdrom"

ide0:0.startConnected = "false"

ethernet0.present = "true"

ethernet0.wakeOnPcktRcv = "false"

ethernet0.networkName = "112CampusSubnet"

ethernet0.addressType = "vpx"

guestOS = "rhel4"

annotation = "_esXpress: 2009-04-29 05:12 - ERR - 0/2/2 disks, (0.0%) 0m/0g/103g (100% Data), 7h:29m:25s ON"

uuid.location = "56 4d c7 f7 e9 2b d5 af-79 f0 c9 ef 7c bb f1 9f"

uuid.bios = "50 2a c7 ac df 16 ca a2-19 cd 86 35 e9 a4 a2 00"

snapshot.action = "keep"

sched.cpu.min = "0"

sched.cpu.units = "mhz"

sched.cpu.shares = "normal"

sched.mem.minsize = "0"

sched.mem.shares = "normal"

toolScripts.afterPowerOn = "true"

toolScripts.afterResume = "true"

toolScripts.beforeSuspend = "true"

toolScripts.beforePowerOff = "true"

cpuid.1.eax = "xxxx

[kjb007]

The 2nd hard drive will be using the bus from the first scsi controller. If your controller type was changed, but your OS does not have the proper driver, then you wouldn't be able to boot, and not see any drives, but you can try to force the controller back to BusLogic and try it out.

-KjB

VMware vExpert

[Texiwill]

Hello,

Moved to Virtual Machine and Guest OS forum.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, DABCC Analyst
====
Author of the books 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment' available for pre-order now
'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
SearchVMware Pro|Blue Gears|Top Virtualization Security Links|Virtualization Security Round Table Podcast