e1000: eth0 NIC Link is Down

[P4thos]

Hi all,

This night, we had a linux machine that has lost network connectivity.

Looking at the machine itself, I get the following logs : "Apr 19 03:01:57 machine-name kernel: [24351941.242260] e1000: eth0 NIC Link is Down" which explain why the machine has lost the connectivity.

I'm trying to understand why the network interface goes down. I have look at ESX logs but no result.

Is there somewhere on vmware where I can understand what happend ? ( i.e : if the problem is from vmware or not , etc ... )

Machine is a Linux Debian 7 with vmware-tools installed and driver e1000 for network card

Thanks in advance

Regards

[Finikiez]

Hi!

You need to look into vmkernel.log of ESXi host where VM was running and vmware.log in VM's folder on datastore.

[P4thos]

Thanks for the update.

Indeed, I had a look at the vmkernel.log and I saw the following :

2018-04-19T01:01:00.452Z cpu16:102454)NetPort: 1782: disabled port 0x400001b

2018-04-19T01:01:00.454Z cpu0:102454)CBT: 1569: Disconnecting the cbt device 446012ea-cbt with filehandle 1147146986

2018-04-19T01:01:00.487Z cpu20:102454)CBT: 1569: Disconnecting the cbt device 662012e6-cbt with filehandle 1713378022

2018-04-19T01:01:00.793Z cpu1:102454)CBT: 2316: Created device 7d7312f0-cbt for cbt driver with filehandle 2104693488

2018-04-19T01:01:00.793Z cpu1:102454)WARNING: CBT: 2080: Unsupported ioctl 61

2018-04-19T01:01:00.793Z cpu1:102454)WARNING: CBT: 2080: Unsupported ioctl 60

2018-04-19T01:01:00.793Z cpu1:102454)WARNING: CBT: 2080: Unsupported ioctl 44

2018-04-19T01:01:00.793Z cpu1:102454)VSCSI: 4005: handle 14987(vscsi0:0):Using sync mode due to sparse disks

2018-04-19T01:01:00.793Z cpu1:102454)VSCSI: 4047: handle 14987(vscsi0:0):Creating Virtual Device for world 101949 (FSS handle 1655771890) numBlocks=104857600 (bs=512)

2018-04-19T01:01:00.793Z cpu1:102454)VSCSI: 273: handle 14987(vscsi0:0):Input values: res=0 limit=-2 bw=-1 Shares=1000

2018-04-19T01:01:00.808Z cpu1:102454)CBT: 2316: Created device 3e7812f6-cbt for cbt driver with filehandle 1048056566

2018-04-19T01:01:00.808Z cpu1:102454)WARNING: CBT: 2080: Unsupported ioctl 61

2018-04-19T01:01:00.808Z cpu1:102454)WARNING: CBT: 2080: Unsupported ioctl 60

2018-04-19T01:01:00.808Z cpu1:102454)WARNING: CBT: 2080: Unsupported ioctl 44

2018-04-19T01:01:00.808Z cpu1:102454)VSCSI: 4005: handle 14988(vscsi0:1):Using sync mode due to sparse disks

2018-04-19T01:01:00.808Z cpu1:102454)VSCSI: 4047: handle 14988(vscsi0:1):Creating Virtual Device for world 101949 (FSS handle 713757432) numBlocks=52428800 (bs=512)

2018-04-19T01:01:00.808Z cpu1:102454)VSCSI: 273: handle 14988(vscsi0:1):Input values: res=0 limit=-2 bw=-1 Shares=1000

2018-04-19T01:01:00.808Z cpu1:102454)NetPort: 3110: resuming traffic on DV port 225

2018-04-19T01:01:00.808Z cpu1:102454)Team.etherswitch: TeamESPolicySet:5832: Adding uplink LAG_Customers-0 portset DvsPortset-1 to frp allowedmap 0x3

2018-04-19T01:01:00.808Z cpu1:102454)Team.etherswitch: TeamESPolicySet:5832: Adding uplink LAG_Customers-1 portset DvsPortset-1 to frp allowedmap 0x3

2018-04-19T01:01:00.808Z cpu1:102454)Team.etherswitch: TeamESPolicySet:5932: Port 0x400001b frp numUplinks 2 active 2(max 2) standby 0

2018-04-19T01:01:00.808Z cpu1:102454)Team.etherswitch: TeamESPolicySet:5940: Update: Port 0x400001b frp numUplinks 2 active 2(max 2) standby 0

2018-04-19T01:01:00.808Z cpu1:102454)NetPort: 1575: enabled port 0x400001b with mac 00:50:56:94:7c:4b

mac 00:50:56:94:7c:4b is the one of the affected machine

I see the port has been disabled and enabled. But I don't understand why

[Finikiez]

Messages between this two lines indicate that snapshot had been created. More that likely you have backup at this time. Nothing wrong with this.

Strange thig that Guest OS couldn't restore connection.

What is ESXi version you use?

What is Guest OS?

Have you ever had this problem before?

[P4thos]

You are right. We have veeam for our backup system and this is why you see snapshot.

ESXi version is 6.0.0, 6921384

VM OS is Debian 7

This is the 1rst time the problem occurred.

I have asked our backup team if they put an update on veeam or change a configuration but they told me nothing has changed. Veeam logs are ok.

[Finikiez]

03:01:57 is in a minute after snapshot creation time

is there anything else in vmkernel and vmware.log after the moment when snapshot has been created?

Or maybe any other messages in Debian logs regarding e1000 driver?

[P4thos]

In the log vmware.log, nothing special or at least nothing more than the previous log written in that ticket.

I have compared le log from that machine and another one ( his twin, machines are in cluster ) to see if there is something different. But the logs are identical ( I'm talking about log made during snapshot ).

On the machine itsef, I see logs " Kernel : e1000: eth0 NIC Link is Down" and then the machine trigger a Call Trace regarding driver e1000 :

Apr 19 03:04:33 machine_name kernel: [24352097.784609] kworker/0:0     D ffff88013fc127c0     0  4695      2 0x00000000

Apr 19 03:04:33 machine_name kernel: [24352097.784617]  ffff880137ac3180 0000000000000046 ffff880100000000 ffffffff8160d020

Apr 19 03:04:33 machine_name kernel: [24352097.784624]  00000000000127c0 ffff880136c4dfd8 ffff880136c4dfd8 ffff880137ac3180

Apr 19 03:04:33 machine_name kernel: [24352097.784629]  ffff880139540c08 000000018110a61a 000000c0af981fcd 7fffffffffffffff

Apr 19 03:04:33 machine_name kernel: [24352097.784635] Call Trace:

Apr 19 03:04:33 machine_name kernel: [24352097.784683]  [<ffffffff813534e3>] ? schedule_timeout+0x2c/0xdb

Apr 19 03:04:33 machine_name kernel: [24352097.784709]  [<ffffffff8103817f>] ? set_next_entity+0x32/0x55

Apr 19 03:04:33 machine_name kernel: [24352097.784729]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa

Apr 19 03:04:33 machine_name kernel: [24352097.784743]  [<ffffffff8100d66d>] ? __switch_to+0x101/0x283

Apr 19 03:04:33 machine_name kernel: [24352097.784748]  [<ffffffff81353129>] ? wait_for_common+0xa0/0x119

Apr 19 03:04:33 machine_name kernel: [24352097.784757]  [<ffffffff8103f81d>] ? try_to_wake_up+0x197/0x197

Apr 19 03:04:33 machine_name kernel: [24352097.784775]  [<ffffffff8105b196>] ? wait_on_work+0xe6/0x11c

Apr 19 03:04:33 machine_name kernel: [24352097.784780]  [<ffffffff8105a7b8>] ? worker_set_flags+0x8f/0x8f

Apr 19 03:04:33 machine_name kernel: [24352097.784785]  [<ffffffff8105be3e>] ? __cancel_work_timer+0xb2/0xf4

Apr 19 03:04:33 machine_name kernel: [24352097.784804]  [<ffffffffa005c88a>] ? e1000_down_and_stop+0x2f/0x48 [e1000]

Apr 19 03:04:33 machine_name kernel: [24352097.784813]  [<ffffffffa005f1db>] ? e1000_down+0x108/0x15e [e1000]

Apr 19 03:04:33 machine_name kernel: [24352097.784822]  [<ffffffffa006137f>] ? e1000_reset_task+0x53/0x6c [e1000]

Apr 19 03:04:33 machine_name kernel: [24352097.784827]  [<ffffffff8105bac7>] ? process_one_work+0x179/0x299

Apr 19 03:04:33 machine_name kernel: [24352097.784832]  [<ffffffff8105caa8>] ? worker_thread+0xc2/0x145

Apr 19 03:04:33 machine_name kernel: [24352097.784836]  [<ffffffff8105c9e6>] ? manage_workers.isra.25+0x15b/0x15b

Apr 19 03:04:33 machine_name kernel: [24352097.784846]  [<ffffffff8105fbf1>] ? kthread+0x76/0x7e

Apr 19 03:04:33 machine_name kernel: [24352097.784862]  [<ffffffff8135b634>] ? kernel_thread_helper+0x4/0x10

Apr 19 03:04:33 machine_name kernel: [24352097.784868]  [<ffffffff8105fb7b>] ? kthread_worker_fn+0x139/0x139

Apr 19 03:04:33 machine_name kernel: [24352097.784874]  [<ffffffff8135b630>] ? gs_change+0x13/0x13

Apr 19 03:04:33 machine_name kernel: [24352097.785073] kworker/0:1     D ffff88013fc127c0     0 16493      2 0x00000000

Apr 19 03:04:33 machine_name kernel: [24352097.785078]  ffff880137de7100 0000000000000046 0000000000000000 ffffffff8160d020

Apr 19 03:04:33 machine_name kernel: [24352097.785084]  00000000000127c0 ffff88013af03fd8 ffff88013af03fd8 ffff880137de7100

Apr 19 03:04:33 machine_name kernel: [24352097.785089]  ffff88013fc0d500 00000001810419ed 0000000000000002 ffff88013763d000

[Finikiez · Accepted solution]

Ok, as you see this is e1000 driver crash in Debian OS.

This is not a VMware issue as e1000 driver is included in guest os.

This needs to be checked in known bugs in Debian.

[P4thos]

OK thanks for your help.

I thought the crash was due because of a vmware side issue.

I will check the Debian bug