Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?

[BobAgi]

So I am running a VMWare ESX virtual machine with Ubuntu Server 16.04 LTS and I was updating it today.

Then I got this message from the console at the end of sudo apt upgrade:

update-initramfs: Generating /boot/initrd.img-4.4.0-165-generic

W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.

gzip: stdout: No space left on device

E: mkinitramfs failure find 141 cpio 141 gzip 1

update-initramfs: failed for /boot/initrd.img-4.4.0-165-generic with 1.

dpkg: error processing package initramfs-tools (--unpack):

subprocess installed post-installation script returned error exit status 1

E: Sub-process /usr/bin/dpkg returned an error code (1)[/code]

Worrysome, so I checked the drive space:

[code]

~$ df -h

Filesystem                         Size  Used Avail Use% Mounted on

udev                               476M     0  476M   0% /dev

tmpfs                              100M   15M   85M  15% /run

/dev/mapper/agivpnserver--vg-root   28G  4.8G   22G  18% /

tmpfs                              497M     0  497M   0% /dev/shm

tmpfs                              5.0M     0  5.0M   0% /run/lock

tmpfs                              497M     0  497M   0% /sys/fs/cgroup

/dev/sda1                          472M  461M     0 100% /boot

tmpfs                              100M     0  100M   0% /run/user/1000

# pvdisplay

  --- Physical volume ---

  PV Name               /dev/sda5

  VG Name               agivpnserver-vg

  PV Size               29.52 GiB / not usable 2.00 MiB

  Allocatable           yes (but full)

  PE Size               4.00 MiB

  Total PE              7557

  Free PE               0

  Allocated PE          7557

  PV UUID               UyylJC-4uof-Gvmz-ZscZ-vYoq-Ycbg-kyP5AF

Notice that the virtual disk I use has lots of free space but just not the boot partition on /dev/sda1...

What can I do to remedy this?

I am just about to upgrade to Ubuntu Server 18.04 LTS, so presumably I need a larger boot partition.

I am not an Ubuntu admin level user so the disk space here was allocated by the Ubuntu installer when it was set up 3 years ago.

[continuum]

Sounds like the VM has been upgraded a few times already so there is no more free space in /boot to build new boot-files.

Probably you can delete older files from /boot that are no longer required.

Seeing a list of files in /boot would be useful.

[BobAgi]

Here is the files list:

# ls -l

total 462346

-rw-r--r-- 1 root root  1252309 Oct  2  2018 abi-4.4.0-138-generic

-rw-r--r-- 1 root root  1252429 Dec  5  2018 abi-4.4.0-141-generic

-rw-r--r-- 1 root root  1252376 Jan 16  2019 abi-4.4.0-142-generic

-rw-r--r-- 1 root root   190611 Oct  2  2018 config-4.4.0-138-generic

-rw-r--r-- 1 root root   190611 Dec  5  2018 config-4.4.0-141-generic

-rw-r--r-- 1 root root   190580 Jan 16  2019 config-4.4.0-142-generic

-rw-r--r-- 1 root root   190560 Feb  7  2019 config-4.4.0-143-generic

-rw-r--r-- 1 root root   190580 Mar 26  2019 config-4.4.0-145-generic

-rw-r--r-- 1 root root   190871 Jun 11 04:47 config-4.4.0-151-generic

-rw-r--r-- 1 root root   190871 Jun 25 01:43 config-4.4.0-154-generic

-rw-r--r-- 1 root root   190973 Aug 27 04:24 config-4.4.0-161-generic

-rw-r--r-- 1 root root   190973 Sep 17 15:18 config-4.4.0-165-generic

drwxr-xr-x 5 root root     1024 Oct  7 14:23 grub

-rw-r--r-- 1 root root 40665721 Sep  5 06:59 initrd.img-4.4.0-138-generic

-rw-r--r-- 1 root root 40670031 Sep  5 06:58 initrd.img-4.4.0-141-generic

-rw-r--r-- 1 root root 40680922 Sep  5 06:58 initrd.img-4.4.0-142-generic

-rw-r--r-- 1 root root 40679389 Sep  5 06:58 initrd.img-4.4.0-143-generic

-rw-r--r-- 1 root root 40687554 Sep  5 06:57 initrd.img-4.4.0-145-generic

-rw-r--r-- 1 root root 40685710 Sep  5 06:57 initrd.img-4.4.0-151-generic

-rw-r--r-- 1 root root 40687028 Sep  5 06:59 initrd.img-4.4.0-154-generic

-rw-r--r-- 1 root root 40700264 Oct  7 14:23 initrd.img-4.4.0-161-generic

-rw-r--r-- 1 root root 40706976 Oct  7 14:23 initrd.img-4.4.0-165-generic

drwx------ 2 root root    12288 Feb 23  2018 lost+found

-rw-r--r-- 1 root root      255 Oct  2  2018 retpoline-4.4.0-138-generic

-rw-r--r-- 1 root root      255 Dec  5  2018 retpoline-4.4.0-141-generic

-rw-r--r-- 1 root root      255 Jan 16  2019 retpoline-4.4.0-142-generic

-rw------- 1 root root  3904838 Oct  2  2018 System.map-4.4.0-138-generic

-rw------- 1 root root  3905747 Dec  5  2018 System.map-4.4.0-141-generic

-rw------- 1 root root  3904797 Jan 16  2019 System.map-4.4.0-142-generic

-rw------- 1 root root  3905647 Feb  7  2019 System.map-4.4.0-143-generic

-rw------- 1 root root  3906115 Mar 26  2019 System.map-4.4.0-145-generic

-rw------- 1 root root  3908794 Jun 11 04:47 System.map-4.4.0-151-generic

-rw------- 1 root root  3908872 Jun 25 01:43 System.map-4.4.0-154-generic

-rw------- 1 root root  3911629 Aug 27 04:24 System.map-4.4.0-161-generic

-rw------- 1 root root  3911756 Sep 17 15:18 System.map-4.4.0-165-generic

-rw------- 1 root root  7166688 Oct  2  2018 vmlinuz-4.4.0-138-generic

-rw------- 1 root root  7171392 Dec  5  2018 vmlinuz-4.4.0-141-generic

-rw------- 1 root root  7184032 Jan 16  2019 vmlinuz-4.4.0-142-generic

-rw------- 1 root root  7190840 Feb 12  2019 vmlinuz-4.4.0-143-generic

-rw------- 1 root root  7188984 Mar 27  2019 vmlinuz-4.4.0-145-generic

-rw------- 1 root root  7198776 Jun 17 12:19 vmlinuz-4.4.0-151-generic

-rw------- 1 root root  7199000 Jun 25 11:58 vmlinuz-4.4.0-154-generic

-rw------- 1 root root  7206200 Aug 28 08:03 vmlinuz-4.4.0-161-generic

-rw------- 1 root root  7201336 Sep 18 08:13 vmlinuz-4.4.0-165-generic

And, yes I have made lots of

apt update

apt upgrade

operations to keep the system up-to-date.

Is there a command to remove un-needed old files here?

I mean files that are *really* not needed....

I mean a safe command like autoremove in apt?

[continuum]

I am not aware of such a safe command.

I would check with uname -a and a look at the bootoptions at startup.

Once you know which kernel versions are required you should be able to identify those versions that you will not need anymore.

Then create a snapshot of the VM - so that you can revert to snapshot if something goes wrong and then manually delete the old files like

-rw-r--r-- 1 root root 40665721 Sep  5 06:59 initrd.img-4.4.0-138-generic

-rw-r--r-- 1 root root 40670031 Sep  5 06:58 initrd.img-4.4.0-141-generic

-rw-r--r-- 1 root root 40680922 Sep  5 06:58 initrd.img-4.4.0-142-generic

-rw-r--r-- 1 root root 40679389 Sep  5 06:58 initrd.img-4.4.0-143-generic

-rw-r--r-- 1 root root 40687554 Sep  5 06:57 initrd.img-4.4.0-145-generic

[BobAgi]

Unfortunately I had not yet read your post on making a snapshot in VMWare. 

Looked at StackExchange for a solution too.

So I ran this to verify the currently running kernel:

# uname -r

4.4.0-165-generic

So figured that the kernel running is 4.4.0-165

Looking at the StackExchange post I found what I deemed a "safe" apt command to use:

apt autoremove linux-image-a.b.c linux-image-x.y.z

So for testing that would become like this in my case:

# apt autoremove linux-image-4.4.0-138

However, unfortunately this screwed up since it grabbed a lot more than 138...

Now after it finished I have:

# ls -l

total 102005

-rw-r--r-- 1 root root   190973 Aug 27 04:24 config-4.4.0-161-generic

-rw-r--r-- 1 root root   190973 Sep 17 15:18 config-4.4.0-165-generic

drwxr-xr-x 5 root root     1024 Oct 15 10:35 grub

-rw-r--r-- 1 root root 40700264 Oct  7 14:23 initrd.img-4.4.0-161-generic

-rw-r--r-- 1 root root 40706976 Oct  7 14:23 initrd.img-4.4.0-165-generic

drwx------ 2 root root    12288 Feb 23  2018 lost+found

-rw------- 1 root root  3911629 Aug 27 04:24 System.map-4.4.0-161-generic

-rw------- 1 root root  3911756 Sep 17 15:18 System.map-4.4.0-165-generic

-rw------- 1 root root  7206200 Aug 28 08:03 vmlinuz-4.4.0-161-generic

-rw------- 1 root root  7201336 Sep 18 08:13 vmlinuz-4.4.0-165-generic

# df -h

Filesystem                         Size  Used Avail Use% Mounted on

udev                               476M     0  476M   0% /dev

tmpfs                              100M   15M   85M  15% /run

/dev/mapper/agivpnserver--vg-root   28G  2.5G   25G  10% /

tmpfs                              497M     0  497M   0% /dev/shm

tmpfs                              5.0M     0  5.0M   0% /run/lock

tmpfs                              497M     0  497M   0% /sys/fs/cgroup

/dev/sda1                          472M  109M  339M  25% /boot

tmpfs                              100M     0  100M   0% /run/user/1000

Thought that it would be OK, but when doing apt update/upgrade I get to a difficult situation:

# apt update

....

# apt upgrade

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

Some packages could not be installed. This may mean that you have

requested an impossible situation or if you are using the unstable

distribution that some required packages have not yet been created

or been moved out of Incoming.

The following information may help to resolve the situation:

The following packages have unmet dependencies:

linux-image-extra-4.4.0-138-generic : Depends: linux-image-4.4.0-138-generic but it is not going to be installed

E: Broken packages

And of course I was too hasty, I could have copied the entire /boot partition content over to somewhere in /root where there is plenty of space...

But I didn't.

So it seems I need to manually remove the linux-image-extra-4.4.0-138-generic package, but how?

Can apt be used here?

I used the following dpkg command also found at StackExchange to list related packages:

# dpkg -l | grep linux-image

rc  linux-image-4.4.0-116-generic         4.4.0-116.140     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-4.4.0-138-generic         4.4.0-138.164     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-4.4.0-141-generic         4.4.0-141.167     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-4.4.0-142-generic         4.4.0-142.168     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-4.4.0-143-generic         4.4.0-143.169     amd64  Signed kernel image generic

rc  linux-image-4.4.0-145-generic         4.4.0-145.171     amd64  Signed kernel image generic

rc  linux-image-4.4.0-151-generic         4.4.0-151.178     amd64  Signed kernel image generic

rc  linux-image-4.4.0-154-generic         4.4.0-154.181     amd64  Signed kernel image generic

ii  linux-image-4.4.0-161-generic         4.4.0-161.189     amd64  Signed kernel image generic

ii  linux-image-4.4.0-165-generic         4.4.0-165.193     amd64  Signed kernel image generic

rc  linux-image-4.4.0-87-generic          4.4.0-87.110      amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-extra-4.4.0-116-generic   4.4.0-116.140     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

rH  linux-image-extra-4.4.0-138-generic   4.4.0-138.164     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-extra-4.4.0-141-generic   4.4.0-141.167     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-extra-4.4.0-142-generic   4.4.0-142.168     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

rc  linux-image-extra-4.4.0-87-generic    4.4.0-87.110      amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

ii  linux-image-generic                   4.4.0.165.173     amd64  Generic Linux kernel image

Am I dead in the bucket now?

[continuum]

> Unfortunately I had not yet read your post on making a snapshot in VMWare.

? - thats a basic feature - should be available with the webinterface in any ESXi version.

I think the real question is why your Ubuntu thinks that it still needs the old kernel files.

I would check in synaptic and uninstall older kernel images that you dont need anymore.

[BobAgi]

Cannot get to the web interface since by browsers refuse (both Chrome and Firefox) due to its use of Flash....

But we do have the deprecated VMWare vSphere Client which can connect...

However its now probably too late.

EDIT:

I used a combination of your replies and StackExchange advice to do the following (from history):

  200  apt-cache search linux-image-4.4.0-138-generic

  201  apt remove linux-image-4.4.0-138-generic  => succeeded

  202  apt update  => also succeeded

  203  apt list --upgradable

  204  apt upgrade  => Ran successfully now!!

So now I will take a snapshot in VMWare and then move on to the next, which is upgrading distribution to 18.04.

[BobAgi]

FINALLY

I took a snapshot of the current state after the cleanup above and then I used the vSphere Client to access the console for the server where I could do the upgrade to 18.04.

It took a while but it worked just fine in the end.

I will probably revisit the issue with the small boot partition and how to expand its size at a leter time. For now it works anyway.