7 Replies Latest reply on Oct 17, 2019 6:19 PM by BobAgi

    Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?

    BobAgi Enthusiast

      So I am running a VMWare ESX virtual machine with Ubuntu Server 16.04 LTS and I was updating it today.

      Then I got this message from the console at the end of sudo apt upgrade:

       

      update-initramfs: Generating /boot/initrd.img-4.4.0-165-generic

      W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.

       

      gzip: stdout: No space left on device

      E: mkinitramfs failure find 141 cpio 141 gzip 1

      update-initramfs: failed for /boot/initrd.img-4.4.0-165-generic with 1.

      dpkg: error processing package initramfs-tools (--unpack):

      subprocess installed post-installation script returned error exit status 1

      E: Sub-process /usr/bin/dpkg returned an error code (1)[/code]

       

      Worrysome, so I checked the drive space:

      [code]

      ~$ df -h

      Filesystem                         Size  Used Avail Use% Mounted on

      udev                               476M     0  476M   0% /dev

      tmpfs                              100M   15M   85M  15% /run

      /dev/mapper/agivpnserver--vg-root   28G  4.8G   22G  18% /

      tmpfs                              497M     0  497M   0% /dev/shm

      tmpfs                              5.0M     0  5.0M   0% /run/lock

      tmpfs                              497M     0  497M   0% /sys/fs/cgroup

      /dev/sda1                          472M  461M     0 100% /boot

      tmpfs                              100M     0  100M   0% /run/user/1000

       

      # pvdisplay

        --- Physical volume ---

        PV Name               /dev/sda5

        VG Name               agivpnserver-vg

        PV Size               29.52 GiB / not usable 2.00 MiB

        Allocatable           yes (but full)

        PE Size               4.00 MiB

        Total PE              7557

        Free PE               0

        Allocated PE          7557

        PV UUID               UyylJC-4uof-Gvmz-ZscZ-vYoq-Ycbg-kyP5AF

       

      Notice that the virtual disk I use has lots of free space but just not the boot partition on /dev/sda1...

       

      What can I do to remedy this?

      I am just about to upgrade to Ubuntu Server 18.04 LTS, so presumably I need a larger boot partition.

       

      I am not an Ubuntu admin level user so the disk space here was allocated by the Ubuntu installer when it was set up 3 years ago.

        • 1. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
          continuum Guru
          User ModeratorsvExpertCommunity Warriors

          Sounds like the VM has been upgraded a few times already so there is no more free space in /boot to build new boot-files.

          Probably you can delete older files from /boot that are no longer required.

          Seeing a list of files in /boot would be useful.

          • 2. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
            BobAgi Enthusiast

            Here is the files list:

             

            # ls -l

            total 462346

            -rw-r--r-- 1 root root  1252309 Oct  2  2018 abi-4.4.0-138-generic

            -rw-r--r-- 1 root root  1252429 Dec  5  2018 abi-4.4.0-141-generic

            -rw-r--r-- 1 root root  1252376 Jan 16  2019 abi-4.4.0-142-generic

            -rw-r--r-- 1 root root   190611 Oct  2  2018 config-4.4.0-138-generic

            -rw-r--r-- 1 root root   190611 Dec  5  2018 config-4.4.0-141-generic

            -rw-r--r-- 1 root root   190580 Jan 16  2019 config-4.4.0-142-generic

            -rw-r--r-- 1 root root   190560 Feb  7  2019 config-4.4.0-143-generic

            -rw-r--r-- 1 root root   190580 Mar 26  2019 config-4.4.0-145-generic

            -rw-r--r-- 1 root root   190871 Jun 11 04:47 config-4.4.0-151-generic

            -rw-r--r-- 1 root root   190871 Jun 25 01:43 config-4.4.0-154-generic

            -rw-r--r-- 1 root root   190973 Aug 27 04:24 config-4.4.0-161-generic

            -rw-r--r-- 1 root root   190973 Sep 17 15:18 config-4.4.0-165-generic

            drwxr-xr-x 5 root root     1024 Oct  7 14:23 grub

            -rw-r--r-- 1 root root 40665721 Sep  5 06:59 initrd.img-4.4.0-138-generic

            -rw-r--r-- 1 root root 40670031 Sep  5 06:58 initrd.img-4.4.0-141-generic

            -rw-r--r-- 1 root root 40680922 Sep  5 06:58 initrd.img-4.4.0-142-generic

            -rw-r--r-- 1 root root 40679389 Sep  5 06:58 initrd.img-4.4.0-143-generic

            -rw-r--r-- 1 root root 40687554 Sep  5 06:57 initrd.img-4.4.0-145-generic

            -rw-r--r-- 1 root root 40685710 Sep  5 06:57 initrd.img-4.4.0-151-generic

            -rw-r--r-- 1 root root 40687028 Sep  5 06:59 initrd.img-4.4.0-154-generic

            -rw-r--r-- 1 root root 40700264 Oct  7 14:23 initrd.img-4.4.0-161-generic

            -rw-r--r-- 1 root root 40706976 Oct  7 14:23 initrd.img-4.4.0-165-generic

            drwx------ 2 root root    12288 Feb 23  2018 lost+found

            -rw-r--r-- 1 root root      255 Oct  2  2018 retpoline-4.4.0-138-generic

            -rw-r--r-- 1 root root      255 Dec  5  2018 retpoline-4.4.0-141-generic

            -rw-r--r-- 1 root root      255 Jan 16  2019 retpoline-4.4.0-142-generic

            -rw------- 1 root root  3904838 Oct  2  2018 System.map-4.4.0-138-generic

            -rw------- 1 root root  3905747 Dec  5  2018 System.map-4.4.0-141-generic

            -rw------- 1 root root  3904797 Jan 16  2019 System.map-4.4.0-142-generic

            -rw------- 1 root root  3905647 Feb  7  2019 System.map-4.4.0-143-generic

            -rw------- 1 root root  3906115 Mar 26  2019 System.map-4.4.0-145-generic

            -rw------- 1 root root  3908794 Jun 11 04:47 System.map-4.4.0-151-generic

            -rw------- 1 root root  3908872 Jun 25 01:43 System.map-4.4.0-154-generic

            -rw------- 1 root root  3911629 Aug 27 04:24 System.map-4.4.0-161-generic

            -rw------- 1 root root  3911756 Sep 17 15:18 System.map-4.4.0-165-generic

            -rw------- 1 root root  7166688 Oct  2  2018 vmlinuz-4.4.0-138-generic

            -rw------- 1 root root  7171392 Dec  5  2018 vmlinuz-4.4.0-141-generic

            -rw------- 1 root root  7184032 Jan 16  2019 vmlinuz-4.4.0-142-generic

            -rw------- 1 root root  7190840 Feb 12  2019 vmlinuz-4.4.0-143-generic

            -rw------- 1 root root  7188984 Mar 27  2019 vmlinuz-4.4.0-145-generic

            -rw------- 1 root root  7198776 Jun 17 12:19 vmlinuz-4.4.0-151-generic

            -rw------- 1 root root  7199000 Jun 25 11:58 vmlinuz-4.4.0-154-generic

            -rw------- 1 root root  7206200 Aug 28 08:03 vmlinuz-4.4.0-161-generic

            -rw------- 1 root root  7201336 Sep 18 08:13 vmlinuz-4.4.0-165-generic

             

            And, yes I have made lots of

            apt update

            apt upgrade

            operations to keep the system up-to-date.

             

            Is there a command to remove un-needed old files here?

            I mean files that are *really* not needed....

            I mean a safe command like autoremove in apt?

            • 3. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
              continuum Guru
              Community WarriorsUser ModeratorsvExpert

              I am not aware of such a safe command.

              I would check with uname -a and a look at the bootoptions at startup.

              Once you know which kernel versions are required you should be able to identify those versions that you will not need anymore.

              Then create a snapshot of the VM - so that you can revert to snapshot if something goes wrong and then manually delete the old files like

              -rw-r--r-- 1 root root 40665721 Sep  5 06:59 initrd.img-4.4.0-138-generic

              -rw-r--r-- 1 root root 40670031 Sep  5 06:58 initrd.img-4.4.0-141-generic

              -rw-r--r-- 1 root root 40680922 Sep  5 06:58 initrd.img-4.4.0-142-generic

              -rw-r--r-- 1 root root 40679389 Sep  5 06:58 initrd.img-4.4.0-143-generic

              -rw-r--r-- 1 root root 40687554 Sep  5 06:57 initrd.img-4.4.0-145-generic

              • 4. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
                BobAgi Enthusiast

                Unfortunately I had not yet read your post on making a snapshot in VMWare. 

                Looked at StackExchange for a solution too.

                 

                So I ran this to verify the currently running kernel:

                # uname -r

                4.4.0-165-generic

                 

                So figured that the kernel running is 4.4.0-165

                 

                Looking at the StackExchange post I found what I deemed a "safe" apt command to use:

                apt autoremove linux-image-a.b.c linux-image-x.y.z

                 

                So for testing that would become like this in my case:

                # apt autoremove linux-image-4.4.0-138

                 

                However, unfortunately this screwed up since it grabbed a lot more than 138...

                Now after it finished I have:

                # ls -l

                total 102005

                -rw-r--r-- 1 root root   190973 Aug 27 04:24 config-4.4.0-161-generic

                -rw-r--r-- 1 root root   190973 Sep 17 15:18 config-4.4.0-165-generic

                drwxr-xr-x 5 root root     1024 Oct 15 10:35 grub

                -rw-r--r-- 1 root root 40700264 Oct  7 14:23 initrd.img-4.4.0-161-generic

                -rw-r--r-- 1 root root 40706976 Oct  7 14:23 initrd.img-4.4.0-165-generic

                drwx------ 2 root root    12288 Feb 23  2018 lost+found

                -rw------- 1 root root  3911629 Aug 27 04:24 System.map-4.4.0-161-generic

                -rw------- 1 root root  3911756 Sep 17 15:18 System.map-4.4.0-165-generic

                -rw------- 1 root root  7206200 Aug 28 08:03 vmlinuz-4.4.0-161-generic

                -rw------- 1 root root  7201336 Sep 18 08:13 vmlinuz-4.4.0-165-generic

                 

                # df -h

                Filesystem                         Size  Used Avail Use% Mounted on

                udev                               476M     0  476M   0% /dev

                tmpfs                              100M   15M   85M  15% /run

                /dev/mapper/agivpnserver--vg-root   28G  2.5G   25G  10% /

                tmpfs                              497M     0  497M   0% /dev/shm

                tmpfs                              5.0M     0  5.0M   0% /run/lock

                tmpfs                              497M     0  497M   0% /sys/fs/cgroup

                /dev/sda1                          472M  109M  339M  25% /boot

                tmpfs                              100M     0  100M   0% /run/user/1000

                 

                Thought that it would be OK, but when doing apt update/upgrade I get to a difficult situation:

                # apt update

                ....

                # apt upgrade

                Reading package lists... Done

                Building dependency tree

                Reading state information... Done

                Calculating upgrade... Done

                Some packages could not be installed. This may mean that you have

                requested an impossible situation or if you are using the unstable

                distribution that some required packages have not yet been created

                or been moved out of Incoming.

                The following information may help to resolve the situation:

                 

                The following packages have unmet dependencies:

                linux-image-extra-4.4.0-138-generic : Depends: linux-image-4.4.0-138-generic but it is not going to be installed

                E: Broken packages

                 

                And of course I was too hasty, I could have copied the entire /boot partition content over to somewhere in /root where there is plenty of space...

                But I didn't.

                 

                So it seems I need to manually remove the linux-image-extra-4.4.0-138-generic package, but how?

                Can apt be used here?

                 

                I used the following dpkg command also found at StackExchange to list related packages:

                 

                # dpkg -l | grep linux-image

                rc  linux-image-4.4.0-116-generic         4.4.0-116.140     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-4.4.0-138-generic         4.4.0-138.164     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-4.4.0-141-generic         4.4.0-141.167     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-4.4.0-142-generic         4.4.0-142.168     amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-4.4.0-143-generic         4.4.0-143.169     amd64  Signed kernel image generic

                rc  linux-image-4.4.0-145-generic         4.4.0-145.171     amd64  Signed kernel image generic

                rc  linux-image-4.4.0-151-generic         4.4.0-151.178     amd64  Signed kernel image generic

                rc  linux-image-4.4.0-154-generic         4.4.0-154.181     amd64  Signed kernel image generic

                ii  linux-image-4.4.0-161-generic         4.4.0-161.189     amd64  Signed kernel image generic

                ii  linux-image-4.4.0-165-generic         4.4.0-165.193     amd64  Signed kernel image generic

                rc  linux-image-4.4.0-87-generic          4.4.0-87.110      amd64  Linux kernel image for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-extra-4.4.0-116-generic   4.4.0-116.140     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

                rH  linux-image-extra-4.4.0-138-generic   4.4.0-138.164     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-extra-4.4.0-141-generic   4.4.0-141.167     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-extra-4.4.0-142-generic   4.4.0-142.168     amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

                rc  linux-image-extra-4.4.0-87-generic    4.4.0-87.110      amd64  Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP

                ii  linux-image-generic                   4.4.0.165.173     amd64  Generic Linux kernel image

                 

                Am I dead in the bucket now?

                • 5. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
                  continuum Guru
                  User ModeratorsCommunity WarriorsvExpert

                  > Unfortunately I had not yet read your post on making a snapshot in VMWare.

                  ? - thats a basic feature - should be available with the webinterface in any ESXi version.

                  I think the real question is why your Ubuntu thinks that it still needs the old kernel files.

                  I would check in synaptic and uninstall older kernel images that you dont need anymore.

                  • 6. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
                    BobAgi Enthusiast

                    Cannot get to the web interface since by browsers refuse (both Chrome and Firefox) due to its use of Flash....

                     

                    But we do have the deprecated VMWare vSphere Client which can connect...

                    However its now probably too late.

                     

                    EDIT:

                    I used a combination of your replies and StackExchange advice to do the following (from history):

                     

                      200  apt-cache search linux-image-4.4.0-138-generic

                      201  apt remove linux-image-4.4.0-138-generic  => succeeded

                      202  apt update  => also succeeded

                      203  apt list --upgradable

                      204  apt upgrade  => Ran successfully now!!

                     

                    So now I will take a snapshot in VMWare and then move on to the next, which is upgrading distribution to 18.04.

                    • 7. Re: Help! My Ubuntu 16.04 LTS vm has run out of space on /root, how to fix?
                      BobAgi Enthusiast

                      FINALLY

                      I took a snapshot of the current state after the cleanup above and then I used the vSphere Client to access the console for the server where I could do the upgrade to 18.04.

                      It took a while but it worked just fine in the end.

                      I will probably revisit the issue with the small boot partition and how to expand its size at a leter time. For now it works anyway.