With ESX 4.0 the Service Console (or COS) is now installed on a virtual disk (esxconsole.vmdk) placed in a VMFS datastore.
The big problem is that the default partion schema build a "small" disk with around 8 GB of total space (note that is a virtual disk that contains more than one partition, conversely to most best practices on vmdk design ).
This mean that you have few space for the / partition (around 5 GB), expecially if you add 3rd part software (like Dell OpenManage or similar tools).
So if you have the / partition near at 100% of usage, what can you do?
The right solution is reinstall the host, better with ESXi instead of ESX (because the "legacy" ESX is at his end-of-life, next version will be only ESXi based).
Otherwise it is possible to extend the vmdk?
Officially no... it's a vmdk, but it's locked, there is no official way to move it and to change his size.
It's silly because is a vmdk... so why is not possible to use hot-add and Storage vMotion?
Move COS vmdk
To move the esxconsole.vmdk there is a good guide on this site:
Extend COS vmdk
This could be one way... but as written before is not a official way, it is unsupported and it is a "dirty" way!
If you can move the datastore to another host, then it is quite simple, you will have an offline access to the esxconsole.vmdk file and you can change it as you want.
But usually it is on the local datastore, and when the host is running it is locked...
And because it is on VMFS datastore is not so simple access it offline (for example with a Linux live distro).
The solution is follow the previous guide and use vmdkfstools command instead of cp to copy your vmdk and resize it.
The steps are:
reboot your host in Troubleshootin move (probably it is not necessay)
edit the /etc/vmware/esx.conf and change the "/boot/cosvmdk" value to a new vmdk (for example from esxconsole.vmdk to cos.vmdk)
reboot the host, it will go to the recovery shell (because the new vmdk does not exist!)
change directory in your COS datastore and your COS folder with cd /vmfs/volumes/DATASTORE_NAME/COS_FOLDER
build a copy of your COS vmdk (just to keep the original file) with vmkfstools -i esxconsole.vmdk cos.vmdk
the new file will have this line in the configuration file (I simple delete the single line, but probably is not necessary): ddb.deletable = "true"
resize the COS file, for example to set the new size to 10G use this command: vmkfstools -X 10G cos.vmdk
reboot your host in Troubleshooting mode (now is necessay)
type esxcfg-boot --rebuild
reboot in normal mode
it will boot probably again with the old file, if yes change again the configuration file /etc/vmware/esx.conf and reboot
now check the name of your cos "disk" with fdisk -l
use fdisk /dev/sdX (where X is the letter of the COS disk) to build a new primary partition (there is a primary partition that is free)
reboot again (to read the new partition table)
use mkfs.ext3 to format the new partition
mount it under /opt (of where you have to install the additional tools)
change /etc/fstab to mount the partition at boot time
Note that you need a physical access to the console (or a DRAC/ILOE/Network KVM access) because the recovery shell works without networking. The Troubleshooting mode instead has the networking and SSH enabled (and you can force it from grub configuration file).
The procedure has been tested on a ESX 4.0 U2 running into a VM on a vSphere 4.1 environment.
A solution more simple and effective is to add a second vmdk to the COS: