Highlighted
Enthusiast
Enthusiast

/storage/archive 100% usage

Good morning,

/storage/archive is 100% full on a vcenter 6.7 appliance

1.png

How to free some space on this partition ?

Tags (2)
33 Replies
Highlighted
Enthusiast
Enthusiast

The /storage/archive partition can be full by design, as it is aimed at storing as much WAL history as possible, and is automatically cleaned up by the archiver service by removing automatically the oldest WAL segments.

Note:- These files can be removed manually but not recommended as it will be automatically done by the archiver service. In any case it will not cause any issues to VCSA functionality. Also do not increase any partition on VCSA. VMware does not recommend increasing VCSA partition.

WAL data is Write-Ahead logging for the Postgres DB - transaction logging essentially. It should auto-rotate and engineering has designed the partition to fill up to maximize transaction history. The fix was to exclude this partition from the health alerting. It will still fill up but won't throw an alarm.

This issue is resolved in vCenter Server 6.7 Update 1b. As per the fix, we should not get warnings in the Health Status portion of the Summary Tab in the VAMI.

https://kb.vmware.com/s/article/57829

Highlighted
Enthusiast
Enthusiast

Meanwhile vROPS keeps generating critical alerts like crazy... Smiley Sad

Is there a way to create an ignore rule per individual disk on a specific VM in vROPS?

Highlighted
Contributor
Contributor

Its normal

See VMware Knowledge Base  57829

0 Kudos
Highlighted
Contributor
Contributor

Look i get the message that its by design and not harmful, but my nimble SAN appliance is alerting me multiple times daily about this partition on this VM as being full.

I can ignore the messages, but damn vmware that is shit design. Why dont you configure it to use 79% space and not give everyone a hard time here. Or maybe someone can tell me how to tweak the garbage cleanup script to run at like 70% as opposed to 100%. I am on vmware version 6.7.0.32000 of the vcenter appliance. Will updating it help? I see there is 6.7.0.40000 available. i would guess no, but correct me if i am wrong.

EDIT: if you are in the same situation, i found a funny hack work around. Simply set the alert warning to 100% of free space, AND THEN increase the datastore size by 1gb. But dont grow the disk Smiley Happy so then it will never be able to reach 100% because the actual physical disk is larger than the partition!

0 Kudos
Highlighted
Commander
Commander

As the VMware said here It's fixed in the VCSA 6.7 U1b

The /storage/archive partition can be full by design, as it is aimed at storing as much WAL history as possible, and is automatically cleaned up by the archiver service by removing automatically the oldest WAL segments.

However you can run ls -lrth in the /storage/archive/vpostgres and remove ( run rm) very old files

Please mark my comment as the Correct Answer if this solution resolved your problem
0 Kudos
Highlighted
Contributor
Contributor

if you find out let me know ! Cant believe they release this (profanity removed by moderator) into production like this !

0 Kudos
Highlighted
Enthusiast
Enthusiast

However: to prevent this from happening again, I've written some Nagios / Icinga checks for the vCenter Server Appliance:

https://blog.krogloth.de/vmware-vcenter-server-appliance-monitoring/

Maybe some finds them useful 🙂

-- Alex (VMware VCAP-DCV, NetApp NCIE, LPIC 2)
0 Kudos
Highlighted
Contributor
Contributor

Hi ,

  • The VAMI page shows the warning message:

File system /storage/archive is low on storage space. Increase the size of disk /storage/archive.

This is a Known warning till 6.7 it fixed in 6.7U1b and above.

This Warning is Harmless as it is by Design to full the file system so we can igonore it.

Please refer the below KB for more details :

VMware Knowledge Base : KB 57829

0 Kudos
Highlighted
Contributor
Contributor

Sorry i am having trouble reading between the lines here. How do i resolve this?

I woke up this AM to my vcenter telling me "Root Disk Exhaustion" which is not a great message.

Then i look at my disk is full, same as you all.

IF this is by design, why is vcenter alerting me?

I am WAY passed u1. i am on vcenter version 6.7.0.44000 which apparently is 6.7 U3g (LATEST)

I had just upgraded the vcenter this week and did not have this problem before that.

pastedImage_0.png

how do i resolve, can someone please post the steps that work for you? is it a database in gzip format? i assume if i delete a bunch of database files my database stops working. i have 14M left of the device.

EDIT: reading over the replies, i already had this problem! haha... and fixed the nimble alert portion of it, but now the error has infected my vmware console it seems...

0 Kudos
Highlighted
Virtuoso
Virtuoso

If you read all the post you should notice that this is some kind of circulation logging and in your up to date vCenter version you shouldnt get an alarm anymore.  Are you sure that the alarm is related to /storage/archive?

Regards,
Joerg

0 Kudos
Highlighted
Commander
Commander

If you ensure your problem is related to the VCSA disk space, please read my posts about how to fix your problem:

VCSA Low Disk Space Problem - Part 1

VCSA Low Disk Space Problem - Part 2

Please mark my comment as the Correct Answer if this solution resolved your problem
0 Kudos
Highlighted
Contributor
Contributor

I am pretty sure its that partition as none of the others are as full.

2020-07-06_14h39_00.png

0 Kudos
Highlighted
Contributor
Contributor

I'm running 6.7 Update 3j on my vCSA, and I can tell you that it's not eating its tail.

Which causes the services all to fail.

Which, in this particular instance, means my userbase can't log into their workstations (VDI).

0 Kudos