Highlighted
Enthusiast
Enthusiast

/storage/archive 100% usage

Good morning,

/storage/archive is 100% full on a vcenter 6.7 appliance

1.png

How to free some space on this partition ?

Tags (2)
33 Replies
Highlighted
Virtuoso
Virtuoso

Welcome to the club..... its quite often that a VCSA partition runs out of free space. Smiley Sad

cd /storage/archive
du -sh *

to identify the folder which holds most of the data and than go deeper and deeper. But please be carefull because for normal people its /storage/log which runs out of space.

My VCSA 6.5 doesnt have a /storage/archive to i cant tell you whats in and its save to remove.

Regards,

Joerg

0 Kudos
Highlighted
Enthusiast
Enthusiast

The files are in /storage/archive/vpostgres/

It looks like DB checkpoints.

The older ones are from July !

1.png

The last one is named "partial"

2.png

0 Kudos
Highlighted
Expert
Expert

Check this KB VMware Knowledge Base

0 Kudos
Highlighted
Contributor
Contributor

This happened to me a few weeks ago, and after I expanded the drive a couple of times it kept filling! I logged a call with VMware and they informed me that this is actually by design(!). It's an engineering change to use all available space on that disk (it is transaction log archiving); there is apparently automatic housekeeping that will remove the oldest files but not before the health status of the appliance is flagged as critical and our own in-house monitoring complains...

I was told a KB article is in preparation for this and a fix is possibly coming in 6.7 U2. We just stopped monitoring space on that disk and all remains well!

John

Highlighted
Enthusiast
Enthusiast

We're in the same boat, our /storage/archive is also full. Has the KB been released yet?


Please vote my post if it was useful to you!

Visit my blog at https://vcloudvision.com!
0 Kudos
Highlighted
Expert
Expert

Hi

This doesnt look like a bug however as per design looks like:

The filesystem is intentionally filled up.

We should not run the disk usage check on /storage/archive since it can and will run at 100% full per design.

Please mark this answer as correct and helpfull if this helps you.

regards

Gayathri

Highlighted
Contributor
Contributor

Yes - there is VMware Knowledge Base

Highlighted
Enthusiast
Enthusiast

Unfortunately, I found out about this after increasing the partition a few times and now I have 270GB of *.gz files in this partition.  Are these safe to delete until the fix in U2 is put into place?  Suggestions are welcome.

0 Kudos
Highlighted
Expert
Expert

I dont see a supported/recommended way to manually clear or delete files from storage/archive.

Because as it is stated in article: it is aimed at storing as much WAL history as possible, and is automatically cleaned up by the archiver service by removing automatically the oldest WAL segments.

Please consider marking this answer as correct and helpful if this helps you.

regards

GS

0 Kudos
Highlighted
Enthusiast
Enthusiast

But what if it isn't automatically cleaning them up Smiley Happy?


Please vote my post if it was useful to you!

Visit my blog at https://vcloudvision.com!
0 Kudos
Highlighted
Enthusiast
Enthusiast

The KB mentioned above was updated. U1b has the fix included

0 Kudos
Highlighted
Contributor
Contributor

I have upgraded my Vcenter 6.5 directly to 6.7U1b, and I do experience that issue.

95% of disk usage with files  in /storage/archive/vpostgres and dated from 2019-02-01 to 2019-02-11

0 Kudos
Highlighted
Expert
Expert

I am seeing same thing my /storage/archive partition is completely full

I am on 6.7 U1b build 11727113

any idea if I can delete these files?

Screen Shot 2019-04-24 at 7.11.27 AM.png

0 Kudos
Highlighted
Virtuoso
Virtuoso

Its by design that this /storage/archive  runs up to 100% and than older files are deleted to make place for new ones. The "bug" which exists in 6.7 was that VAMI reports an full filesystem. This is fixed in 6.7u2 i think.

Regards

Joerg

0 Kudos
Highlighted
Contributor
Contributor

I moved them to a dir and after 1 h deleted them.Everything still works fine  : )

0 Kudos
Highlighted
Commander
Commander

BTW, This issue is resolved in 6.7 Update 1b as per the release notes.

  • You might see health status warnings for low space in the archive partitionYou might see health status warnings in the Summary tab of the vCenter Server Appliance interface such as File system /storage/archive is low on storage space. Increase the size of disk /storage/archive. You might also see such warnings in the node status in the system configuration settings when using the vSphere Client. System health checks trigger the warning when the partition exceeds the limits specified in the statsMonitor.xml file, but the warning is not indicating a problem, since the archive service automatically cleans old segments, such as WAL history.This issue is resolved in this release.

Thanks,

MS

0 Kudos
Highlighted
Contributor
Contributor

it did not clean the old segments  ;  )  Thats why.    (update2 by the way....)

0 Kudos
Highlighted
Contributor
Contributor

find this:

Solution:

  1. SSH into the vCSA and login as root and execute the “shell” command to get shell access.
  2. Run the “df -h” command and verify the “/storage/archive” mount is at 90+% of use.
  3. Access the vCenter Server that the vCSA instance is running on and increase Disk 13 of the vCSA VM Hardware by a significant amount.
  4. In the SSH session to the vCSA, run the autogrow script “/usr/lib/applmgmt/support/scripts/autogrow.sh” .
  5. Run the “df -h” command and verify the “/storage/archive” mount Use percentage has decreased.
  6. After some time has passed, verify the vSphere Client System Configuration Node Health is “Good” .
  7. Verify the vCenter Server Appliance VAMI Health Status for Storage is “Good” .

It's ok.

Highlighted
Enthusiast
Enthusiast

Hello. If, this is the immediate solution, increase the space on the file system; after you must add fix for vCenter Server 6.7 Update 1b,

https://kb.vmware.com/s/article/57829

https://blog.ganser.com/increase-disk-space-on-vcsa-6-7/

pastedImage_0.png

pastedImage_1.png

pastedImage_2.png

Regards, RA