vCenter — File system `/storage/log` is low on storage space

After a recent VCSA reboot, I was seeing the infamous `no healthy upstream` error from vCenter.

The first place to check for issues like this is VMware’s Virtual Appliance Management Interface (VAMI), located by default via HTTPS on port 5480. An administrator can use the appliance root password for this particular interface.

When reviewing this issue with the VAMI, I saw the following error:

Now, VCSA by design automatically rotates most logs available on the appliance using the open-source tool but nothing in this directory appears to be managed:

I’d say this particular log partition is going to need some manual cleanup every now and then. To open up the CLI, SSH into vCenter and execute the following command:

First, let’s get an idea of how full the disks are:

Note: The -m switch converts units into Megabytes

The log partition is definitely full. To take an inventory of disk usage, we’ll use the du utility, with the s (summarize) and m (megabytes) switches enabled, and then pass the output to sort with the n (numerical) and r (reverse) switches enabled to focus on the most important first.

Examining these folders further, quite a few of these are old and never rotated. VMware provides the following guidance on what’s safe or isn’t. Generally, Linux has issues with files being deleted out from under it, so obviously rotated logs can be safely removed. If this is a production system, I’d recommend calling VMware GSS instead of taking it upon yourself. The above command ( du -sm * | sort -nr) can be used in any working directory to see what is filling up the logs the most. Here are a few examples of what I deleted to make room:

From here, I like to verify that space is cleared:

Catalina and Tomcat names for the same thing. This software package proxies inbound HTTP requests to specific applications, allowing many developers to build code without having to construct a soup-to-nuts HTTP server. Other similar (but more recent) projects include Python’s

With HTTP Proxies and servers, it is useful to keep comprehensive records indicating “who did what”, both for security reasons (“whodunit”) and for debugging reasons. As a result, Tomcat is a serious log-hog wherever it exists, and it almost never reviews old logs. This is why I evaluated the change as relatively safe.

If this was not an appliance, I would have added a logrotate spec to automatically delete old files from this directory, but it is not recommended to alter VCSA in this way.

Originally published at https://blog.engyak.co.

--

--

--

I am a network engineer based out of Alaska, pursuing various methods of achieving SRE/NRE

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

XDC Network AMA with WadzPay

Empathy is a Must-Have, Not a Nice-to-Have

RabbitMQ setup in local using Docker.

Exploring Video Streaming #1

Integration of Github + Jenkins + Kubernetes and making data persistent.

Activate Azure4Students as a University of Cologne student

How no-code can help Product Managers

Why did we choose FAST API over Flask and Django for our RESTFUL Micro-services

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nick Schmidt

Nick Schmidt

I am a network engineer based out of Alaska, pursuing various methods of achieving SRE/NRE

More from Medium

NSX-T 3.2 and NSX ALB (Avi) Deployment Error — “Controller is not reachable. {0}”

Google Images best alternative — Naver Images API

360° IT Check #28 — Stack Overflow’s Copy and Paste Statistics, the Log4j Saga, And More!

PoP Os 20.04 doesn’t boot after entering Hybrid or Compute mode