Disk full

Disk full errors are really, really bad. Here's how to recover.

First, identify the files that are taking up all the disk space. Remove these files. If they can't be removed, you have to compress them to a different partition or move them off of the host.

Next, identify the processes that created these files. There is a possibility that they are still creating these files, and doing it at a rate that can't be managed. If this is the case, they will have to be killed and fixed. Otherwise, you can buy time by removing the large files from time to time.

Now, before you move on to the task of fixing the broken code, you're going to want to find all the side effects of a full disk and repair them ASAP.

The worst effect is going to be partial transactions. You need to understand how the system's transactions work in order to fix this properly. Most mature systems recover well from disk full errors, but some do not.

After you have the system running again, you can repair the offending code. Most likely, it was a logfile that just had too many messages in it. You may want to investigate a logging system that compresses older log files or moves them off of the host. Or you may want to investigate having more meaningful and fewer log messages.