LinkedIn Sourceforge Twitter

Vincent's Blog

Pleasure in the job puts perfection in the work (Aristote)

Do not log in /tmp

Posted on 2020-10-25 12:10:00 from Vincent in OpenBSD

On one of my OpenBSD machine, I had, after several months of usage, a full /tmp filesystem. Despite by checking files, none were big enough to take full space of /tmp. Why df shows a full filesystem ? and why no big files identified by ls, neither by du ?


Introduction

On one of my machine, I had a troubles because of /tmp filesystem is becoming full. The machine did not crash, but was not working correctly.

After investigation, the df command confirms the filesystem full.

But by doing ls or du, both do not list a big culprit file(s).

Nevertheless a reboot of the machine solve the situation.

So the mystery was why this behavior ?

The culprit is a log file

You have to know that, as a standard behavior, OpenBSD remove file older than 7 days from /tmp.

You can see this in your /etc/daily file.

find  .... -type f -atime +7 -execdir rm -f -- {}

Unfortunately, I had a daemon writing to a log file located in /tmp.

In such situation, the rm command remove the file from the /tmp folder, but since the filedescriptor is still open, the entries to the file are still there.
The rm will take effect as soon as the daemon free the filedescriptor.

To prove this, by killing the daemon free the space in /tmp.

Free space can be made by removing files AND by killing process writing to those files.

Possible solutions

Not tested, but one of the possible solution would be to detect if the file has an open filedescriptor before removing it. Maybe fstat could be used for that.

In /etc/daily, instead of doing:

find -x . \
    \( -path './ssh-*' -o -path ./.X11-unix -o -path ./.ICE-unix \
    -o -path './tmux-*' \) \
    -prune -o -type f -atime +7 -execdir rm -f -- {} \; 2>/dev/null

We could do:

find -x . \
    \( -path './ssh-*' -o -path ./.X11-unix -o -path ./.ICE-unix \
    -o -path './tmux-*' \) \
    -prune -o -type f -atime +7 -print | while read FILE
do
    [ "$(fstat "$FILE" | wc -l)" == "1" ] && rm -f -- "$FILE"
done

To be tested ;-)



4, 3
displayed: 3948
Comments:

1. From Philippe St-Jacques on Tue Nov 10 18:05:26 2020

Hello Vincent, I wanted to mention that one should not mount a filesystem in /tmp either, even for testing. I accidentally erased all my NAS two years ago because I added a line in /etc/fstab to mount my NAS using nfs and the 'auto' flag. I was testing different flags for NFS to work, but what happened when I rebooted OpenBSD, it was kind enough to clean all files in /tmp, which included my NAS ! I guess I was the only one stupid enough to do this, as I was new to OpenBSD and did not know it was clearing /tmp at each boot. Keep up the good work with this blog. Philippe

2. From Vincent on Tue Nov 10 22:03:28 2020

But you are right, the cleanup of /tmp should be a bit more clever. An other "side effect", is when you use a laptop and you work on it until 01:30 AM or more in the night. Indeed, at that time the "daily" script will be triggered by cron and the remove I talk will be triggered. If your laptop is mainly stopped and started via suspend, you will loose several files older than 7 days. Thanks for your support ;-)




What is the last letter of the word Python?