No space left on device but df still shows available space

Written by Claudio Kuenzler - 1 comments

Published on February 24th 2015 - Listed in Linux

A website running PHP suddenly returned a 500 internal server error. By analyzing the logs, the following entries appeared, as soon as the website was requested:

[error] [client 1.2.3.4] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP Warning:  session_start(): open(/var/www/web/files/tmp/sess_2p27rr5idh5danjle3e237c4s4, O_RDWR) failed: No space left on device (28) in /var/www/web/updates/concrete5.6.3.1_updater/concrete/startup/session.php on line 34
[error] [client 1.2.3.4] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP Warning:  Unknown: open(/var/www/web/files/tmp/sess_suh3bf1kj5tm77ftp8kkaqma26, O_RDWR) failed: No space left on device (28) in Unknown on line 0
[error] [client 1.2.3.4] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/www/web/files/tmp) in Unknown on line 0
[error] [client 1.2.3.4] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP Warning:  session_start(): open(/var/www/web/files/tmp/sess_1q9pmqhmlh5p3cp41ohs2u6p44, O_RDWR) failed: No space left on device (28) in /var/www/web/updates/concrete5.6.3.1_updater/concrete/startup/session.php on line 34
[error] [client 1.2.3.4] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP Warning:  Unknown: open(/var/www/web/files/tmp/sess_gjbe8a55db44htincmlpi7u986, O_RDWR) failed: No space left on device (28) in Unknown on line 0
[error] [client 1.2.3.4] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/www/web/files/tmp) in Unknown on line 0

The message cannot be mistaken: There is no space left on the device to write the session file.

Strangely enough, df -h showed a completely different story:

df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/vg0-var   60G   41G   16G  73% /

But by manually trying to create a file, the message was confirmed:

cd /var/www/web/files/tmp; touch bla
touch: cannot touch `bla': No space left on device

And that's when I remembered the inodes (OK, granted, after a couple of strange looks to the output):

df -i
Filesystem           Inodes   IUsed  IFree IUse% Mounted on
/dev/mapper/vg0-var 3932160 3932160      0  100% /

Yep. No inodes left on the device. Therefore no more files can be created, even though there would be still enough available space left.

Possible solutions:
1) Remove files
2) Increase file system

Add a comment

Comments (newest first)

Alexander from Schweiz wrote on Feb 24th, 2015:

But don't assume, that IFree = 0 means, that there are no inodes free...

With btrfs, there are always "0" inodes free, but still you're able to create files.


$ df -Ti .
Dateisystem Typ Inodes IBenutzt IFrei IUse% Eingehängt auf
/dev/mapper/Intern-Daten btrfs 0 0 0 - /data

$ ls -la
insgesamt 0
drwxrwxr-x 1 ask ask 0 Feb 24 17:42 .
drwxrwxr-x 1 ask ask 264 Feb 24 17:42 ..

$ date > testfile

$ ls -la
insgesamt 4
drwxrwxr-x 1 ask ask 16 Feb 24 17:42 .
drwxrwxr-x 1 ask ask 264 Feb 24 17:42 ..
-rw-rw-r-- 1 ask ask 29 Feb 24 17:42 testfile

$ df -Ti .
Dateisystem Typ Inodes IBenutzt IFrei IUse% Eingehängt auf
/dev/mapper/Intern-Daten btrfs 0 0 0 - /data


Just sayin' :)