Where did all my inodes go?

[Crusty]

So for a while now I've been having an issue with a server and it using more and more inodes with little to no disk usage. It wasn't a problem until the FS started to run out of inodes.

I found a neat little perl script to spit out the directories that used the most inodes and low and behold a .spam folder from someones mailbox shows almost 210k inodes(out of 500k) being used and only 1.3GB of disk usage.

web2-dltx:/home/vmail/bobmart.com/bob/.spam# du -sh new
1.3G    new

web2-dltx:/home/vmail/bobmart.com/bob/.spam# perl ~/inode.pl
      0 ./cur
      0 ./tmp
 208453 ./new
      8 .

So I cd into the directory and try to delete the files...

web2-dltx:/home/vmali/bobmart.com/bob/.spam/new# rm *
-bash: /bin/rm: Argument list too long

Oh noes I broke rm 😱

 

[Gamingphreek]

I believe 'rm *' is antiquated and has been replaced by doing a recursive remove on the directory. So 'rm -rf /home/vmali/bobmart.com/bob/.spam/new' should remove everything.

You also try switching into that directory and then using 'rm -rf .' and seeing what happens. I'm not sure it will work, but worth trying I guess.

-Kevin

Edit: Actually the Linux Kernel imposes an argument limit on commands it seems. This website details 4 ways around the problem. I have a hard time believing that an rm -rf on the directory itself will not work though.
http://www.linuxjournal.com/article/6060

 

[Crusty]

I believe 'rm *' is antiquated and has been replaced by doing a recursive remove on the directory. So 'rm -rf /home/vmali/bobmart.com/bob/.spam/new' should remove everything.

You also try switching into that directory and then using 'rm -rf .' and seeing what happens. I'm not sure it will work, but worth trying I guess.

-Kevin

Edit: Actually the Linux Kernel imposes an argument limit on commands it seems. This website details 4 ways around the problem. I have a hard time believing that an rm -rf on the directory itself will not work though.
http://www.linuxjournal.com/article/6060

I understand how rm works.... 😛 was more posting this for the fact that someone had 200k unread spam messages in their inbox.

 

[Red Squirrel]

This is actually one of the reasons I like the mbox format better. That and it's easier to setup. 😛 I've never got maildir working properly.

I wonder if the new ext4 handles inodes better such as having a higher limit, or no limit.

There's a way to use find in order to mass delete but I forget how. Super long command.

 

[Gamingphreek]

I understand how rm works.... 😛 was more posting this for the fact that someone had 200k unread spam messages in their inbox.

Oh jeez I'm sorry! 🙂 - On a similar note, I didn't know that the Kernel imposes parameter limitations so I did get to learn something.

200K unread spam though - I don't even know that I've received 200K messages in my entire life.

-Kevin

 

[Crusty]

Oh jeez I'm sorry! 🙂 - On a similar note, I didn't know that the Kernel imposes parameter limitations so I did get to learn something.

200K unread spam though - I don't even know that I've received 200K messages in my entire life.

-Kevin

What's scarier is that there are already 130 new ones sitting there! I might have to adjust some of the settings on my spam chain if this keeps up, or maybe I can finally convince someone to start migrating to Google Apps.

 

[Red Squirrel]

Oh jeez I'm sorry! 🙂 - On a similar note, I didn't know that the Kernel imposes parameter limitations so I did get to learn something.

200K unread spam though - I don't even know that I've received 200K messages in my entire life.

-Kevin

Sounds about right. I get a few hundred come in per day on all my accounts combined. I have my spam filter tweaked quite well, I rarely get false negatives or false positives. Last time I checked my spam folder it was over 9000. I should clear that once in a while. 😛

 

[skyking]

IIRC when I used spamassassin I set a level where the worst of the worst was dropped on the floor, and then I watched the folder for a while. I kept tweaking that up until I was getting most of the crap at the door.

 

[Nothinman]

So for a while now I've been having an issue with a server and it using more and more inodes with little to no disk usage. It wasn't a problem until the FS started to run out of inodes.

I found a neat little perl script to spit out the directories that used the most inodes and low and behold a .spam folder from someones mailbox shows almost 210k inodes(out of 500k) being used and only 1.3GB of disk usage.

web2-dltx:/home/vmail/bobmart.com/bob/.spam# du -sh new
1.3G    new

web2-dltx:/home/vmail/bobmart.com/bob/.spam# perl ~/inode.pl
      0 ./cur
      0 ./tmp
 208453 ./new
      8 .

So I cd into the directory and try to delete the files...

web2-dltx:/home/vmali/bobmart.com/bob/.spam/new# rm *
-bash: /bin/rm: Argument list too long

Oh noes I broke rm 😱

That's what you get for using a filesystem with a statically allocated inode table. =)

This is actually one of the reasons I like the mbox format better. That and it's easier to setup. I've never got maildir working properly.

The mbox format is never a better option. It's more prone to corruption and has locking issues because everything's in one file. Maildir is infinitely better and any MTA worth using supports it just fine.

I wonder if the new ext4 handles inodes better such as having a higher limit, or no limit.

According to this output, no:

/sbin/mkfs.ext4 ./test
mke2fs 1.41.9 (22-Aug-2009)
./test is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
128016 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
63 block groups
8192 blocks per group, 8192 fragments per group
2032 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

There's a way to use find in order to mass delete but I forget how. Super long command.

find . -type f -print0 | xargs -0 rm -f

That's not really super long. Just don't forget to adjust the find command to match the files you really want to delete.

 

[Crusty]

That's what you get for using a filesystem with a statically allocated inode table. =)

I didn't really have a choice in the filesystem. Those servers are all VPS that can be imaged with various things, none of them with advanced install options!

The mbox format is never a better option. It's more prone to corruption and has locking issues because everything's in one file. Maildir is infinitely better and any MTA worth using supports it just fine.

I agree, Maildir is far better to deal with and I don't recall having to do anything 'special' to get Postfix to deliver to them and Dovecot to read them for client access. Maybe enabling an option in a configuration file, but nothing like having to recompile the kernel!!

 

[Colt45]

ln -s /dev/null /home/vmail/bobmart.com/bob/.spam

 

[Red Squirrel]

The mbox format is never a better option. It's more prone to corruption and has locking issues because everything's in one file. Maildir is infinitely better and any MTA worth using supports it just fine.

I can't say I've ever had any issues with it, even mailboxes reaching in the 10GB range. I'm pretty sure the MTA takes care of locking and ensures it can't corrupt. It's snot like a PST file... those are super unstable when they get in the GB range.

I'd probably research deeper into how to get maildir to work if it were not for the inode limit, but as proven here, this is why maildir is a bad idea on a big server, the inode limit. To me this is a serious limitation though and hopefully they fix that in future versions of the file system. Guess there's always razorFS and other 3rd party file systems but those are more involved.

 

[Nothinman]

I can't say I've ever had any issues with it, even mailboxes reaching in the 10GB range. I'm pretty sure the MTA takes care of locking and ensures it can't corrupt. It's snot like a PST file... those are super unstable when they get in the GB range.

Size is irrelevant, if you're reading a mailbox and another message comes in and then your mail client saves your mailbox you've got a chance of corruption or at best the last message just gets dropped. There's no enforced file locking on unix like there is in Windows. There's a dozen different scenarios that could result in lost messages or a completely corrupt mbox file.

I'd probably research deeper into how to get maildir to work if it were not for the inode limit, but as proven here, this is why maildir is a bad idea on a big server, the inode limit. To me this is a serious limitation though and hopefully they fix that in future versions of the file system. Guess there's always razorFS and other 3rd party file systems but those are more involved.

No, Maildir is the best option for a big mail server because there's no contention issues. There's just no way around it it, mbox may be simple but it sucks and any MTA worth using supports Maildir. The number of inodes used is a very small consideration relative to everything else. AFAIK any rev of ext will never fix this issue because a static inode table is one of the most basic design decisions of the filesystem. The answer is other filesystems like XFS, JFS, etc and eventually Btrfs that do have dynamically allocated inodes.

 

[VinDSL]

Heh! I saw the N-word and had to comment...

I recently switched my production web site to a new server, and almost immediately ran out of inodes.

Evidently 750,000 inodes wasn't enough, and BIND, MySQL, et cetera, quit working.

Tech Support ratcheted it up to 7,500,000 inodes.

LoL! That should last me a while! :sneaky:

Anyway, carry on. I'm enjoying this thread...

 

[Red Squirrel]

You can add inodes? how does that work? In fact I can't seem to find how much inodes I have. It's something I totally tend to forget about keeping track of. I have 200GB free on my 1.7TB volume, but I could be on the verge of busting the inode limit depending on how many files I've got. Most of my files are big so think I'm ok. VMs, movies, mp3's, emails in mbox format, misc stuff etc... I'm actually doing a huge cleanup as I speak, since 200GB is cutting it close, well not really. My first hard drive was 10GB LOL.


Edit: just found the command to check inodes:

[root@borg incoming]# tune2fs -l /dev/md0
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          660ff2b7-269f-4706-b7e8-e0fc8eb8147f
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              122101760
Block count:              488379968
Reserved block count:     24418998
Free blocks:              107105823
Free inodes:              120981484
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      907
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Sat Sep 20 02:17:02 2008
Last mount time:          Sat Dec  5 20:00:05 2009
Last write time:          Sat Dec  5 20:00:05 2009
Mount count:              37
Maximum mount count:      20
Last checked:             Sat Sep 20 02:17:02 2008
Check interval:           15552000 (6 months)
Next check after:         Thu Mar 19 02:17:02 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Journal inode:            8
First orphan inode:       101883911
Default directory hash:   tea
Directory Hash Seed:      3082ffd3-ce4b-4811-8d4d-c7c8740be07a
Journal backup:           inode blocks
[root@borg incoming]#

Guess I'm not doing too bad at all!

 

[VinDSL]

You can add inodes? how does that work? In fact I can't seem to find how much inodes I have.

I use...

df -hi

And, if you're asking me how Tech Support added inodes - I *think* they are imposing an artificial limit, e.g. a quota.

Soooo... I don't *think* they actually added inodes. All they did was ratchet up my disk quota, in a conf file (or whatever). 😉

 

[n0cmonkey]

Heh! I saw the N-word and had to comment...

I recently switched my production web site to a new server, and almost immediately ran out of inodes.

Evidently 750,000 inodes wasn't enough, and BIND, MySQL, et cetera, quit working.

Tech Support ratcheted it up to 7,500,000 inodes.

LoL! That should last me a while! :sneaky:

Anyway, carry on. I'm enjoying this thread...

Partitions are good.

 

[Nothinman]

Partitions are good.

I prefer logical volumes, but the idea is the same.

 

[n0cmonkey]

I prefer logical volumes, but the idea is the same.

Just a different type of "partition." 😉