Linux: Handling Directories with Lots of Files

I have a project where we are anticipating a large number of files per directory and had concerns about how well ext3 would perform with 50,000 files in one directory. Actually, because of the nature of the project it would be 3 times that as the files get saved in 3 formats.

For some time, ext3 has support built-in for directory indexing. In some samples, this has allowed tests that took 38 minutes without the directory indexing option to be completed in 11 seconds (the example was for adding 100,000 files to a directory).

To turn this on for an existing ext3 file system, you just use the tune2fs program to turn on directory indexing on the file system. Backup first.

For instance:

 
see what options are in effect:
# dumpe2fs -fh /dev/hda
(the line "Filesystem features:" should show dir_index, if already enabled)

enable directory indexing:
# tune2fs -O dir_index /dev/hda

convert existing directories to directory indexing
(only run on a non-mounted file system (probably after
booting with a recovery CD), after a backup):
# e2fsck -Df /dev/hda

check to see if a directory is using dir_index:
# lsattr /var
(see if the 'I' flag is set)

According to some docs, we can now go to millions of files per dir with no performance hit.

The information I read indicated the performance hit can be experienced on ext3 file systems because by default, they use either a linked list or block method of storage for the directory. This means a doing a file scan thru the directory instead of an indexed lookup and this means a huge performance hit.

More about these utilities can be found here.