Moving a file into a directory with over 7000 files

Question

I'm on a GoDaddy shared hosting Linux server ("unlimited" hosting).

I have a folder with product images for an e-commerce store that contains over 7000 images.

GoDaddy told me I'm not allowed to have over 1024 files in a single directory and that I need to reformat the folder structure, which isn't possible as this software is maintained and run by a Point-of-Sale system at my client's brick and mortar store.

I've SSH'd in and changed permissions on the folder I'm moving the image to and the image itself to 777 and issued the following command:

mv img.gif product_images/img.gif

and I get the following error:

mv: cannot move 'img.gif' to 'product_images/img.gif': File too large

The file is not "too large"; it is only 49 bytes (its a 1x1 gif)!

If I try drag+drop upload from Windows into the directory via FTP I get the following errors:

An error occurred copying a file to the FTP Server. Make sure you have permission to put files on the server.

Details:
200 TYPE is now 8-bit binary
227 Entering Passive Mode (184,168,167,92,197,60)
553-Can't open that file: File too large
553 Rename/move failure: no such file or directory

I DO have permission to put files on the server...its how I put the img.gif file there in the first place. The directory does exist. I can ls its contents just fine.

The previous support guy (who is not longer available) could do it - I just don't know how.

How can I go about moving files into this very stubborn directory? Any ideas?

Answer 1

There might be one trick you can try, if GoDaddy allows hardlinks:

ln img.gif product_images/img.gif

Granted, if the error message is simply misleading (and it's actually a "too many files" error, rather than an "out of disk space" error), this probably still won't work. But if this succeeds, you will have identical img.gif's both in your original directory and your subdirectory, but the file pointers will point to the same physical file, so no additional space will be used (aside from the pointer).

If all goes well, you can safely delete the original file, and the one in the subdirectory will remain:

rm img.gif

If all else fails, download all the files again (hint: tar+gz them to download them much faster), delete them from the website, reorganize them (more on that below), then reupload them.

Solving the root problem

One common solution is to create subdirectory structures for the first few characters in the filename. For example:

images/widget001.png --> images/w/i/widget001.png

You can then write a .htaccess RewriteRule so your web app doesn't break. It will still try to download the image from images/widget001.png, but the URL rewriting engine will serve up images/w/i/widget001.png instead.

You can create however many levels of subdirectories you need in order to get the number of files per directory down. This works especially well if the files have randomly-generated names. If all the names are similar (e.g., they all match the format "img#####.png"), you can create your directory structure based on some other part of the name, instead of using the first few letters.

NOTE: You should keep the full filename for easier future maintenance, rather than renaming it to images/w/i/dget001.png in the previous example. One reason is that it's much easier to flatten and rebuild the directory structure if you find out you need to deepen the directory structure again later.

EDIT: +1 to baraboom, who also mentioned the subdirectory+URL rewrite while I was writing my answer. :)

Answer 2

So, after googling godaddy 1024 file limit and reading a few results, it seems like your best option might be to find a new host.

Barring that, the least painful way I can think of to fix the problem would be to logically organize the files into subdirectories and then use Apache's rewrite rules to map the old paths to the new paths. This would save you changing anything in the actual application. The following would be one example:

1) Download the directory to your local computer;

2) Organize the files into sub-directories, it shouldn't be too bad with only 7000 files and the 1024 limit. For example, you could create 36 sub directories, one for each letter of the alphabet and one for each number, 0-9, and then place each subset of files into the appropriate subdirectory: file awhatever.gif would go into a/, file 11whatever would go into 1/, etc.

3) Create / edit the .htaccess file and create a rewrite rule that would map any request for product_images/awhatever.gif to product_images/a/awhatever.gif.

The .htaccess file would look something like this:

RewriteEngine On
RewriteRule ^(.)([^/]+)$ $1/$1$2 [L]

Answer 3

GoDaddy has directly told you that it doesn't allow what you want to do. The GoDaddy help pages tell you that what you want to do is not allowed. They even tell you that you'll get these sorts of error messages if you try to do what you are not allowed to do. And here you are, asking, "How, despite being told that I cannot by GoDaddy itself, do I do this?".

GoDaddy does not allow you to do this. Find someone else who does, or keep below the 1024 directory entries limit.

Answer 4

Get around the issue by creating a CSS sprite-sheet for the product images. You'll reduce your number of files exponentially.