Argument list too long

[Armitage]

I have a directory with on the order of 10K files in it that match a particular pattern (*.obs), and I need to do stuff like run an MD5SUM on everything, copy move or delete all the files of that pattern, etc. I can accomplish the task by breaking it up with a more specific pattern of course (1*.obs, 2*.obs, 3*.obs ... etc.), but it's a PITA

I don't even know exactly where the limit is - it's somewhere > 3800, but I haven't chased it down exactly.

So why does this limit exist?
Is this a limitation of BASH, or do all of the shells suffer from this problem?
I'm half tempted to go poking around the source to fix it :| In the past, I've written a wrapper script around the offending commands to handle it, but that's such a kludge.

 

[Armitage]

Googled up one solution I didn't know about - xargs
http://techrepublic.com.com/5100-6347-5108413.html#

Still a kludge though. Apparently fixing it involves a kernel hack.

 

[n0cmonkey]

I run into that all the time. It's a PITA. BTW, your link is asking for registration stuff or something.

 

[Armitage]

Originally posted by: n0cmonkey
I run into that all the time. It's a PITA. BTW, your link is asking for registration stuff or something.

Huh - that's odd. The link from google took me right to it In fact if I even reload the page I already have it redirects it to the registration page

Oh well

Linux administrators have traditionally used the xargs command to manage large argument lists, such as file listings. The xargs command also has many other uses.

But handling long file listings is a very useful function. For example, if you have a directory with a large number of files, you may have experienced the "too many arguments" problem associated with trying to list or delete files in a directory.

Let's say that you have a directory full of clip art and want to delete all of the JPEG files. You attempt to use the following:

rm *.jpg

You may find that rm can't handle the large number of files, and the command fails. You can still remove the files, but it requires the use of a few extra commands.

Use the following command to remove all of the JPEG files from the current directory and all subdirectories:

$ find . -type f -name '*.jpg' -print|xargs -l50 rm -f

This command uses the find command to locate all files in and below the current directory with the name "*.jpg" and print them to standard output. The command then pipes the output from find as input into xargs, which then executes rm -f on a maximum of 50 files at a time.

You can use the xargs utility in many more situations. It's a powerful tool that allows you to efficiently chain together commands that you otherwise couldn't connect, manage files efficiently,