Why does this not work?
ls *.txt | xargs cat > all.txt
(I want to join the contents of all text files into a single 'all.txt' file.) find with -exec should also work, but I would really like to understand the xargs syntax.
Thanks
4 Answers
ls *.txt | xargs cat >> all.txt
might work a bit better, since it would append to all.txt instead of creating it again after each file.
By the way, cat *.txt >all.txt would also work. :-)
answered Sep 28, 2010 at 11:08
-
7
-
1However, the ... | xargs cat >> all.txt or > all.txt always return error with xargs: unmatched single quote ... Is it because xargs takes everything after it as the command?– ajoSep 28, 2010 at 11:12
-
1Do you have filenames with spaces? If so, then use something like "find /your/path -iname '*.txt' -print0 | xargs -0 cat >>all.txt" instead Sep 28, 2010 at 11:17
-
1no, I replaced all the filename spaces with . But thinking of it, some filenames are likely to include single quotes as in listing_O'Connor.txt, this might be the problem!– ajoSep 28, 2010 at 11:29
-
Yes, that's the problem then. :) The easiest and the sanest way is to use find with -print0 combined with xargs -0 -- then the whole chain will use NULL character as a separator and whitespace and special characters will be taken care of automatically. Sep 28, 2010 at 11:37
If some of your file names contain ', " or space xargs will fail because of the separator problem
In general never run xargs without -0 as it will come back and bite you some day.
Consider using GNU Parallel instead:
ls *.txt | parallel cat > tmp/all.txt
or if you prefer:
ls *.txt | parallel cat >> tmp/all.txt
Learn more about GNU Parallel http://www.youtube.com/watch?v=OpaiGYxkSuQ
all.txt is a file in the same directory, so cat gets confused when it wants to write from the same file to the same file.
On the other hand:
ls *.txt | xargs cat > tmp/all.txt
This will read from textfiles in your current directory into the all.txt in a subdirectory (not included with *.txt).
-
Still the following error: xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option– ajoSep 28, 2010 at 11:29
-
1
You could also come across a command line length limitation. Part of the reason for using xargs is that it splits up the input into safe command-line-sized chunks. So, imagine a situation in which you have hundreds of thousands of .txt files in the directory. ls *.txt will fail. You would need to do
ls | grep .txt$ |xargs cat > /some/other/path/all.txt
.txt$ in this case is a regular expression matching everything that ends in .txt (so it's not exactly like *.txt, since if you have a file called atxt, then *.txt would not match it, but the regular expression would.)
The use of another path is because, as other answers have pointed out, all.txt is matched by the pattern *.txt so there would be a conflict between input and output.
Note that if you have any files with ' in their names (and this may be the cause of the unmatched single quote error), you would want to do
ls | grep --null .txt$ | xargs -0 cat > /some/other/path/all.txt
The --null option tells grep to use output separated by a \0 (aka null) character instead of the default newline, and the -0 option to `xargs tells it to expect its input in the same format. This would work even if you had file names with newlines in them.
lsfor this. If you really can't usecat *.txt >all.txtthen tryprintf '%s\0' *.txt | xargs -r0 cat >alland thenmv all all.txtto avoid having the file referencing itself.