-1

I have 200,000 XML files in the folder on a RHEL7 Linux server. I need to zip all 200,000 XML files. I am using the Tar command but getting the error "Argument list too long" is there any other way in this 200,000 need zip only 10,000

tar -cvf xml.tar *.xml

All xml files need to be archived separately as individual archive that has original file name in the archive name.

Original files:

1.xml
2.xml
...
n.xml

Result of archiving OP wants:

1.xml.tgz
2.xml.tgz
...
n.xml.tgz
  • While I edited in the command verbatim from a now-deleted comment by OP, apparently that's not the case. OP's comments from a now-deleted tar answer: "its working but all the files ziped and genarated into single folder i need zip each and every file : Example : filename .xml its convert in to filename1 .xml .zip" and "sorry i need to ".zip"". – Bob Aug 10 '18 at 15:44
  • 2
    kesavanagaprasadthonta, please clarify: do you mean you ran the tar command, but you don't actually want a tar file? And do you want individual zips or gzips? They are different formats, and require different commands. – Bob Aug 10 '18 at 15:46
  • @Bob shell simply choke on a huge list of files in command line when OP used *.xml – Alex Aug 10 '18 at 16:33
  • 1
    IMHO making a big single TAR you are going to reproduce the problem somewhere else. Directories with over 10K files are always a problem. It could be much more practical to make TARs of a few thousands files each. – xenoid Aug 10 '18 at 16:55
  • @Bob Maybe you shouldn't have deleted your answer yet. Now we have a new answer about tar that is IMHO not as good as yours. – Kamil Maciorowski Aug 10 '18 at 16:58
1

You exceeded maximum command line length. Command line has finite length that you can test with getconf ARG_MAX command. When you running shell command that includes glob pattern such as * in directory that contain huge amount of files then command line is overfilled and one will receive error message "Argument list too long", so it isn't a tar problem. Keep this it in mind when you use other commands with glob patterns that applied to huge amount files.

To resolve your issue, you can use find program that will "walk"
through the directory and feed tar.

To archive all files as a single compressed tar archive you can use:

find . -name "*.xml" -print | tar -czvf xml.tgz -T -

To archive all files individually as compressed tar archives(not really sure why need to by tar'ed if it's a single file, but as you wish), use

find . -name "*.xml" -exec tar -czvf '{}'.tgz '{}' \;

To archive all files individually as gzip archives, use:

find . -name "*.xml" -exec gzip '{}' \;

Be warned, command above will remove original files (!!!)

To archive all files individually as zip archives, use:

find . -name "*.xml" -exec zip '{}'.zip '{}' \;

P.S. I added also missed(?) option to compress tar archive.

  • find . -name ' *.xml' | tar -T - would work just as well? – xenoid Aug 10 '18 at 16:53
  • @xenoid Doesn't looks like it would work – Alex Aug 10 '18 at 16:55
  • @Alex Very nice, I didn't know tar can also use a list from pipe. I have been using things like find . -name "*.xml" -print | zip xml.zip -@ for zip archive – Bernard Wei Aug 10 '18 at 18:47
  • @BernardWei I glad you found it useful. You probably might want to post your solution here too, since it not clear if OP wants a zip or tar archive. – Alex Aug 10 '18 at 18:53
  • 1
    @kesavanagaprasadthonta make up your mind! It seems from the above comment you want *.xml -> *.xml.zip and now you say *.xml.gz. And what is it about *.xml.tgz? Moreover zip and gzip (beside bzip2) are very different. There's no option to zip directly from tar. If you want separate files why don't say it clearly? Read How to Ask or you'll just get more downvotes – phuclv Aug 11 '18 at 10:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.