I have the following files:

-rw-r--r--  1 xyz xyz    122 May 01 01:00  Test1
-rw-r--r--  1 xyz xyz    122 May 01 01:00  Test2
-rw-r--r--  1 xyz xyz    122 May 05 05:00  Test3
-rw-r--r--  1 xyz xyz    122 May 05 05:00  Test4

The contents of the files contains a description row, followed by data, separated by commas. E.g.:


ID, Name, Job, Address
1111, John, Janitor, 1234 Corson Ave


ID, Name, Job, Address
2222, Tim, Cashier, 3245 Elliot St

Can someone please suggest a command that would concatenate the files above based on modified date while ignoring the first line in the file, which is the description (ID, Name, Job, Address).

E.g. This is what I would like after the script:


1111, John, Janitor, 1234 Corson Ave
2222, Tim, Cashier, 3245 Elliot St

Same thing for May5_file_after_concatenate. The directory I have will contain many millions of files. I will need for it to automatically sort out base on modified date and concatenate them accordingly.

I am on Centos 7.

  • By "merge", do you mean "concatenate", or is it possible to have two lines coming from different files with the same ID that would need a special treatment? – choroba Sep 20 '16 at 16:16
  • yes. I meant concatenate. There does not need to be special treatment of same ID's as that is fine if there is more than one. Thanks! – user3567212 Sep 20 '16 at 16:21

In bash, you can use the following script:

#! /bin/bash
for file in "$@" ; do
    date=$(stat -c%y "$file" | cut -f1 -d' ')
    tail -n+2 "$file" >> "$date"

It iterates over the given files, retrieves the last modification date from it, and then adds everything starting from line 2 to a file named by the date.

If there are two many files to fit on a command line or into parameters, call the script on smaller groups of files sequentially (with xargs or manually).

Converting the date format should be easy :-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.