Being somewhat new to linux, I am trying to copy but append existing information from multiple updated lists under different directories, to the existing files that may have some matching information, but without duplication.

/dir1/file1
/dir1/file2
/dir2/file1
/dir2/file2

Then within the files for example:

/dir1/file1:

aaa
bbb
ccc

/dir2/file1:

aaa
bbb
ccc
ddd
eee

So I know normally I could cat /dir1/file1 >> /dir2/file1, but then the result would look like this containing the duplicate lines:

aaa
bbb
ccc
ddd
eee
aaa
bbb
ccc

Versus the other files may be: /dir1/file2

mmm
nnn
ooo

/dir2/file2

mmm
nnn
zzz
yyy

I would want the final /dir2/file2 file to contain:

mmm
nnn
ooo
zzz
yyy

So how would I do this to avoid duplicate lines across multiple different files in different folders? We're talking 468 files in approximately 115 folders, where dir1 has the original older data, and dir2 has the newer data but may be missing some of the original/custom lines/entries. These are simply data files read by another program so it isn't code, but I still would prefer not to have several GB worth of extra unneeded data caused by the duplication. This is a server (CentOS 6) with cli only, and direct/full ssh root access.

Any help would be appreciated.

  • 1
    It's easy enough to skip common lines at the start of the file, but if you want to skip all duplicates you will need to specify exactly what ordering you want. If the ordering is immaterial, then cat File1 File2|sort|uniq>File3; mv File3 File2 will do it. – AFH Apr 6 '16 at 17:51

For Debian distros the package john includes unique, which works like an unsorted uniq:

apt install john

After installation, run:

cat /dir1/file2 /dir2/file2 | unique foo ; cat foo

Output:

mmm
nnn
ooo
zzz
yyy

Note that as *nix utils go the present version of unique (v1.8.0-2) is a bit crude, it can't output to STDOUT, or any existing file. Therefore a new temporary output filename foo is necessary.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.