Being somewhat new to linux, I am trying to copy but append existing information from multiple updated lists under different directories, to the existing files that may have some matching information, but without duplication.
/dir1/file1
/dir1/file2
/dir2/file1
/dir2/file2
Then within the files for example:
/dir1/file1:
aaa
bbb
ccc
/dir2/file1:
aaa
bbb
ccc
ddd
eee
So I know normally I could cat /dir1/file1 >> /dir2/file1, but then the result would look like this containing the duplicate lines:
aaa
bbb
ccc
ddd
eee
aaa
bbb
ccc
Versus the other files may be: /dir1/file2
mmm
nnn
ooo
/dir2/file2
mmm
nnn
zzz
yyy
I would want the final /dir2/file2 file to contain:
mmm
nnn
ooo
zzz
yyy
So how would I do this to avoid duplicate lines across multiple different files in different folders? We're talking 468 files in approximately 115 folders, where dir1 has the original older data, and dir2 has the newer data but may be missing some of the original/custom lines/entries. These are simply data files read by another program so it isn't code, but I still would prefer not to have several GB worth of extra unneeded data caused by the duplication. This is a server (CentOS 6) with cli only, and direct/full ssh root access.
Any help would be appreciated.
cat File1 File2|sort|uniq>File3; mv File3 File2
will do it. – AFH Apr 6 '16 at 17:51