Being somewhat new to linux, I am trying to copy but append existing information from multiple updated lists under different directories, to the existing files that may have some matching information, but without duplication.
/dir1/file1 /dir1/file2 /dir2/file1 /dir2/file2
Then within the files for example:
aaa bbb ccc
aaa bbb ccc ddd eee
So I know normally I could cat /dir1/file1 >> /dir2/file1, but then the result would look like this containing the duplicate lines:
aaa bbb ccc ddd eee aaa bbb ccc
Versus the other files may be: /dir1/file2
mmm nnn ooo
mmm nnn zzz yyy
I would want the final /dir2/file2 file to contain:
mmm nnn ooo zzz yyy
So how would I do this to avoid duplicate lines across multiple different files in different folders? We're talking 468 files in approximately 115 folders, where dir1 has the original older data, and dir2 has the newer data but may be missing some of the original/custom lines/entries. These are simply data files read by another program so it isn't code, but I still would prefer not to have several GB worth of extra unneeded data caused by the duplication. This is a server (CentOS 6) with cli only, and direct/full ssh root access.
Any help would be appreciated.