linux - append but skip existing data

Question

Being somewhat new to linux, I am trying to copy but append existing information from multiple updated lists under different directories, to the existing files that may have some matching information, but without duplication.

/dir1/file1
/dir1/file2
/dir2/file1
/dir2/file2

Then within the files for example:

/dir1/file1:

aaa
bbb
ccc

/dir2/file1:

aaa
bbb
ccc
ddd
eee

So I know normally I could cat /dir1/file1 >> /dir2/file1, but then the result would look like this containing the duplicate lines:

aaa
bbb
ccc
ddd
eee
aaa
bbb
ccc

Versus the other files may be: /dir1/file2

mmm
nnn
ooo

/dir2/file2

mmm
nnn
zzz
yyy

I would want the final /dir2/file2 file to contain:

mmm
nnn
ooo
zzz
yyy

So how would I do this to avoid duplicate lines across multiple different files in different folders? We're talking 468 files in approximately 115 folders, where dir1 has the original older data, and dir2 has the newer data but may be missing some of the original/custom lines/entries. These are simply data files read by another program so it isn't code, but I still would prefer not to have several GB worth of extra unneeded data caused by the duplication. This is a server (CentOS 6) with cli only, and direct/full ssh root access.

Any help would be appreciated.

Answer 1

For Debian distros the package john includes unique, which works like an unsorted uniq:

apt install john

After installation, run:

cat /dir1/file2 /dir2/file2 | unique foo ; cat foo

Output:

mmm
nnn
ooo
zzz
yyy

Note that as *nix utils go the present version of unique (v1.8.0-2) is a bit crude, it can't output to STDOUT, or any existing file. Therefore a new temporary output filename foo is necessary.