I have a txt file that can contain up to 13.000.000 randomly generated codes seperated with "\n" by the fputcsv function in php.
What is the most effective way of removing duplicates from this file?
I think the problem here is that Excel has a limit of 1,048,576 rows and your dataset has 13 million entries. Each entry should be on a separate line.
I would use the regular expression option. Below is a brief summary from the SO answer.
^(.*?)$\s+?^(?=.*^\1$)into the search box
If this data is an export from a database you might want to consider amending your SQL statement prior to the export by adding
DISTINCT after the
SELECT. See here for an example.