I bet all of the time is spent comparing the files' creation dates: though not really proportional to N^2 (it really goes like N log N for smart sorting algorithms), this is still quite long.
The following command
find . -type f -printf '%T+ %p\n' | sort -r | head -1
finds all files in the current directory, lists them from newest to oldest, then prints only the first item.
It can be conveniently modified for instance as follows:
find . -type f -mtime -1 -printf '%T+ %p\n' | sort -r | head -1
This commands finds only files created in the last day (-mtime -1), and sorts only these.
Hopefully, this should be enough to restrict the search to a small number of items.
if you need a finer control, you can use:
find . -type f -mmin -5 -printf '%T+ %p\n' | sort -r | head -1
which, for instance, selects only files created/modified less than 5 minutes ago (-mmin -5).
This way, the ordering is restricted to a manageable number of objects. If necessary, the command can be conveniently aliased, or scripted, for typing convenience.