How can you remove duplicates from bash history?

Question

Since I'm saving history from different sessions this is an issue where erasedups can't help because I'm using the following:

PROMPT_COMMAND="$PROMPT_COMMAND;history -a"

Is there an easy way to delete duplicates in history?

Answer 1

It is possible to remove duplicated lines which are already in .bash_history by running

nl ~/.bash_history | sort -k 2  -k 1,1nr| uniq -f 1 | sort -n | cut -f 2 > unduped_history

followed by

cp unduped_history ~/.bash_history

I would also recommend to put the following in your ~/.bashrc:

export HISTCONTROL=ignoreboth:erasedups

Answer 2

Many more ways to do this here as well as some other ongoing settings / options:

https://unix.stackexchange.com/questions/48713/how-can-i-remove-duplicates-in-my-bash-history-preserving-order

Also

sort ~/.bash_history | uniq -u

or in vim text editor :sort u

If some of the suggestions including the one above don't work immediately. After running the code I do:

history -c

to clear the history first then restore the no duplications version over it:

mv unduped_history ~/.bash_history

Answer 3

I've created a small Python script deduplicate.py for this. If a line is duplicated, the script only keeps the last duplicate. This works nicely when searching the command history with fzf using Ctrl+r since it shows later entries first:

#!/usr/bin/env python

# Deduplicates the lines of a file.
# Doesn't change the file, just writes the result
# to standard out.

import sys

if len(sys.argv) >= 2:
    unique_lines = []

    file_name = sys.argv[1]

    with open(file_name, 'r') as fi:
        for line in reversed(list(fi)):
            if line not in unique_lines:

                # If a command occurs multiple times in the file, we
                # keep the more recent one (the one closer to the end
                # of .bash_history).
                # This is useful when searching .bash_history with FZF
                # using ctrl+r, since per default, FZF reverses the
                # lines in .bash_history and we see more recent
                # commands first.
                unique_lines.insert(0, line)

    for unique_line in unique_lines:
        print(unique_line, end='')

else:
    print('Please provide an input file path', file=sys.stderr)

I call it hourly with this systemd service located at ~/.config/systemd/user/deduplicate_bash_history.service:

[Unit]
Description=Remove duplicate lines from ~/.bash_history.

[Service]
# We use -c to get a login shell for accessing the home directory
ExecStart=/bin/bash -c "~/scripts/deduplicate.py ~/.bash_history > ~/.bash_history_deduplicated && mv --force ~/.bash_history_deduplicated ~/.bash_history"
# We have to create a temporary file, since the following direct method would create a .bash_history containing only the entry of this command:
#/bin/bash -c "~/scripts/deduplicate.py ~/.bash_history > ~/.bash_history"

and this timer at ~/.config/systemd/user/deduplicate_bash_history.timer:

[Unit]
Description=Remove duplicate lines from ~/.bash_history.

[Timer]
OnCalendar=hourly

[Install]
WantedBy=timers.target

I activate the timer with

systemctl --user daemon-reload && systemctl --user enable deduplicate_bash_history.timer

and make sure it's among the services using

systemctl --user list-timers --all