Since I'm saving history from different sessions this is an issue where erasedups can't help because I'm using the following:


Is there an easy way to delete duplicates in history?

3 Answers 3


It is possible to remove duplicated lines which are already in .bash_history by running

nl ~/.bash_history | sort -k 2  -k 1,1nr| uniq -f 1 | sort -n | cut -f 2 > unduped_history

followed by

cp unduped_history ~/.bash_history

I would also recommend to put the following in your ~/.bashrc:

export HISTCONTROL=ignoreboth:erasedups
  • 1
    Why ignoreboth? That activates both ignoredups and ignorespace and the latter does not seem relevant here. Also, your command will simply empty .bash_history, you want cp unduped_history ~/.bash_history or cat unduped_history > ~/.bash_history.
    – terdon
    Feb 27, 2014 at 17:31
  • I agree that ignorespace could be omitted. I missed to type cp by accident. I added the the extra option -k 1,1nr to the first sort to take the last occurence of a duplicated commands. Feb 27, 2014 at 17:49
  • Great, now I can upvote that really cool nl trick with a clear conscience :)
    – terdon
    Feb 27, 2014 at 17:50
  • lot's of magic options, what do they do? Will this work if timestamps are added?
    – CervEd
    May 7, 2021 at 13:33

Many more ways to do this here as well as some other ongoing settings / options:



sort ~/.bash_history | uniq -u

or in vim text editor :sort u

If some of the suggestions including the one above don't work immediately. After running the code I do:

history -c

to clear the history first then restore the no duplications version over it:

mv unduped_history ~/.bash_history

I've created a small Python script deduplicate.py for this. If a line is duplicated, the script only keeps the last duplicate. This works nicely when searching the command history with fzf using Ctrl+r since it shows later entries first:

#!/usr/bin/env python

# Deduplicates the lines of a file.
# Doesn't change the file, just writes the result
# to standard out.

import sys

if len(sys.argv) >= 2:
    unique_lines = []

    file_name = sys.argv[1]

    with open(file_name, 'r') as fi:
        for line in reversed(list(fi)):
            if line not in unique_lines:

                # If a command occurs multiple times in the file, we
                # keep the more recent one (the one closer to the end
                # of .bash_history).
                # This is useful when searching .bash_history with FZF
                # using ctrl+r, since per default, FZF reverses the
                # lines in .bash_history and we see more recent
                # commands first.
                unique_lines.insert(0, line)

    for unique_line in unique_lines:
        print(unique_line, end='')

    print('Please provide an input file path', file=sys.stderr)

I call it hourly with this systemd service located at ~/.config/systemd/user/deduplicate_bash_history.service:

Description=Remove duplicate lines from ~/.bash_history.

# We use -c to get a login shell for accessing the home directory
ExecStart=/bin/bash -c "~/scripts/deduplicate.py ~/.bash_history > ~/.bash_history_deduplicated && mv --force ~/.bash_history_deduplicated ~/.bash_history"
# We have to create a temporary file, since the following direct method would create a .bash_history containing only the entry of this command:
#/bin/bash -c "~/scripts/deduplicate.py ~/.bash_history > ~/.bash_history"

and this timer at ~/.config/systemd/user/deduplicate_bash_history.timer:

Description=Remove duplicate lines from ~/.bash_history.



I activate the timer with

systemctl --user daemon-reload && systemctl --user enable deduplicate_bash_history.timer

and make sure it's among the services using

systemctl --user list-timers --all

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .