36

I'm going to transfer a private Git repository from win32 box to Ubuntu. Though I can do a final dos2unix commit, but I'd like to rewrite the whole history, so some Git GUI will display log/diff correctly. E.g., gitg will insert empty lines for each CR/LF.

4 Answers 4

30

You can use git filter-branch for that, with the --tree-filter option, and specifying --all for the branch.

Here's an example (started in an empty directory with a Unix-type text file:

Preparation:

$ hexdump -C testfile 
00000000  61 0d 0a 62 0d 0a 63 0d  0a                       |a..b..c..|
00000009

$ git init
Initialized empty Git repository in /home/seigneur/tmp/a/.git/

$ git add testfile && git commit -m "dos file checked in"
[master (root-commit) df4970f] dos file checked in
 1 files changed, 3 insertions(+), 0 deletions(-)
 create mode 100644 testfile

The command:

$ git filter-branch --tree-filter 'git ls-files -z | xargs -0 dos2unix' -- --all

Output:

Rewrite df4970f63e3196216d5986463f239e51eebb4014 (1/1)dos2unix: converting file testfile to Unix format ...

Ref 'refs/heads/master' was rewritten

$ hexdump -C testfile 
00000000  61 0a 62 0a 63 0a                                 |a.b.c.|
00000006

I strongly recommend doing a full backup beforehand. Running that from your Linux machine (unless you've got a good shell set up in your windows environment) is probably easier.

Edit: had the conversion reversed the first time around.

4
  • 1
    Thank you, this post helped me a lot. I had a few files with spaces in their name, a little change to the original command fixed it: git filter-branch --tree-filter 'git ls-files -z | xargs -0 dos2unix' -- --all . Flags -z and -0 tell git ls-files and xargs to print and interpret null as end of line.
    – Ivan
    Nov 8, 2013 at 14:04
  • 1
    Another alternative to the dos2unix command is to rely on the git itself: git filter-branch --prune-empty --tree-filter 'git add --renormalize .' -- --all Dec 10, 2018 at 6:45
  • git filter-branch is deprecated
    – CervEd
    May 3, 2021 at 11:40
  • I would recommend using git ls-files --eol | grep i/crlf | sed 's/.*\t//' | tr '\n' '\0' | xargs -0 .. to only convert files that a) are checked into the index with CRLF b) not binary c) not document files which may have been configured in .gitattributes
    – CervEd
    May 3, 2021 at 11:48
6

Mat's answer has nailed the issue right on the head. Unfortunately on Ubuntu Linux, starting with version 10.04 (Lucid Lynx), the dos2unix/unix2dos commands are no longer available, and have been replaced by fromdos/todos. Furthermore, both of the sets of the conversion commands have various degree of ignorance to the existence of binary files, thus if your repository contains images, fonts, etc. they are going to be corrupted by this process.

I was able to find a workaround for the binary file corruption issue that uses Linux 'file' command to correctly identify and process only text files as shown below. The command below uses --tag-name-filter option to preserve the existing tags by moving them to the newly amended commits. Also it uses --force flag to ensure that the command will work in the event you have run tree-filter on your repository before.

git filter-branch --force --tree-filter 'git ls-files | xargs file | sed -n -e "/.*: .*text.*/s/\(.*\): .*/\1/p" | xargs fromdos' --tag-name-filter cat -- --all
3

And without any additional tools (like 'fromdos', 'dos2unix', etc.):

git filter-branch --force --tree-filter 'git ls-files | xargs file | sed -n -e "/.*: .*text.*/s/\(.*\): .*/\1/p" | xargs -0 sed -i"" -e "s/"$(printf "\015")"$//"' --tag-name-filter cat -- --all

Crossplatform (OS X, FreeBSD, Linux) useful analog 'fromdos', 'dos2unix':

sed -i'' -e 's/'"$(printf '\015')"'$//'

Perhaps useful 'unix2dos':

sed -i '' -e 's|$|'"`printf '\015'`"'|' file.name

If are you absolytely shure what are you doing, you can use this simple inline command for delete "/r" from all files in current directory ".":

find . -type f -exec sed -i'' -e 's/'"$(printf '\015')"'$//' {} \;
4
  • 1
    Rather change \r\n to \n instead of removing \r only
    – xdevs23
    Sep 7, 2016 at 8:35
  • I think that corresponding sed invocation can be replaced with shorter one: sed -n -e "s/\(.*\): .*text.*/\1/p"
    – dma_k
    Nov 15, 2018 at 14:38
  • use git ls-files --eol to filter files that a) aren't binary b) already correct
    – CervEd
    May 3, 2021 at 11:42
  • also, use the -z flag and xargs -0 to avoid problems with spaces
    – CervEd
    May 3, 2021 at 11:44
1

git filter-branch has several issues, mainly in terms of performance as is deprecated in favor of git filter-repo, a separate project which consists of several python scripts.

To convert all line-endings in a repository, use lint-history

lint-history dos2unix # CRLF => LF
lint-history unix2dos # LF => CRLF

You can also do other linting task, like add a newline at the end of the file

lint-history sed -i '$a\'

This assumes the lint-history script is in your path and that git filter-repo is installed properly with symlinks and stuff.

If you don't have a full filter-repo installation with lint-history but only git-filter-repo you can run:

git filter-repo --blob-callback '
  if not b"\0" in blob.data[0:8192]:
    filename = ".git/info/tmpfile"
    with open(filename, "wb") as f:
      f.write(blob.data)
    subprocess.check_call(["sed", "-i", "$a\\", filename])
    with open(filename, "rb") as f:
      blob.data = f.read()
    os.remove(filename)
  '

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .