3

I have a bare git repository on my Raspberry Pi which only I use. When pushing to it today, I got this error message:

Counting objects: 460, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (367/367), done.
remote: fatal: pack has bad object at offset 1641: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

As you can see, the repository is accessed via ssh. (I changed the IP address.)

I tried several times but got the same error (even with the same numbers). Then I decided to create a new repository by deleting the folder of the old one, creating a folder with the same name, and executing git init --bare inside it.

Now I get this error when pushing to it:

Counting objects: 3129, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2265/2265), done.
remote: fatal: pack has bad object at offset 426983445: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

What is the problem and how do I get it to work again?

I run git version 1.9.1 on a kernel 3.19 64 bit Linux machine.


Updating with extra output:

laptop-14-04:~/Documents Container$ GIT_TRACE=1 git push --porcelain --progress --recurse-submodules=check origin refs/heads/master:refs/heads/master
trace: built-in: git 'push' '--porcelain' '--progress' '--recurse-submodules=check' 'origin' 'refs/heads/master:refs/heads/master'
trace: run_command: 'ssh' 'christoph@111.111.111.111' 'git-receive-pack '\''/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'\'''
christoph@111.111.111.111's password: 
trace: run_command: 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
trace: exec: 'git' 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
trace: built-in: git 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
Counting objects: 3383, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2257/2257), done.
remote: fatal: pack has bad object at offset 426983770: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

laptop-14-04:~/Documents Container$ git count-objects -Hv
count: 0
size: 0 bytes
in-pack: 3452
packs: 1
size-pack: 13.03 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

laptop-14-04:~/Documents Container$ git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (3452/3452), done.
laptop-14-04:~/Documents Container$ git gc
Counting objects: 3452, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2254/2254), done.
Writing objects: 100% (3452/3452), done.
Total 3452 (delta 915), reused 3452 (delta 915)

laptop-14-04:~/Documents Container$ git push --porcelain --progress --recurse-submodules=check origin refs/heads/master:refs/heads/master
christoph@111.111.111.111's password: 
Counting objects: 3383, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2257/2257), done.
remote: fatal: pack has bad object at offset 426983770: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

I now installed a new OS (Ubuntu 16.04 64 bit running the 4.4.0-21-generic kernel) and git is now version 2.7.4. I didn't copy the old repo to the new system but instead only copied the contents of it and created a new repo. Furthermore I deleted the repo on my Raspberry Pi and created a new bare repo on it. I now used SmartGit for adding the files to the repo and for trying to push it. However, the problem still exists:

Screenshot of SmartGit error dialog.


It works if I create a bare repository locally, add it as the remote, and then push to it. I can then transfer the repository directory to the Raspberry Pi and use it as a remote via ssh. So it seems like the problem only occurs when pushing a lot of data (or possibly big commits) over a network.

  • it looks like your local repository is broken ... – Jakuje Apr 19 '16 at 21:43
  • Can you run git fsck --full and git gc? Does it help? Also do fsck on your Raspberry storage device and perform some memory testing. Try running command following with GIT_TRACE=1. – kenorb Apr 19 '16 at 22:43
  • 1
    What's your git version? How big are your pack files (du -hs .git/objects/pack)? Are you using some kind of VPN or proxy? – kenorb Apr 19 '16 at 22:52
  • I ran the commands but didn't figure out where to place GIT_TRACE=1. – UTF-8 Apr 20 '16 at 8:43
2

The pack-objects (man git-pack-objects) died of signal 13 (broken pipe), because git was unable to inflate (uncompress) the object and it failed with error (error code -5 could mean out-of-mem or overwrite/overlap error).

Explanation

According to zlib manual, the errors are defined as follow:

#define Z_OK            0
#define Z_STREAM_END    1
#define Z_NEED_DICT     2
#define Z_ERRNO        (-1)
#define Z_STREAM_ERROR (-2)
#define Z_DATA_ERROR   (-3)
#define Z_MEM_ERROR    (-4)
#define Z_BUF_ERROR    (-5)
#define Z_VERSION_ERROR (-6)

where -5 means no progress is possible or if there was not enough room in the output buffer.

Z_BUF_ERROR if no progress is possible or if there was not enough room in the output buffer when Z_FINISH is used. Note that Z_BUF_ERROR is not fatal, and inflate() can be called again with more input and more output space to continue decompressing.

Here is what we can read in zlib FAQ:

Before making the call, make sure that avail_in and avail_out are not zero. When setting the parameter flush equal to Z_FINISH, also make sure that avail_out is big enough to allow processing all pending input. Note that a Z_BUF_ERROR is not fatal--another call to deflate() or inflate() can be made with more input or output space. A Z_BUF_ERROR may in fact be unavoidable depending on how the functions are used, since it is not possible to tell whether or not there is more output pending when strm.avail_out returns with zero. See zlib Usage Example for a heavily annotated example.


Solution

This may be related to few things:

  • the object being pushed is too big so zlib is of memory, so you need more space in the output zlib buffer,

    In this case, try increasing http.postBuffer, e.g.

    git config http.postBuffer 134217728 # =~ 128MB
    

    Alternatively use bfg to strip larger blobs, e.g.

    java -jar bfg.jar --strip-blobs-bigger-than 100M some-big-repo.git
    
  • your object is corrupted, so run git fsck --full and git gc

    Potential reason could be corrupted memory or storage device, so retry on clean repository or different computer.

  • could be a git bug, since it shouldn't abort on Z_BUF_ERROR, but to provide more output space or more input, see: zLib inflate() hangs while uncompressing buffer

    You may report a git bug report to the mailing list.

  • could be a gzip inflate issue it-self (e.g. Is this a bug in this gzip inflate method?)

  • could be an old kernel bug (<= 2.6.32-rc4), so upgrade your kernel

    See: Bug#547503: git-core: "git clone" fails on armel

    The only possibly relevant kernel change I could find was commit 5a3a29f (ARM: 5691/1: fix cache aliasing issues between kmap() and kmap_atomic() with highmem, commit 7929eb9 upstream) from 2.6.31.1. So though I also have my doubts, we could be lucky.msg00049


Other useful commands to consider:


See also:


Here is failing relevant Git code (builtin/index-pack.c):

git_inflate_init(&stream);
stream.next_out = buf;
stream.avail_out = buf == fixed_buf ? sizeof(fixed_buf) : size;

do {
    unsigned char *last_out = stream.next_out;
    stream.next_in = fill(1);
    stream.avail_in = input_len;
    status = git_inflate(&stream, 0);
    use(input_len - stream.avail_in);
    if (sha1)
        git_SHA1_Update(&c, last_out, stream.next_out - last_out);
    if (buf == fixed_buf) {
        stream.next_out = buf;
        stream.avail_out = sizeof(fixed_buf);
    }
} while (status == Z_OK);
if (stream.total_out != size || status != Z_STREAM_END)
    bad_object(offset, _("inflate returned %d"), status);
git_inflate_end(&stream);

and git_inflate() from zlib.c

status = inflate(&strm->z,
         (strm->z.avail_in != strm->avail_in)
         ? 0 : flush);
  • My kernel isn't that old. I have 16 GB of ram and according to gnome-system-monitor I don't go above 1.8 GiB when attempting to push. Regrading the too big object: I took the command from the question you linked which set it to 50 MB but added 2 zeros. I ran the mentioned commands. – UTF-8 Apr 20 '16 at 8:51
  • The output of the commands you added: pastebin.com/pr3ppsWF – UTF-8 Apr 20 '16 at 8:53
  • I now tried git config http.postBuffer 13421772800 (added 2 zeros to this) on the now system. Didn't work. (Same error.) The program involved in the alternative command seems to have big problems itself as it throws exceptions. This is the output. It also says something about a dirty file. What does it mean by that? – UTF-8 Apr 23 '16 at 22:19
  • I'm not familiar with bfg, but the exception says: No such file or directory. If the file exist, maybe it's a bug not parsing white spaces properly. Or this bug sounds similar (probably parent folder doesn't have write permissions?). – kenorb Apr 25 '16 at 14:02
  • 2
    I forgot to accept this answer which I caught up on, now. It was a bug, I reported it to the git mailing list, they were quick to fix it, the fix was merged into upstream, and the bug has been fixed in git's releases for several months, now. – UTF-8 Dec 1 '16 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.