git: “pack has bad object” when pushing to remote

Question

I have a bare git repository on my Raspberry Pi which only I use. When pushing to it today, I got this error message:

Counting objects: 460, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (367/367), done.
remote: fatal: pack has bad object at offset 1641: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

As you can see, the repository is accessed via ssh. (I changed the IP address.)

I tried several times but got the same error (even with the same numbers). Then I decided to create a new repository by deleting the folder of the old one, creating a folder with the same name, and executing git init --bare inside it.

Now I get this error when pushing to it:

Counting objects: 3129, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2265/2265), done.
remote: fatal: pack has bad object at offset 426983445: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

What is the problem and how do I get it to work again?

I run git version 1.9.1 on a kernel 3.19 64 bit Linux machine.

---

Updating with extra output:

laptop-14-04:~/Documents Container$ GIT_TRACE=1 git push --porcelain --progress --recurse-submodules=check origin refs/heads/master:refs/heads/master
trace: built-in: git 'push' '--porcelain' '--progress' '--recurse-submodules=check' 'origin' 'refs/heads/master:refs/heads/master'
trace: run_command: 'ssh' 'christoph@111.111.111.111' 'git-receive-pack '\''/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'\'''
christoph@111.111.111.111's password: 
trace: run_command: 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
trace: exec: 'git' 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
trace: built-in: git 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
Counting objects: 3383, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2257/2257), done.
remote: fatal: pack has bad object at offset 426983770: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

laptop-14-04:~/Documents Container$ git count-objects -Hv
count: 0
size: 0 bytes
in-pack: 3452
packs: 1
size-pack: 13.03 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

laptop-14-04:~/Documents Container$ git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (3452/3452), done.
laptop-14-04:~/Documents Container$ git gc
Counting objects: 3452, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2254/2254), done.
Writing objects: 100% (3452/3452), done.
Total 3452 (delta 915), reused 3452 (delta 915)

laptop-14-04:~/Documents Container$ git push --porcelain --progress --recurse-submodules=check origin refs/heads/master:refs/heads/master
christoph@111.111.111.111's password: 
Counting objects: 3383, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2257/2257), done.
remote: fatal: pack has bad object at offset 426983770: inflate returned -5
error: pack-objects died of signal 13
error: failed to push some refs to 'ssh://christoph@111.111.111.111/media/christoph/afacc396-ec79-4920-9105-513ca4616c06/git/Documents'

---

I now installed a new OS (Ubuntu 16.04 64 bit running the 4.4.0-21-generic kernel) and git is now version 2.7.4. I didn't copy the old repo to the new system but instead only copied the contents of it and created a new repo. Furthermore I deleted the repo on my Raspberry Pi and created a new bare repo on it. I now used SmartGit for adding the files to the repo and for trying to push it. However, the problem still exists:

---

It works if I create a bare repository locally, add it as the remote, and then push to it. I can then transfer the repository directory to the Raspberry Pi and use it as a remote via ssh. So it seems like the problem only occurs when pushing a lot of data (or possibly big commits) over a network.

Answer 1

The pack-objects (man git-pack-objects) died of signal 13 (broken pipe), because git was unable to inflate (uncompress) the object and it failed with error (error code -5 could mean out-of-mem or overwrite/overlap error).

Explanation

According to zlib manual, the errors are defined as follow:

#define Z_OK            0
#define Z_STREAM_END    1
#define Z_NEED_DICT     2
#define Z_ERRNO        (-1)
#define Z_STREAM_ERROR (-2)
#define Z_DATA_ERROR   (-3)
#define Z_MEM_ERROR    (-4)
#define Z_BUF_ERROR    (-5)
#define Z_VERSION_ERROR (-6)

where -5 means no progress is possible or if there was not enough room in the output buffer.

Z_BUF_ERROR if no progress is possible or if there was not enough room in the output buffer when Z_FINISH is used. Note that Z_BUF_ERROR is not fatal, and inflate() can be called again with more input and more output space to continue decompressing.

Here is what we can read in zlib FAQ:

Before making the call, make sure that avail_in and avail_out are not zero. When setting the parameter flush equal to Z_FINISH, also make sure that avail_out is big enough to allow processing all pending input. Note that a Z_BUF_ERROR is not fatal--another call to deflate() or inflate() can be made with more input or output space. A Z_BUF_ERROR may in fact be unavoidable depending on how the functions are used, since it is not possible to tell whether or not there is more output pending when strm.avail_out returns with zero. See zlib Usage Example for a heavily annotated example.

---

Solution

This may be related to few things:

• the object being pushed is too big so zlib is of memory, so you need more space in the output zlib buffer,

  In this case, try increasing http.postBuffer, e.g.

  git config http.postBuffer 134217728 # =~ 128MB
  

  Alternatively use bfg to strip larger blobs, e.g.

  java -jar bfg.jar --strip-blobs-bigger-than 100M some-big-repo.git
  

• your object is corrupted, so run git fsck --full and git gc

  Potential reason could be corrupted memory or storage device, so retry on clean repository or different computer.

• could be a git bug, since it shouldn't abort on Z_BUF_ERROR, but to provide more output space or more input, see: zLib inflate() hangs while uncompressing buffer

  You may report a git bug report to the mailing list.

• could be a gzip inflate issue it-self (e.g. Is this a bug in this gzip inflate method?)

• could be an old kernel bug (<= 2.6.32-rc4), so upgrade your kernel

  See: Bug#547503: git-core: "git clone" fails on armel

  The only possibly relevant kernel change I could find was commit 5a3a29f (ARM: 5691/1: fix cache aliasing issues between kmap() and kmap_atomic() with highmem, commit 7929eb9 upstream) from 2.6.31.1. So though I also have my doubts, we could be lucky.^msg00049

---

Other useful commands to consider:

• GIT_TRACE=1 git push origin
• git count-objects -Hv to check for excess of size limits

---

See also:

• Z_BUF_ERROR -5 while trying to decompress zlib data at SO
• zlib inflate returning a buffer error at SO

---

Here is failing relevant Git code (builtin/index-pack.c):

git_inflate_init(&stream);
stream.next_out = buf;
stream.avail_out = buf == fixed_buf ? sizeof(fixed_buf) : size;

do {
    unsigned char *last_out = stream.next_out;
    stream.next_in = fill(1);
    stream.avail_in = input_len;
    status = git_inflate(&stream, 0);
    use(input_len - stream.avail_in);
    if (sha1)
        git_SHA1_Update(&c, last_out, stream.next_out - last_out);
    if (buf == fixed_buf) {
        stream.next_out = buf;
        stream.avail_out = sizeof(fixed_buf);
    }
} while (status == Z_OK);
if (stream.total_out != size || status != Z_STREAM_END)
    bad_object(offset, _("inflate returned %d"), status);
git_inflate_end(&stream);

and git_inflate() from zlib.c

status = inflate(&strm->z,
         (strm->z.avail_in != strm->avail_in)
         ? 0 : flush);