Is git really that fast?

Question

I remember Linus said it can process 4gb of data in about a fraction of a second in his laptop.

I don't remember exactly the quote, it is in this video.

Is it really that fast?

I mean, if Linus is saying it it must be true, but if anyone of you is using git, does it show compared with others VCS?

EDIT

Here's the quote: ...Full diff of the kernel (22k files)[...]in, [...] just over a second...

Aaaand ... here ... The merge it self ( 22k+ file) takes less than half a second

Answer 1

Yes, git is indeed very fast. Here are some explanations given by Eric Sink

Why is Git so Fast?

In the DVCS world, Git has a reputation for being really fast. I am curious about how Git got this way.

When I started thinking about this question, seven different answers came to my mind. Some of those answers seem more interesting or correct than others.

One: Maybe Git is fast simply because it's a DVCS.

There's probably some truth here. One of the main benefits touted by the DVCS fanatics is the extra performance you get when everything is "local".

But this answer isn't enough. Maybe it explains why Git is faster than Subversion, but it doesn't explain why Git is so often described as being faster than the other DVCSs.

Two: Maybe Git is fast because Linus Torvalds is so smart.

This might very well be correct. But it's not interesting.

Fine. So Linus is smarter than all of us. But how did he use those smarts to make Git so fast? What are the details?

Three: Maybe Git is fast because it's written in C instead of one of those newfangled higher-level languages.

Nah, probably not. Lots of people have written fast software in C#, Java or Python.

And lots of people have written really slow software in traditional native languages like C/C++. Adobe writes most of their stuff in C++, and they don't have any trouble making sure that release N+1 is slower than release N.

Four: Maybe Git is fast because being fast is the primary goal for Git.

This is another one of those high-level answers that is probably correct but doesn't have the kind of details about which I am curious.

Still. Take some time to read through the archives of the Git developers mailing list. These people spend a LOT of time talking about performance issues.

Five: Maybe Git is fast because it does less.

One of my favorite recent blog entries is this piece which claims that the way to make code faster is to have it do less.

Predictably, people came out of the woodwork to say how wrong this guy was. That's what happens to almost any blog entry about performance tuning or optimization. Readers ignore anything correct in the article and quibble about little stuff.

But this guy was essentially correct. One way to make software faster is to make it do less.

For example, the way you get something in the Git index is you use the "git add" command. Git doesn't scan your working copy for changed files unless you explicitly tell it to. This can be a pretty big performance win for huge trees. Even when you use the "remember the timestamp" trick, detecting modified files in a really big tree can take a noticeable amount of time.

Or maybe Git's shortcut for handling renames is faster than doing them more correctly like Bazaar does.

Six: Maybe Git is fast because it doesn't use much external code.

Very often, when you are facing a decision to use somebody else's code or write it yourself, there is a performance tradeoff. Not always, but often. Maybe the third party code is just slower than the code you could write yourself if you had time to do it. Or maybe there is an impedance mismatch between the API of the external library and your own architecture.

This can happen even when the library is very high quality. For example, consider libcurl. This is a great library. Tons of people use it. But it does have one problem that will cause performance problems for some users: When using libcurl to fetch an object, it wants to own the buffer. In some situations, this can end up forcing you to use extra memcpys or temporary files. The reason all the low level calls like send() and recv() allow the caller to own the loop and the buffer is because this is the best way to avoid the need to make extra copies of the data on disk or in memory.

People make fun of those with NIH Syndrome, but my observation is that folks who suffer from this disorder tend to create faster software, even if they also tend to ship everything late. :-)

Maybe Git is fast because every time they faced one of these "buy vs. build" choices, they decided to just write it themselves.

Seven: Maybe Git isn't really that fast.

If there is one thing I've learned about version control it's that everybody's situation is different. It is quite likely that Git is a lot faster for some scenarios than it is for others.

How does Git handle really large trees? Git was designed primary to support the efforts of the Linux kernel developers. A lot of people think the Linux kernel is a large tree, but it's really not. Many enterprise configuration management repositories are FAR bigger than the Linux kernel.

Final thoughts

This week's version control blog entry raises more questions than answers. I'm not a Git user, nor have I looked much at its code, so I don't really know why it's so fast. I'm just curious. If you have better answers than mine (and I admit that's a low hurdle), feel free to send them to me or post them in my comments.

But FWIW, I have decided it is time for me to become a Git user. When I was writing about Git a few weeks ago, a lot of Git users kept telling me I just don't get it. I've spent more time thinking about version control implementation and design than most folks, so I tend to think I actually do "get it". But my curiosity is piqued, and I hate to pass up an opportunity to learn something, so I'm going to give it a try. I've got a small project here at SourceGear that I work on part-time with a couple other people. We've decided to switch to Git and see how it goes. I'll let you know what I find out.

Answer 2

I am happy with the speed of Git for the things I use it for. One of the advantages of a DVCS over a centralised solution (like Subversion) is that there is no network communication overhead for most operations (except fetch/push, naturally). Things running on your local machine are almost always going to be faster than having to ask another machine on the network to do the same thing for you.

Answer 3

The website Why Git is Better Than X gives some statistics on speed and other things as well, in comparison to a handful of other SCM systems.

Answer 4

I know that anecdotal evidence isn't worth much but, on my work repo svn status took 20 minutes vs git status taking 3 seconds. ~11 gig repo (I don't have it with me so I can't be more exact) with lots of bin files.

Answer 5

Example: Two weeks ago Linus Torvalds realized he could take Mozilla's SHA1 hashing code included in GIT, refactor and rewrite it in C so that it beat specialized ASM implementations and OpenSSL's own implementation.

Linus' sha1 is much faster!

Seemingly, as soon as Linus gets to work on doing something fast on exactly his computer and architecture (SHA1 was optimized for among other things his Nehalem processor), there is no one stopping him. Here he describes how he refactored the C implementation, watched the assembler that GCC produces and rewrites it until GCC generates what he wants on most platforms (in the end they tested Nehalem, Intel Atom, PowerPC and ARM).

End result is that next version of git will include Linus new SHA1 function in portable C, 0-30% faster, unnoticable to some (unless you fsck your repo a lot) but optimized to death.

Answer 6

I'm a system programmer of nearly 50 years experience. I program by sense of smell.

I am amazed at the speed that git has. the ability to swap between two huge trees of source code instantly is amazing.

But sometimes, just sometimes, a mysterious gap appears in the system's behavior. This happens when actual writes are being done in my work directory -- like untarring a library. The gap appears in the behavior of the shell after such a command.

And then it hit me: it's a disk cache flush at the system level. The documentation mentions that Git implements a new filesystem type. When I checkout files in my working directory, the i-nodes point to stuff that is totally under 'git-filesystem' rules.

So when i do a 'git checkout' and marvel at how fast the tree comes in, I am being fooled. Only the directory structure is there, and all marked as 'you gotta ask git-file-system' to actually do a read or write operation at the system level. git-fs does the actual system level read and gets the data from the info on it's simulated hard-drive: the archive in the .git subdirectory of the tree.

When I write any new info into one of the files in the directory tree, git-fs goes to work and keeps track of all the updates, all using the lowest level device/filesystem hooks --- That's also how git knows what has changed in the directory tree so quickly.

A fabulous use of filesystem technology, and who would better know the details of how to get use out of it than Linus.

I did a bit more research and found this reference for those who want to chase the details: http://www.seas.upenn.edu/~cse400/CSE400_2008_2009/websites/grant/final.pdf

Ciao!