Going down the rabbit hole of Git's new bundle-URI

[jakub_g]

This is super interesting, as I maintain a 1M commits / 10GB size repo at work, and I'm researching ways to have it cloned by the users faster. Basically for now I do a very similar thing manually, storing a "seed" repo in S3 and having a custom script to fetch from S3 instead of doing `git clone`. (It's faster than cloning from GitHub, as apart from not having to enumerate millions of objects, S3 doesn't throttle the download, while GH seem to throttle at 16MiB/s.)

Semi-related: I always wondered but never got time to dig into what exactly are the contents of the exchange between server and client; I sometimes notice that when creating a new branch off main (still talking the 1M commits repo), with just one new tiny commit, the amount of data the client sends is way bigger than I expected (tens of MBs). I always assumed the client somehow established with the server that it has a certain sha, and only uploads missing commit, but it seems it's not exactly the case when creating a new branch.

[maccard]

Funny you say this. At my last job I managed a 1.5TB perforce depot with hundreds of thousands of files and had the problem of “how can we speed up CI”. We were on AWS, so I synced the repo, created an ebs snapshot and used that to make a volume, with the intention of reusing it (as we could shove build intermediates in there too.

It was faster to just sync the workspace over the internet than it was to create the volume from the snapshot, and a clean build was quicker from the just sync’ed workspace than the snapshotted one, presumably to do with however EBS volumes work internally.

We just moved our build machines to the same VPC as the server and our download speeds were no longer an issue.

[captn3m0]

The linux kernel does the same thing, and publishes bundle files over CDN[0] for CI systems using a script called linux-bundle-clone[1]

[0]: https://www.kernel.org/best-way-to-do-linux-clones-for-your-...

[1]: https://web.git.kernel.org/pub/scm/linux/kernel/git/mricon/k...

[bastardoperator]

Have you looked into Scalar? It's built into MSFT git and designed to deal with repos that are much larger internally.

  microsoft/git is focused on addressing these performance woes and making the monorepo developer experience first-class. The Scalar CLI packages all of these recommendations into a simple set of commands.

https://github.com/microsoft/scalar

https://github.com/microsoft/git

[schacon]

To try this feature out, you could have the server advertise a bundle ref file made with `git bundle create [bundle-file] --branches` that is hosted on a server within your network - it _should_ make a pretty big difference in local clone times.

[yjftsjthsd-h]

I can't imagine you haven't looked at this, but I'm curious: Do shallow clones help at all, or if not what was the problem with them? I'm willing to believe that there are usecases that actually use 1M commits of history, but I'd be interested to hear what they are.

[schacon]

Yeah, it basically has to advertise everything it has, so if you have a lot of references, it can be a quite large exchange before anything is done.

[djfivyvusn]

Have you tried downloading the .zip archive of the repo? Or does that run into similar throttling?

[sunnybeetroot]

Why does a user need all 1M commits? Can they perform their work with only a few?

[ks2048]

How much bandwidth and time is wasted cloning the entire history of large projects when people only need single snapshot in a single branch?

According to SO, newer versions of git can do,

  git init
  git remote add origin <url>
  git fetch --depth 1 origin <sha1>
  git checkout FETCH_HEAD

[acheong08]

git clone --depth 1 works as well. If you're just cloning to build and not contributing it makes much more sense

[jes5199]

I have a vague recollection that GitHub is optimized for whole repo cloning and they were asking projects not to do shallow fetching automatically, for performance reasons

[bobbylarrybobby]

I believe there is a bit of a footgun here because if you don't git clone then you don't fetch all branches, just the default. Can be very confusing and annoying if you know a branch exists on remote but don't have it locally (the first time you hit it, at least).

[autarch]

> This has resulted in a contender for the world's smallest open source patch:

Hah, got you beat: https://github.com/eki3z/mise.el/pull/12/files

It's one ASCII character, so a one-byte patch. I don't think you can get smaller than that.

[wavemode]

A commit which does nothing more than change permissions of a file would probably beat that, from an information theory perspective.

You might say, "nay! the octal triple of a file's unix permissions requires 3+3+3 bits, which is 9, which is greater than the 8 bits of a single ascii character!"

But, actually, Git does not support any file permissions other than 644 and 755. So a change from one to the other could theoretically be represented in just one bit of information.

[retroflexzy]

There is a cursor rendering fix in xf86-video-radeonhd (or perhaps -radeon) that flips a single bit.

It took the group several years to narrow in on.

[timdorr]

Sure you can: https://github.com/timdorr/-/commit/9e5a571abd3fc4f8714e8c40...

[schacon]

Damn you!

I did find it a little funny that my patch was so small but my commit message was so long. Also, I haven't successfully landed it yet, I keep being too lazy to roll more versions.

[ZeWaka]

That's a line modification, so presumably you'd count just an insertion or just a deletion as 'smaller'.

[pR0Ps]

Same: https://github.com/ianare/exif-py/pull/75/files

I feel like a bit of a fraud because this was the PR that got me the "Mars 2020 Contributor" badge...

[geenat]

git needs built in handling of large binary files without a ton of hassle, it's all I ask. It'd make git universally applicable to all projects.

mercurial had it for ages.

svn had it for ages.

perforce had it for ages.

just keep the latest binary, or last x versions. Let us purge the rest easily.

[pjc50]

As someone who used and administered p4 for ages, I regard git as a regression in this regard. Making git a fully distributed system is really expensive for certain use cases. My current employer still uses p4 for large integrated-circuit workflow assets.

A previous workplace was trying to migrate from svn to git, when we realized that every previous official build had checked in the resulting binaries. A sane thing to do in svn, where the cost is only on the server, but would have resulted in a naive conversion costing 50Gb on every client.

[GrantMoyer]

Getting better: https://git-scm.com/docs/partial-clone

[Cthulhu_]

If git lfs fulfills this role, it could become pre-installed with new git installations.

[robertlagrant]

Nothing to do with the article, but I appreciate the slightly idiosyncratic GitButler YouTube videos that explain how bits of Git work.

[schacon]

If you want us to cover something on Bits and Booze, just let me know! :)

[andrewshadura]

Interestingly, Mercurial had solved the bundles more than ten years ago and back then they already worked better than Git's today

[capitainenemo]

Not the only mercurial feature where that's the case.. sad, I keep rooting for the project to implement mercurial frontend over a git db, but they seem to be limited by missing git features.

[dgfitz]

Someone once put together an llm backed list of things people on hn post about a lot, mine was about this “other” dvcs system.

It is superior, and it’s not even much of a comparison.

[theamk]

how did it solve them, and how are mercurial's bundles better than git's ones?

if I am reading the manpage right, the feature set seems pretty compatible. "hg bundle" looks pretty identical to "git bundle".. and "hg clone"'s "-R" option seems pretty similar to "git clone"'s "--reference".

[nine_k]

But branches were more problematic.

[mbac32768]

One consequence of git clone is that if you have mega repos, it kind of ejects everything else from your cache for no win.

You'd actually rather special case full clones and instruct the storage layer to avoid adding to the cache for the clone. But this isn't always possible to do.

Git bundles seem like a good way to improve the performance of other requests, since they punt off to a CDN and protect the cache.

[jedimastert]

This actually might solve a massive CI problem we've been having...will report back tomorrow