Conversation
monsieuricon

K. Ryabitsev-Prime 🍁

Got git bundle clones working reliably. This is a feature in newer git versions that allows offloading downloads to a CDN. But:

a) it's not currently very useful without redoing how we generate bundles -- we first made it work for Android's "repo" tool, which is very blunt in how it works.
b) it must be enabled in the client (set transfer.bundleURI=true) to even trigger -- it's not on by default.

I'll play with bundle generation next, especially for stable.git -- maybe we can prebuild bundles for specific branches.
3
1
7
Anarcat@kolektiva.social

anarcat

Reply to @monsieuricon

@monsieuricon i was *just* about to ask you why shallow clones are garbage from your last blog, i guess that bundles missing might not help with that for sure. and god i wish that client would flip, it's a huge deal for large monorepos (e.g. our firefox clone...)

0
0
0
Spider@mastodon.social

Spider

Reply to @monsieuricon

@monsieuricon id love this. We do weekly ish builds of kernel and to keep size down I put bundles into a docker image in our local cache (only docker images get cached, thus…) but it’s still pretty rough to get up to a current tag.

So the bundle clones would help me at least to not hammer the bandwidth each attempt.

1
0
0
rpardini@social.linux.pizza

Ricardo Pardini

Reply to @monsieuricon

@monsieuricon This is what we've been doing for Armbian build system: https://github.com/armbian/shallow

This takes the kernel.org bundles (torvalds & stable), updates them from live trees, and produces .git tarballs in OCI (ORAS) artifacts backed by GH's CDN, ready to go, in a few variants (full tree, shallow-per-version, which includes all -rc tags and stable point releases).

eg to get a shallow 6.18.y tree: get oras `apt install oras`, then `oras pull ghcr.io/armbian/shallow/kernel-git-shallow-6.18:latest`, untar, and you get a `.git`.

Compared to shallow cloning from git, it reduces server load drastically, but also by delivering a .git (instead of cloning) it saves client cycles and time.

build/armbian uses a few heuristics (disk free space/speed) to auto-determine for the (casual) user if a full or shallow tree is better suited for their specific build...

1
0
0
monsieuricon

K. Ryabitsev-Prime 🍁

Reply to @rpardini@social.linux.pizza
@rpardini shipping tarballs of a git tree is perfectly reasonable for infrastructure you trust. blobcatthumbsup

I've considered publishing something like this for others as well, but I wouldn't honestly recommend grabbing a tarball of a git tree from somewhere and running commands inside it (git implicitly trusts the hooks) -- so the potential for abuse was too great.
0
0
0
monsieuricon

K. Ryabitsev-Prime 🍁

Reply to @Spider@mastodon.social
@Spider the bundles are fully set up now, so if you set transfer.bundleURI=true, then you will immediately benefit from them, plus any local CDN caching.
0
0
1
About social.kernel.org

Terms of service

  1. Please do not use this service in violation of the Linux Kernel Code of Conduct. Doing so will result in your account suspension with the referral of the matter to the CoC committee.

  2. "Repeating"/"boosting" someone else's status on this platform will be treated as endorsement and will fall under rule #1.

  3. You are encouraged to use this platform to promote your work on the Linux Kernel, but there is no restriction on permitted topics (with the exception of anything covered by #1 above).

  4. There is no requirement to post in English, but it should be considered the primary language of communication on this platform.

Privacy notice

The admins of this service have access to all posted statuses. They aren't looking, but if it's something they shouldn't know about, then you should not post it on this platform.

Please see the Linux Foundation Privacy Policy, which applies to this platform as well.

Getting your own account

If you would like an account on this instance, please check that the following applies to you:

  • You are listed in MAINTAINERS or CREDITS

  • OR: You have a kernel.org account or email address

  • OR: You have a long and established history of involvement with the Linux Kernel

If the above is true and you agree with the Terms of Service and Privacy Notice listed above, please use these instructions to request an account: