Conversation
K. Ryabitsev 🍁
monsieuricon
Edited 7 days ago
PSA: please don't use git.kernel.org directly for your CI needs. It is not viable for us to provide enough capacity to withstand dozens of CI nodes all trying to clone linux.git at the same time for multiple CI farms. You *will* get ip-blocked. Please set up a local mirror and hit that instead.
3
45
54
Paul Barker
pbarker@social.afront.org
@monsieuricon I wonder how much bandwidth waste we could eliminate globally if people used caches and local mirrors correctly in CI.
0
0
0
Jeroen Massar
jeroen@secluded.ch@monsieuricon if there was a way to "subscribe" to new commits, per branch/tag/ref or for everything, one could make a pretty decent mirror that is push instead of pull, as one now had to poll which is not always the most scalable system; push would allow the repo host to push slower if resources would be constrained
1
0
0
mcgrof
mcgrof
@monsieuricon FWIW we encourage and promote a local mirror on kdevops so all CI stuff we use uses the local mirror. A mirror however still does have to rely on git fetching to ensure we get the latest code, we use systemd timers for this and these vary depending on the repo. For Linus' tree we have a refresh of every 5 minutes while for linux-next its every 6 hours. If you think we should change any of these let us know! https://github.com/linux-kdevops/kdevops/blob/main/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
RE: https://social.kernel.org/objects/22653bb0-83b6-4316-a44d-c2b3a1f2eac5
RE: https://social.kernel.org/objects/22653bb0-83b6-4316-a44d-c2b3a1f2eac5
1
0
0
K. Ryabitsev 🍁
monsieuricon
@mcgrof we already make it extremely efficient to mirror git.kernel.org: https://pypi.org/project/grokmirror/ : you can poll that every 10 seconds and know exactly when each repo is updated.
0
2
1
K. Ryabitsev 🍁
monsieuricon
@jeroen Please just use https://pypi.org/project/grokmirror/ -- we make it extremely easy to mirror the entire collection efficiently and grab all updates almost as soon as they are there, in any repo.
0
1
7
mcgrof
mcgrof
@monsieuricon ok this solves the "we have control of the network and at least a server to mirror for CIs", but sometimes it's also desirable to rely on cloud resources for a CI. I believe the solution there is to collaborate for each cloud provider to ensure they also have mirrors and then our software like kdevops can use the prefix of the cloud provider for mirrors. Unless that is already set up?
RE: https://social.kernel.org/objects/22653bb0-83b6-4316-a44d-c2b3a1f2eac5
RE: https://social.kernel.org/objects/22653bb0-83b6-4316-a44d-c2b3a1f2eac5
1
0
0
K. Ryabitsev 🍁
monsieuricon
@mcgrof No, and I don't really see a way to make that work reliably. One of the requests I have is to provide a containerized image that would make it easy for someone to run a git.kernel.org mirror, but that's non-trivial, even if we limit this to just the 3 main repos (torvalds, stable, next). The fickle nature of container storage makes it very easy to introduce git repo corruption, which is really hard to fix without destroying/rebuilding the whole thing.
1
0
0
mcgrof
mcgrof
@monsieuricon well, in that case I suppose we can try to lobby for cloud providers to use https://pypi.org/project/grokmirror/ and hope for the best
0
0
0
Ian Jackson
Diziet@tech.lgbt@monsieuricon I have a small script that you can use as a cacheing git proxy. Small but very effective. It's in my chiark-scripts package
0
0
0