K. Ryabitsev 🍁
@monsieuricon
Moderator
Posts
2041Following
230Followers
2518This account is for Linux/Kernel/FOSS topics in general: #linux, #kernel, #foss, #git, #sysadmin, #infrastructure.
For my personal account, please follow @monsieuricon@castoranxieux.ca.
Montréal, Québec, Canada 🇨🇦🇺🇦
K. Ryabitsev 🍁
monsieuricon
re: Ask fedi: replicating large file collections over slow links
Show content
@mss Correct, where it says "replica-na" it should say "primary-na".
The question of temporary files is actually an important consideration. The content of primary-na is distro data that is copied to the system via rsync with --delay-updates, so everything is written into ~tmp~ dirs and then moved in-place at the end of a successful run. Theoretically, this should be handled correctly by fs-based replication.
The question of temporary files is actually an important consideration. The content of primary-na is distro data that is copied to the system via rsync with --delay-updates, so everything is written into ~tmp~ dirs and then moved in-place at the end of a successful run. Theoretically, this should be handled correctly by fs-based replication.
1
0
0
K. Ryabitsev 🍁
monsieuricon
@amonakov Yes, I find it intriguing for some of its concepts, but it also has a major downside of needing extra 50+ TB for storing the repository. Also, it is really written to solve a different problem -- backing up data as opposed to replicating it to multiple nodes efficiently.
0
0
0
K. Ryabitsev 🍁
monsieuricon
@asdil12 I'm aware of syncthing, but I'd need to see some evidence that it can scale up to 50TB and millions of files, properly recognize things like moves and hardlinks, etc. Unfortunately, I'm not in a position to easily experiment with it.
0
0
2
K. Ryabitsev 🍁
monsieuricon
Ask fedi: replicating large file collections over slow links
Show content
I have a server "primary-na" with 50TB of arbitrary content in /srv, mostly in millions of small files, many of them identical hardlinks. I have 3 other servers across the world (copy-na, copy-eu, copy-ap) where I want to have the exact replica of primary-na's /srv. These replicas may be occasionally unavailable for hours on end, or they may be occasionally slow or under high load. The content on them may also occasionally bitrot and must be identified and healed.
I've researched this multiple times over the last few years, but I've still not found a solution that would beat "just run rsync over it when something changes on replica-na." It's simple and effective, but obviously super inefficient and IO-heavy on both ends.
Any suggestions on how you would do it?
I've researched this multiple times over the last few years, but I've still not found a solution that would beat "just run rsync over it when something changes on replica-na." It's simple and effective, but obviously super inefficient and IO-heavy on both ends.
Any suggestions on how you would do it?
12
4
5
K. Ryabitsev 🍁
monsieuricon
I'm umpiring 11 games in the span of 7 days. I now deeply understand Deakin's dialog line in this clip.
https://www.youtube.com/watch?v=oeFTSnUqKn4
https://www.youtube.com/watch?v=oeFTSnUqKn4
0
0
0
K. Ryabitsev 🍁
monsieuricon
Just a regular reminder that the war isn't over, civilians are still murdered in their homes by cruise missiles, and war crimes are committed daily by the invasion force occupying the sovereign territory of Ukraine.
Until the 1991 borders are restored, all war criminals are in jail, and reparations are paid off, I will continue to #StandWithUkraine, and so should you.
Until the 1991 borders are restored, all war criminals are in jail, and reparations are paid off, I will continue to #StandWithUkraine, and so should you.
1
8
28
K. Ryabitsev 🍁
monsieuricon
To me, the word "applet" means your disk starts churning for 30 seconds, and then there's a grey box on the page that doesn't do anything.
1
3
16
K. Ryabitsev 🍁
monsieuricon
To you, the word "applet" means:
45%
Any small embeddable application
40%
A java applet
13%
What the heck is an "applet"?
4
0
2
K. Ryabitsev 🍁
repeated
repeated
🇨🇦
K. Ryabitsev 🍁
monsieuricon
Going through Ansible tutorials. Thus far, it's mostly ansible-lint yelling at me.
0
0
6
K. Ryabitsev 🍁
monsieuricon
My kid spent the last 5 days building an ALU emulator in Minecraft with redstone and switches. Predictably, I've come up with a lot of "I think I just saw an actual spectre in your CPU" jokes, but thus far failed to execute them.
0
1
14
K. Ryabitsev 🍁
monsieuricon
Not sure what's causing it yet, but our primary mirror replication speeds are down to 20Mbps from the usually healthy 4-5 Gbps. I don't think it's anything on our end, so I'll try to flag our Equinix support overlords.
If you find that mirrors.edge.kernel.org is lagging behind, that would be why.
If you find that mirrors.edge.kernel.org is lagging behind, that would be why.
1
0
4
K. Ryabitsev 🍁
monsieuricon
As someone who has regretted naming something "b4" due to how hard it is to reliably search, I have to giggle at naming something "x".
0
3
19
K. Ryabitsev 🍁
monsieuricon
@vathpela Parents get understandably cautious when there's someone taking pictures of their kids, even in a public setting. In Quebec, the laws are very hostile to street photography, so the only time I can take pictures at my kids' games is if I swear to all parents that I will never share the pictures publicly.
1
0
0
K. Ryabitsev 🍁
monsieuricon
I am umpiring 7 games in 6 days this week. Thankfully, not all of them behind the plate, otherwise I think my squatting muscles would give out by Friday.
1
0
2