First, I run a program that roughly allocates and fills 3 GB of RAM.
RSS:
python3 3156956
Then, I run a second instance of it. Wait for a while, and:
python3 1044036
python3 2380900
Neither of the programs has released any RAM. But the RSS is now roughly half of the RAM that the programs are using. But why? Well, that’s just because the rest of the stuff is in swap and not included in RSS anymore.
I’ve found out that memory is one of the most misunderstood concepts with modern operating systems. And that’s probably because it’s a very complex topic. I really don’t know anything about it, but over the decades, I’ve tried to get even the most important stuff right in my mind, because I’ve dealt a lot with VMs that had very limited RAM and high memory loading at least for short periods of time.
I often hear sysadmins, DBAs, and users claim that the system is just running out of memory, and they’re panicking and asking to get more memory before the system crashes. I just log in, run a program that allocates 4 gigs of RAM, fills it, and releases it. And whoa, now we’ve got 4 gigs of free RAM on a system that was supposedly out of RAM?!
So, when the system is out of memory, just run this to get 4 gigs of more RAM and reduce the RSS of other programs. (Yes, I used the 3 GB version for this demo, but I’ve bumped it up to 4 GB.)
py -c "grab_it = b' ' * 4 * 1024 * 1024 * 1024"
Of course you can use whatever language or memory reservation scheme you like instead of Python.
And for sure, I do realize that you’re posting from kernel.org, which scares me. What’s the most important thing I’ve missed in my post and got completely wrong? The VMs I’ve dealt with are mostly Windows, but basically behave quite similarly. - A link is also a valid reference. - Thank you.
@vbabka @sl My favorite call is fallocate()
with FL_PUNCH_HOLE
flag ;-) With just fallocate()
and mmap()
fairly complex memory management can be implemented.
Everyone’s favorite gpg-agent
demonstrates pretty well vsz/rss ratio:
main 21s
❯ ps -h -p "`pidof gpg-agent`" -o pid,comm,vsz,rss
PID COMM VSZ RSS
3739 gpg-agent 410782624 2480
For the sake of example this was executed in macOS (and for the record, numbers are KiB).
@vbabka @sl If I created a new systems language from scratch I’m not sure if I included to its stdlib anything else than memory mapping primitives.
I don’t understand why even latest of latest languages still implement POSIX API’s “best of tape drives” parts. Like e.g. Rust has only the tape drive API for files and you need to use external crate calle rust-vmm/virtual-memory
to get mmap 🤷