HN Reader

Oh that's interesting. jemalloc is the memory allocator used by redis, among other projects. Wonder what the performance impact will be if they have to change allocators.

All the allocators have the same issue. They largely work against a shared set of allocation APIs. Many of their users mostly engage via malloc and free.

So the flow is like this: user has an allocation looking issue. Picks up $allocator. If they have an $allocator type problem then they keep using it, otherwise they use something else.

There are tons of users if these allocators but many rarely engage with the developers. Many wouldn’t even notice improvements or regressions on upgrades because after the initial choice they stop looking.

I’m not sure how to fix that, but this is not healthy for such projects.

I very recently used jemalloc to resolve a memory fragmentation issue that caused a service to OOM every few days. While jemalloc as it is will continue to work, same as it does today, I wonder what allocator I should reach for in the future. Does anyone have any experiences to share regarding tcmalloc or other allocators that aim to perform better than stock glibc?

How cool would it be to see Doug Lea pick up the torch and create a modern day multi-threaded dlmalloc2!?

What allocator does Apple use?

A bad choice of title, as "postmortem" made me think there was some severe outage caused by jemalloc.

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s easily usable; TCMalloc is great, but is an absolute nightmare to use if you’re not using bazel (this has become slightly less true now that bazel 7.4.0 added cc_static_library so at least you can somewhat easily export a static library, but broadly speaking the point still stands).

I’ve been meaning to ask Qi if he’d be open to cutting a final 6.0 release on the repo before re-archiving.

At the same time it’d be nice to modernize the default settings for the final release. Disabling the (somewhat confusingly backwardly-named) “cache oblivious” setting by default so that the 16 KiB size-class isn’t bloated to 20 KiB would be a major improvement. This isn’t to disparage your (i.e. Jason’s) original choice here; IIRC when I last talked to Qi and David about this they made the point that at the time you chose this default, typical TLB associativity was much lower than it is now. On a similar note, increasing the default “page size” from 4 KiB to something larger (probably 16 KiB), which would correspondingly increase the large size-class cutoff (i.e. the point at which the allocator switches from placing multiple allocations onto a slab, to backing individual allocations with their own extent directly) from 16 KiB up to 64 KiB would be pretty impactful. One of the last things I looked at before leaving Meta was making this change internally for major services, as it was worth a several percent CPU improvement (at the cost of a minor increase in RAM usage due to increased fragmentation). There’s a few other things I’d tweak (e.g. switching the default setting of metadata_thp from “disabled” to “auto”, changing the extent-sizing for slabs from using the nearest exact multiple of the page size that fits the size-class to instead allowing ~1% guaranteed wasted space in exchange for reducing fragmentation), but the aforementioned settings are the biggest ones.

Your work was so impactful over a long period from Firefox to Facebook. Honored to have been a small part of it.

Thanks for everything, JE!

jemalloc is always the first thing I installed whenever I had to provision bare servers.

If jemalloc is somehow the default allocator in Linux, I think it will not have a hard time retaining contributors.

Jason, here is a story about how much your work impacts us. We run a decently sized company that processes hundreds of millions of images/videos per day. When we first started about 5 years ago, we spent countless hours debugging issues related to memory fragmentation.

One fine day, we discovered Jemalloc and put it in our service, which was causing a lot of memory fragmentation. We did not think that those 2 lines of changes in Dockerfile were going to fix all of our woes, but we were pleasantly surprised. Every single issue went away.

Today, our multi-million dollar revenue company is using your memory allocator on every single service and on every single Dockerfile.

Thank you! From the bottom of our hearts!

Kind of nuts that he worked on Jemalloc for over a decade while having personal preference for garbage collection. I'm surprised he doesn't have more regret.

The article mentioned the influence of large-scale profiling on both jemalloc and tcmalloc, but doesn't mention mimalloc. I consider mimalloc to be on par with these others, and now I am wondering whether Microsoft also used large scale profiling to develop theirs, or if they just did it by dead reckoning.

I still remember the day when I used jemalloc debug features to triage and resolve some nasty memory bloat issues in our code that use RockDB.

Good times.

Switching to jemalloc instantly fixed an irksome memory leak in an embedded Linux appliance I inherited many moons ago. Thank you je, we salute you!

Thank you. Jemalloc was recently recommended to me on some presentation about Java optimization.

I wonder if you did get everything you should from the companies that use it. I mean sometimes I feel that big tech firms only use free software, never giving anything to it, so I hope you were the exception here.

I've used jemalloc in every game engine I've written for years. It's just the thing to do. WAY faster on win32 than the default allocator. It's also nice to have the same allocator across all platforms.

I learned of it from it's integration in FreeBSD and never looked back.

jemalloc has help entertained a lot of people :)

Lesson: Don't let one megacorp dominate or take over your FOSS project. Push back somewhat and say "no" to too much help from one source.

I’ve wondered about this before but never when around people who might know. From my outsider view, jemalloc looked like a strict improvement over glibc’s malloc, according to all the benchmarks I’d seen when the subject came up. So, why isn’t it the default allocator?

been using jemalloc unknowingly for a long time. only after reading this post it hit how much of it was under the hood in things I’ve built. didn’t know the gc-style decay mechanism was that involved, or that it handled fragmentation with time-based heuristics. surprising how much tuning was exposed through env vars. solid closure

Nice post -- so does Facebook no longer use jemalloc at all? Or is it maintenance mode?

Or I wonder if they could simply use tcmalloc or another allocator these days?

Facebook infrastructure engineering reduced investment in core technology, instead emphasizing return on investment.

I believe there’s no other allocator besides jemalloc that can seamlessly override macOS malloc/free like people do with LD_PRELOAD on Linux (at least as of ~2020). jemalloc has a very nice zone-based way of making itself the default, and manages to accommodate Apple’s odd requirements for an allocator that have tripped other third-party allocators up when trying to override malloc/free.

Maybe add a link to the post on the github repo. I feel like this is important context for people visiting the repo in the future

FTA:

> And people find themselves in impossible situations where the main choices are 1) make poor decisions under extreme pressure, 2) comply under extreme pressure, or 3) get routed around.

It doesn't sound like a work place :-(

Looking at all the comments and lightly browsing the source code, I'm amazed. Both at how much impact a memory allocator can make, but also how much code is involved.

I'm not really sure what I expected, but somehow I expect a memory allocator to be ... smaller, simpler perhaps?

> jemalloc was probably booted from Rust binaries sooner than the natural course of development might have otherwise dictated.

FWIW while it was a factor it was just one of a number: https://github.com/rust-lang/rust/issues/36963#issuecomment-...

And jemalloc was only removed two years after that issue was opened: https://github.com/rust-lang/rust/pull/55238

Congrats on the great run and the future. Jemalloc was an inspirational to many memory allocators.

This is a good thing