HN Reader

I'm the author of Theine (both Go and Python). I actually started with the Python version, using Ristretto as a reference implementation, including its hit ratio benchmarks. Naturally, I had to run Ristretto's benchmark first to ensure it was working correctly, which is how I discovered the issue in the first place. After completing the Python version, I moved on to develop the Go version of Theine, which focus on better hit ratio than Ristretto.

Recently, I refactored both the Go and Python versions to adopt Caffeine’s adaptive algorithm for improved hit ratio performance. But now that Otter v2 has switched to adaptive W-TinyLFU approach and more closely aligned with Caffeine’s implementation, I’m considering focusing more on the Python version.

This feels like a good time to do so: the Python community is actively working toward free-threading, and once the GIL is no longer a bottleneck, larger machines and multi-threads will become more viable. Then a high-performance, free-threading compatible caching libraries in Python will be important.

Just wanted to say thanks for such a good write-up and the great work on Otter over the years. We've used Ristretto since the beginning of building SpiceDB and have been watching a lot of the progress in this space over time. We've carved out an interface for our cache usage a while back so that we could experiment with Theine, but it just hasn't been a priority. Some of these new features are exciting enough that I could justify an evaluation for Otter v2.

Another major for on-heap caches that wasn't mentioned their portability: for us that matters because they can compile to WebAssembly.

Thanks for the great writeup.

Out of curiosity, has any benchmarking been done to compare Otter etc vs things like (localhost/Unix socket) Redis?

Much of the complexity of caching comes from trying to work well all workloads. If the workload is known, I think in many cases a specialized simpler cache can outperform some of these libraries.

Enjoyed the article.

How do you compare to ccache as this is my go to cache library. Well the need is mostly on high traffic endpoints, so LRU it's.

I've been using Otter for ~2(?) years, and it's a good time. This post made me realize we should upgrade to v2, as we've had a craving for async refresh.

Thanks to maypok86 for their dedication. The long comment threads on Ristretto issues are endlessly informative and entertaining. ;)

We’re looking for a distributed Go cache.

We don’t want to round trip to a network endpoint in the ideal path, but we run multiple instances of our monolith and we want a shared cache tier for efficiency.

Any architecture/library recommendations?

I'm glad I saw this article. I've been wanting a library like Guava/Caffeine to consider Go to feel more like a real language, but I wasn't aware of the last few years of developments. I am interested in this and I hope my team (and adjacent Go teams) will be too.

on S3-FIFO being problematic, have you looked into TinyUFO? (part of cloudflare/pingora)