Tell HN: Cloudflare's D1 service degraded since 2 days

Public service announcement: if you're seeing very slow Cloudflare D1 query latencies, you're not alone.

I just burned hours trying to figure out why basic D1 SQL queries against a 100MB database are taking 20-30 seconds wall time. Turns out there's an ongoing D1 issue, but it's only being acknowledged in Cloudflare's D1 Discord channel, while cloudflarestatus.com says everything is a-ok.

Posting here in case it saves someone else the debugging rabbit hole.

I have been praising Cloudflare up and down over the past year. As a software programmer, I rarely want to care about DevOps bikeshedding to get my applications up and running.

So after having used GCP, AWS, and all the orchestration baggage that came with it, developing and testing user-facing apps with Miniflare, workerd, D1, and wrangler has been a breeze... until I discovered some of my D1 calls take 20 seconds to resolve on a table with 10 rows. Query takes sub-millisecond.

Status page looking good, all systems operational. Digging through issues, pulling my hair out. Then stumbled upon this post to discover there is indeed an issue, and it is acknowledged on Discord... Discord? Why?

Honestly, I've been the fool here. Discouraged every single soul I know from vendor lock-in, only to vendor lock-in myself because I really liked the product.

Cloudflare is in a great position to dominate the ecosystem. They understand our needs. However, feature creep, half-baked releases, and improper communication don't give me the confidence to continue using their services at scale.

Incident is marked as resolved... tracing outliers tell a different story. Guess it's time to move on. Can't claim you offer a reliable database at scale if a trivial lookup on a 10-row table resolves in 20 seconds.

1 day agoby oraybey

We have also a service impacted by this. There is no status acknowledgement of the severe degradation. Most of the time queries just take 2-10s, some even 20s+ and a few times they even hang indefinitely. It's a CF-internal networking issue and not SQL execution issue because in the meta data you can see SQL processing taking like 1ms. Even a simple "SELECT 123" in their dashboard D1 console can take 10+ seconds.

Our company opened an issue but we have not heard back in 3 days. The issue has been going on for more than the 3 days, we've noticed it nearly a week ago.

You can also find people in the CF community forums or in other places complaining about D1 latency and networking issues going back as much as a year. Nobody from CF replies in their community forums.

I think this is 1. not an acceptable level of service 2. not acceptable level of support 3. very surprising for a company who is all about fast and reliable networking.

Unfortunately I experienced all kinds of problems with Cloudflare and their non-CDN services over the past couple of months and I've also never seen a major cloud service provider suffering from so many dashboard errors and bugs.

Cloudflare folks, if you read this: you need a Snow Leopard year. You've launched many services and products but they are riddled with bugs, performance issues and outright outages. The recent launch of your data warehouse offering which in our testing (and I think even in your presentation) showed data processing speeds measured in the kilobytes/s while not even supporting aggregations at launch would have been embarassing even 20 years ago.

Please please step up your quality game because you do have some interesting offerings with good potential. The two huge outages last year were just surfacing deeper issues that seem to permeate throughout the stack.

2 days agoby eis

After days, there is finally an incident up: https://www.cloudflarestatus.com/incidents/kzvk0c2s5fy7

2 days agoby iowahansen