HN Reader

Traffic to my blog plummeted this year and you can never be entirely sure how it happened. But here are two culprits i identified.

1. Ai overview: my page impressions were high, my ranking was high, but click through took a dive. People read the generated text and move along without ever clicking.

2. You are now a spammer. Around August, traffic took a second plunge. In my logs, I noticed these weird queries in my search page. Basically people were searching for crypto and scammy websites on my blog. Odd, but not like they were finding anything. Turns out, their search query was displayed as an h1 on the page and crawled by google. I was basically displaying spam.

I don't have much control over ai overview because disabling it means I don't appear in search at all. But for the spam, I could do something. I added a robot noindex on the search page. A week later, both impressions and clicks recovered.

Edit: Adding write up I did a couple weeks ago https://idiallo.com/blog/how-i-became-a-spammer

About six months ago Ahrefs recommended I remove some Unicode from the pathing on a personal project. Easy enough. Change the routing, set up permanent redirects for the old paths to the new paths.

I used to work for an SEO firm, I have a decent idea of best practices for this sort of thing.

BAM, I went from thousands of indexed pages to about 100

See screenshot:

https://x.com/donatj/status/1937600287826460852

It's been six months and never recovered. If I were a business I would be absolutely furious. As it stands this is a tool I largely built for myself so I'm not too bothered but I don't know what's going on with Google being so fickle.

Updated screenshots;

https://x.com/donatj/status/1999451442739019895

What I find strange about Google, is that there's a lot of illegal advertising on Google maps - things like accomodation and liquor sellers that don't have permits.

However, if they do it for the statutory term, they can then successfully apply for existing-use rights.

Yet I've seen expert witnesses bring up Google pins on Maps during tribunal over planning permits and the tribunal sort of acts as if it's all legit.

I've even seen the tribunals report publish screenshots from Google maps as part of their judgement.

Google search results have gone shit. I am facing some deindexing issues where Google is citing a content duplicate and picking a canonical URL itself, despite no similar content.

Just the open is similar, but the intent is totally different, and so is the focus keyword.

Not facing this issue in Bing and other search engines.

How does one debug issues like this?

I have a page that ranks well worldwide, but is completely missing in Canada. Not just poorly ranked, gone. It shows up #1 for keyword in the US, but won't show up with precise unique quotes in Canada.

I encountered the same problem. I also use the Bear theme, specifically Hugo Bear. Recently, my blog was unindexed by Bing. Using `site:`, there are no links at all. My blog has been running normally for 17 years without any issues before.

Entirely possible the rss failed validation triggered some spam flag that isn't documented, because documenting anti-spam rules lets spammers break the rules.

The amount of spam has increased enormously and I have no doubt there are a number of such anti-spam flags and a number of false positive casualties along the way.

Google search also favors large, well-known sites over newcomers. For sites that have a lot of competition, this is a real problem and leads to asymmetry and a chicken-and-egg problem. You are small/new, but you can't really be found, which means you can't grow enough to be found. At the same time, you are also disadvantaged because Google displays your already large competitors without any problems!

I'll be honest, I read "Google de-indexed my Bear Blog" and was looking forward to discovering an interesting blog about bears.

I bet Google doesn't know why either...

Breaking News: Google de-indexes random sites all of the time and there is often no obvious reason why. They also penalize sites in a way where pages are indexed but so deep-down that no one will ever find them. Again, there is often no obvious reason.

Sounds similar to https://news.ycombinator.com/item?id=46203343 in terms, that Google decides who survives and who does not in business

In 2025, is it still prohibively expensive to run some community-supported crawler & search engine? Without Google censorship, ads, and ai.

I never really use it but there is a lot in the Yahoo index that google refuses to index.

https://search.yahoo.com/search?p=blog.james-zhan.com&fr=yfp...

Without going into details. The company I work for has potentially millions of pages indexed. Despite new content being published everyday, since around the same October dates we are seeing a decrease in the number of indexed pages.

We have a consultant for the topic but I am not sure how much of that conversation I could share publicly so I will refrain myself of doing so.

But I think I can say that it is not only about data structure or quality. The changes in methodology applied by Google in September might be playing a stronger role than what people initially thought

I have the same issue with DollarDeploy and Bing (and consequently with DuckDuckGo which uses bing)

Primary domain cannot be found via search - Bing knows about brand, LinkedIn, YouTube channel and but refuses to show search results about primary domain.

Bing search console does not give any clue, force reindexing does not help. Google search works fine.

I noticed google not being able to find smaller blogs a few years ago. The sort of blogs I used to like and read a lot - small irregular blog of an expert in something like cryptography, sociology etc kind of disappeared from the search. Then they disappeared for real.

Even when I knew the exact name of article I was looking for google was unable to find it. And yes it still existed,

Parts of Google are all Blackbox-y. Never know when computer says no. And if they had usable ways to contact a human they’d just tell you they don’t know either

I ran into the same thing! My site still isnt indexed and I would REALLY like to not change the URL (its a shop and the url is printed on stuff) - redirects are my last resort.

But basically what happened: In august 2025 we finished the first working version of our shop. I wanted to accelerate indexing after some weeks because only ~50 of our pages were indexed and submitted the sitemap and everything got de-indexed within days. I thought for the longest time that its content quality because we sell niche trading cards and the descriptions are all one liners i made in Excel. ("This is $cardname from $set for your collection or deck!"). And because its single trading cards we have 7000+ products that are very similiar. (We did do all product images ourselves I thought google would like this but alas).

But later we added binders, whole sets and took a lot of care with their product data. The frontpage also got a massive overhaul - no shot. Not one page in index. We still get traffic from marketplaces and our older non-shop site. The shop itself lives on a subdomain (shop.myoldsite.com). The normal site also has a sitemap but that one was submitted 2022. I later rewrote how my sitemaps were generated and deleted the old ones in search console hoping this would help. It did not. (The old sitemap was generated by the shop system and was very large. Some forums mentioned that its better to create a chunked sitemap so I made a script that creates lists with 1000 products at a time as well as an index for them.)

Later observations are:

- Both sitemaps i deleted in GSC are still getting crawled and are STILL THERE. You cant see them in the overview but if you have the old links they still appear as normal.

- We eventually started submitting product data to google merchant center as well. It works 100% fine and our products are getting found and bought. The clicks still even show up in search console!!!! So I have a shop with 0 indexed pages in GSC that gets clicks every day. WTHeck?

So like... I dont even know anymore. Maybe we also have to restart like the person in the blog did and move the shop to a new domain and NEVER give google a sitemap. If I really go that route I will probably delete the cronjob that creates the sitemap in case google finds it by itself. But also like what the heck? I have worked in a web agency for 5 years and created a new webpage about every 2-8 weeks so i roughly launached about 50-70 webpages and shops and i NEVER saw that happen. Is it an ai hallucinating? Is it anti spam gone too far? Is it a straight up bug that they dont see? Who knows. I dont

(Good article though and I hope maybe some other people chime in and googlers browsing HN see this stuff).

The author doesn't know the cause but states "The whole affair is Google’s fault"?

A weird thing: on the hacker news page, in firefox mobile, all the visited links are grey, but the link to this blog post won't turn grey even when visited.

When I reload the page "https://journal.james-zhan.com/google-de-indexed-my-entire-b...", I get

Request URL: https://journal.james-zhan.com/google-de-indexed-my-entire-b...

Request Method: GET

Status Code: 304 Not Modified

So maybe it's the status code? Shouldn't that page return a 200 ok?

When I go to blog.james..., I first get a 301 moved permanently, and then journal.james... loads, but it returns a 304 not modified, even if i then reload the page.

Only when I fully sumbit the URL again in the URL-bar, it responds with a 200.

Maybe crawling also returns a 304, and Google won't index that?

Maybe prompt: "why would a 301 redirect lead to a 304 not modified instead of a 200 ok?", "would this 'break' Google's crawler?"

> When Google's crawler follows the 301 to the new URL and receives a 304, it gets no content body. The 304 response basically says "use what you cached"—but the crawler's cache might be empty or stale for that specific URL location, leaving Google with nothing to index.

We need a P2P internet.

No more Google. No more websites. A distributed swarm of ephemeral signed posts. Shared, rebroadcasted.

When you find someone like James and you like them, you follow them. Your local algorithm then prioritizes finding new content from them. You bookmark their author signature.

Like RSS but better. Fully distributed.

Your own local interest graph, but also the power of your peers' interest graphs.

Content is ephemeral but can also live forever if any nodes keep rebroadcasting it. Every post has a unique ID, so you can search for it later in the swarm or some persistent index utility.

The Internet should have become fully p2p. That would have been magical. But platforms stole the limelight just as the majority of the rest of the world got online.

If we nerds had but a few more years...

At the risk of sounding crazy, I de-indexed my blog myself and rely on the mailing list (which is now approaching 5000 subscribers) + reprints in several other online media to get traffic to me. On a good day, I get 5000 hits, which is quite a lot by Czech language community standards.

Together with deleting my Facebook and Twitter accounts, this removed a lot of pressure to conform to their unclear policies. Especially around 2019-21, it was completely unclear how to escape their digital guillotine which seemed to hit various people randomly.

The deliverability problem still stands, though. You cannot be completely independent nowadays. Fortunately my domain is 9 years old.

We depend too much on Google.