HN Reader

I have an ASCII art Easter egg like this in an SEO product I made. :)

https://www.checkbot.io/robots.txt

I should probably add this SEO tip too because the purpose of robots.txt is confusing: If you want to remove/deindex a page from Google search, you counterintuitively need to allow the page to be crawled in the robots.txt file, and then add a noindex response header or noindex meta tag to the page. This way the crawler gets to see the noindex instruction. Robots.txt controls which pages can be crawled, not which pages can be indexed.

What's the purpose of "User-Agent: DemandbaseWebsitePreview/0.1"? I couldn't find anything about that agent, but I assume it's somehow related to demandbase.com?

But why are it and twitter the only whitelisted entries? Google and bing missing is a bit surprising, but I assume they're whitelisted through a different mechanism (like a google webmaster account)?

This is what happens if your robot isn't nice

  > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com
  HTTP/2 403

If those robots could read, they'd be very upset.

easy guess that length breaks some legacy stuff

but every robots.txt should have a auto-ban trap line

ie. crawl it and die

basically a script that puts the requesting IP into firewall

of course it's possible to abuse that so it has to be monitored

That’s a funny one!

Anyone knows of others like that?

Here is mine: https://FreeSolitaire.win/robots.txt

What does “OUR TREE IS A REDWOOD” refer to? A quick search doesn’t yield any definite results.

Has anyone worked on anything like this for AI scrapers?

That’s cool, if any scrapers would still respect the robots.txt that is

One nice thing about CF's robots.txt is its inclusion of a sitemap:

https://www.cloudflare.com/sitemap.xml

which contains links to educational materials like

https://www.cloudflare.com/learning/ddos/layer-3-ddos-attack...

Potentially interesting to see their flattened IA....