Fetch robots.txt, and a few random thoughts

What robots.txt is

A robots.txt file is a set of instructions for bots. It sits in the source files of most websites, and it is mostly aimed at well-behaved bots like search engine crawlers, since badly-behaved ones aren't likely to follow it anyway.

It's just a text file with no HTML in it, hence the .txt. It's hosted on the web server like any other file, and you can see any site's by taking the homepage URL and adding /robots.txt. Nothing links to it, so people don't stumble across it, but most crawlers look for it before they touch anything else.

You can pull one up with my tool: FetchRobotstxt.

My own experience

Running a few hobby sites in my spare time, keeping track of robots.txt can feel like herding cats. There's always some crawler finding a new corner of your site to explore.

A few months back I got an urgent call from a friend who runs an e-commerce site. They'd just pushed changes live and noticed a huge spike in server load. After some debugging we realised they had accidentally blocked all bots from crawling the main pages in their robots.txt. Whoops. It took a bit of scrambling to roll that back before it hurt their SEO too badly. A good reminder for me to double-check my own configs more carefully.

In general I take a lenient approach with bots unless they're clearly misbehaving. Most of them just want to slurp up content like a hungry vacuum cleaner. As long as they're not hammering the servers or sneaking into places they shouldn't, I say let them crawl. It's the spammy ones, and the legitimate ones gone rogue, that bother me.

For those, targeted blocks in robots.txt can help. I've used other tricks over the years too, like browser fingerprints or awkward URLs a human would never follow. Not foolproof by any means, but it discourages the less motivated scrapers.

robots.txt is one tool in the box. Talking to bot owners and keeping an eye on your logs still matters. With a bit of care you can have friendly relations with most crawlers out there.

Just watch out for the ones acting shady.