Robots.txt fetcher
Read any site's robots.txt as the crawlers see it — the directives it sets, the paths it blocks, and the sitemaps it points to.
SERVER FETCH — THE SITE YOU SUBMIT IS REQUESTED BY OUR SERVER
INPUThttp or https
DIRECTIVES5 recognised
USER-AGENTNames the crawler the rules below apply to. `*` means every crawler.
DISALLOWBlocks access to a path or directory. An empty value blocks nothing.
ALLOWPermits a path that sits inside an otherwise disallowed directory.
SITEMAPPoints crawlers at an XML sitemap. Absolute URL, one line per sitemap.
CRAWL-DELAYRequested delay between requests. Googlebot ignores this directive.
OUTPUTno fetch yet
AWAITING A SITE