Hello Mozzers! I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt.
... crawlers can't access your site's robots.txt file, verify that the robots.txt file is accessible and in the top-level directory of your site.
This is a custom result inserted after the second result.
Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
Blocking crawlers from your live website is a no-no, but so is allowing them to crawl and index your pages that are still under development.
If you have included subdomains in the site audit, the file must be available; otherwise, the crawler will report an error stating that robots.
The robots.txt file is a simple text file on your web server which tells webcrawlers if they can access a file or not. You can always access ...
If you have some URL is not to be crawled by search engines, you must have a robots.text file with a line stating “Disallow: Url” line. It is ...
You definitely do have a robots.txt, and it's blocking all user agents (*) from crawling a lot of subfolders. But based on a quick glance, it ...
A robots.txt file is a text file located on a website's server that serves as a set of instructions for web crawlers or robots, such as search engine spiders.