Our crawler was not able to access the robots.txt file on your site - Moz

Hello Mozzers! I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt.

Understand SEO crawling errors - HubSpot Knowledge Base

... crawlers can't access your site's robots.txt file, verify that the robots.txt file is accessible and in the top-level directory of your site.

Custom Result

This is a custom result inserted after the second result.

My blog cannot get crawled from Moz and Ahrefs - BlackHatWorld

Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.

6 Common Robots.txt Issues & And How To Fix Them

Blocking crawlers from your live website is a no-no, but so is allowing them to crawl and index your pages that are still under development.

How to Fix “Web Crawler Can't Find Robots.txt File” Issue | Sitechecker

If you have included subdomains in the site audit, the file must be available; otherwise, the crawler will report an error stating that robots.

web crawler - Incomplete robots.txt, what happens? - Stack Overflow

The robots.txt file is a simple text file on your web server which tells webcrawlers if they can access a file or not. You can always access ...

What happens when I don't add a robots.txt file to my website? Is this ...

If you have some URL is not to be crawled by search engines, you must have a robots.text file with a line stating “Disallow: Url” line. It is ...

Why do Screaming From & Moz only crawl one page on this website?

You definitely do have a robots.txt, and it's blocking all user agents (*) from crawling a lot of subfolders. But based on a quick glance, it ...

Robots.txt File: A Beginners Guide - Hike SEO

A robots.txt file is a text file located on a website's server that serves as a set of instructions for web crawlers or robots, such as search engine spiders.