ToolStop
Definition

What is robots.txt?

robots.txt is a plain-text file at the root of a website that tells web crawlers which paths they may or may not fetch.

Also called: robots file · crawler directives

The file lives at /robots.txt and uses a simple grammar: 'User-agent:' followed by 'Allow:' and 'Disallow:' rules. A single 'Disallow: /' locks every crawler out of the whole site — usually not what you want.

robots.txt is a request, not a wall. Well-behaved crawlers (Googlebot, Bingbot) obey it; malicious scrapers ignore it. Use HTTP auth or firewall rules for actual access control.

A typical setup allows everything, blocks internal search or admin URLs, and points to the XML sitemap with a 'Sitemap:' directive.

Try a tool that uses robots.txt

Robots.txt Generator

Create a robots.txt file with allow, disallow, and sitemap directives.

Sitemap Generator

Build a valid XML sitemap from a list of URLs with priority and change frequency.

SEO Audit Tool

Audit on-page SEO factors from any HTML source in seconds.

Related definitions

Canonical URL

A canonical URL is the single, preferred web address for a page — declared with a <link rel="canonical"> tag so search engines consolidate ranking signals on one version.

Meta Description

A meta description is a short HTML tag that summarises a page — Google often uses it as the grey snippet under the blue link on the search results page.

Open Graph

Open Graph is a set of <meta property="og:…"> tags, invented by Facebook, that control how your page looks when someone shares the link on social media or chat apps.

Schema Markup (Structured Data)

Schema markup is structured data — usually JSON-LD in a <script> tag — that describes a page's contents in schema.org vocabulary so search engines can render rich results.