What is robots.txt?
robots.txt is the file that tells crawlers which paths they may request. What it can and cannot do, and how Goodspeed sets it on staging and production.
Founder of Goodspeed
robots.txt is a plain text file, served at /robots.txt, that tells crawlers which parts of a site they may request. Well-behaved search engines read it before they crawl. It can point them to your XML sitemap, and it can ask them not to waste time on utility paths such as search results, filters, or staging leftovers.
It is a request, not a security control. A crawler can ignore it. A blocked URL can still appear in search results if other sites link to it. If a page must stay out of the index, use a noindex tag or put it behind authentication. If a path must stay private, use proper access control, not robots.txt.
Why does robots.txt matter?
A badly written file can hide a whole site, or leave a staging environment open. A missing file is usually fine, but a file left over from staging that says "disallow everything" will quietly stop a launch from being crawled. As AI crawlers have become more common, robots.txt is also where teams decide which bots may read public pages, alongside an llms.txt file if you publish one.
It is a small file with outsized consequences, which is why it belongs in technical SEO and in launch-week QA, not as an afterthought.
How does Goodspeed handle robots.txt?
Staging and production get different files. Staging stays closed to public crawlers. Production allows the pages you want crawled, blocks obvious utility paths, and references the sitemap. We check the live file on launch day, because a copied staging rule is a common and expensive mistake. If you want a policy for AI crawlers, we write it down and keep it consistent with llms.txt.
This is part of a build or a website migration, which start from $10,000. A Tune-Up at $5,000 can include a review of a file that has drifted.
What does a typical robots.txt setup look like in practice?
A marketing site allows standard search crawlers, points at /sitemap.xml, and disallows a /search/ path and a preview path the CMS uses. Staging, on a different host, disallows all crawlers. On launch day someone opens /robots.txt on the live domain and confirms the production file is the one being served. A later blog post about AI policy is matched by a change to both robots.txt and llms.txt, rather than only one of them.
What are the common mistakes?
- Shipping the staging "disallow all" file to production.
- Using robots.txt as the only way to keep a page out of the index.
- Blocking CSS or JavaScript that a crawler needs to understand the page.
- Forgetting to mention the sitemap.
- Inviting an AI crawler in llms.txt while blocking it in robots.txt, or the reverse.
Common questions
What is robots.txt?
robots.txt is a text file at the root of a site that gives crawlers instructions about which paths they may or may not request. It is a request, not a lock, and it is the wrong tool if you need to keep a page out of the search index or behind authentication.
Does robots.txt stop a page from being indexed?
Not reliably. If other sites link to a blocked URL, a search engine may still index the URL without crawling the page. To keep a page out of the index, use a noindex tag, or keep the page behind a login. Use robots.txt to manage crawl budget and to keep crawlers out of utility paths.
Should we block AI crawlers in robots.txt?
That is a policy decision, not a technical default. Some teams want AI crawlers to read public docs and blog posts, others do not. If you also publish an llms.txt file, keep the two consistent so you are not inviting a crawler in one file and blocking it in the other.
How does Goodspeed handle robots.txt?
We ship a clear production file, a stricter staging file, and we check that the staging rules never go live by accident. The production file points at the XML sitemap. This is part of a build or a migration, which start from $10,000.
Which related terms should you read next?
You can also browse the glossary, the resources hub, try the migration planner, or book a call.
About the Author
Founder of Goodspeed
Harish Malhi is the founder of Goodspeed. He left Google in 2019 to build his own product, and started Goodspeed to give other founders the same head start. Today the studio designs and builds websites and the whole digital experience around them for funded tech companies, at the moments that matter: a raise, a launch, a press cycle. Goodspeed is a Framer Enterprise Expert, was Bubble’s Agency of the Year in 2024, and has shipped 300+ products, including launch pages for an NVIDIA-backed AI lab and a Webby-nominated site for a Boeing-backed aviation company. It holds a 5.0 rating from 46 reviews on Clutch.
