--- title: "robots.txt Explained: What It Can and Cannot Do | MCC" description: "robots.txt guides crawlers on what to visit, but a wrong line can hide your whole site. Learn the syntax, safe examples and the common mistakes." url: "https://mcc-solutionsnj.com/websites/guides/robots-txt-explained-what-it-can-and-cannot-do" --- Guide · Updated October 12, 2026 # robots.txt Explained: What It Can and Cannot Do The robots.txt file sits at the root of your site and gives instructions to well-behaved crawlers. It is small and easy to edit, and a single misplaced line can remove your whole site from search. Knowing its limits keeps you out of trouble. By [Matthew Coleman](/about), MCC Solutions **Quick answer.** A robots.txt file at yourdomain.com/robots.txt gives instructions to well-behaved crawlers about areas they should not visit. Reputable search engines follow it, but malicious bots ignore it, so it is a request, not a lock, and never use it to hide private information. **Free tools:** [Free website SEO check](/websites/seo-checker) · [Search result preview](/websites/search-result-preview) ## What the file does At yourdomain.com/robots.txt, crawlers look for rules before fetching pages. Rules say which areas a given crawler should not visit. Reputable search engines follow them; malicious bots ignore them. It is a request, not a lock, so never use it to hide private information. ## Basic syntax A group starts with User-agent, naming a crawler or an asterisk for all. Then Disallow lines list paths to avoid, and Allow lines carve out exceptions. A typical small business file might read: User-agent: *, then Disallow: /wp-admin/, then Allow: /wp-admin/admin-ajax.php, then a Sitemap line giving the full sitemap address. Paths are case sensitive and match from the start. A line reading Disallow: / blocks the entire site, which is correct only for sites under construction. ## Blocking is not the same as hiding If a blocked page has links pointing to it, a search engine can still list the address without a description. To keep a page out of results, use a noindex tag instead, and let crawlers reach the page so they can see it. Blocking in robots.txt and adding noindex together backfires, because the crawler never reads the tag. For truly private content, use password protection. ## Mistakes that cause real damage The classic disaster: a developer blocks the site during a redesign and the rule stays after launch. Another is blocking CSS and script folders, which stops search engines from rendering pages correctly. Blocking image folders can remove pictures from image search. Typos, wildcards and trailing characters also change meaning. After any site relaunch, load your robots.txt in a browser and read it line by line. ## Testing and safe editing Search Console shows how Google reads your robots.txt and whether pages are blocked. The URL Inspection tool tells you if a specific address is blocked. WordPress sites may generate a virtual file, edited via an SEO plugin, while other platforms offer a settings area. Keep a copy of the working version before you edit. Change one rule at a time and retest. If unsure, leaving the default is safer than clever tweaks. ## A safe starting file for a small site For many small business sites, the safest file is nearly empty. A WordPress site might disallow only the admin folder while allowing the one file that front-end features need, then list the sitemap address. A site built on a hosted builder usually generates the file for you and often cannot be changed. Before adding any new rule, ask what problem it solves. If you cannot name a specific section that wastes crawler time or exposes clutter, leave it alone. Over-blocking causes more harm than a few extra crawled pages ever will. If you manage several sites or subdomains, remember that each hostname has its own file. A correct file on the main site does not protect or control a booking subdomain that is run by another service. ## Common questions **Do I need a robots.txt file?** Not strictly. Without one, crawlers can visit everything. It is useful for pointing at your sitemap and excluding admin areas. **Can robots.txt remove a page from Google?** No. Use a noindex tag or the removal tools, and keep the page crawlable so the tag is seen. **Does it stop AI crawlers or scrapers?** Only the ones that choose to follow it. Treat it as a polite request. This guide is general information. Search engines change how they work and their guidelines; check Google's current documentation for your situation. Nobody can guarantee a ranking. Let us handle it [### Technical SEO & schema markup Technical SEO for small business websites: indexing, sitemaps, redirects, canonical tags and schema markup for local businesses and services. Technical SEO →](/websites/services/technical-seo-and-schema)[### Website care & updates Monthly website care: updates, backups, security checks, speed and content changes for small businesses. Website care →](/websites/services/website-care) More guides - [Canonical Tags Explained: Handling Duplicate Pages](/websites/guides/canonical-tags-explained-for-duplicate-pages) - [When to Use a Noindex Tag (and When Not To)](/websites/guides/when-to-use-a-noindex-tag-on-your-website) - [LocalBusiness Schema: The Fields That Actually Matter](/websites/guides/localbusiness-schema-fields-that-matter) - [FAQ Schema: What Changed and When to Still Use It](/websites/guides/faq-schema-what-changed-and-when-to-use-it) Free tools - [Website basics check](/websites/seo-checker) - [Is your website working for you? quiz](/websites/quiz) [All guides](/websites/guides) Reach small business owners working on their website and Google listing who read this guide: [advertise here](/advertise#websites). Do it yourself ### Small Business Website Starter A fast, mobile-friendly one-page site you edit in a text editor and host anywhere, with the sections a local service business needs and a plain setup guide. [See what's inside ($29)](/toolkits/small-business-website-starter) [Free sample](/toolkits/small-business-website-starter#free-starter)14-day full refund ## Rather have us handle it? Ask for a quote. [Get a Quote](/websites/#quote)[Call (908) 444-6373](tel:+19084446373)