Add the paths to block, the paths to allow, your sitemap, and whether AI crawlers may read your site. Copy the finished robots.txt into your site root.
Enter your sitemap URL so crawlers can find it, then list the paths you want kept out of search, one per line, such as /admin/ or /cart/. Add any paths inside a blocked folder that should stay open in the Allow box. Choose whether to block AI crawlers like GPTBot, and press Generate. Copy the output into a file named robots.txt at the root of your domain, then test it with the robots.txt checker.
robots.txt is the first file a crawler asks for, and one wrong line can hide your whole site from Google or leave your checkout pages sitting in the index. It is also where you decide whether AI models can train on your content. A file with the right structure, no typos in the directive names and an absolute sitemap URL takes a minute to make and removes a whole class of crawl problems.
At the root of the domain, so it loads at yourdomain.com/robots.txt. A file in a subfolder is ignored. Each subdomain needs its own file.
No. It stops Google crawling the page, but the URL can still be indexed if other pages link to it. To remove a page, let Google crawl it and add a noindex meta tag or header instead.
That depends on whether you want your content used to train models and to answer questions in ChatGPT and similar tools. Blocking GPTBot stops OpenAI's training crawler; blocking ChatGPT-User and OAI-SearchBot also stops your pages being cited in answers, which many sites want to keep.