What Is The robots.txt File?

What Is The robots.txt File?

When working to SEO optimize your website, you may have come across the advice to update your robots.txt file. This is not something you are likely to come across in your content management system (though you can often edit it within your CMS). Let’s take a closer look at the question, “What is the robots.txt file?”

The robots.txt file tells search engine crawlers what pages to specifically crawl. This reduces the number of requests your website receives at once so that it does not slow it down. Now, just because a page is not included in your robots.txt file does not mean it will not be crawled. The crawlers will still follow links on your page and site.

If you want to prevent a page from being crawled, you should either password protect the page or use the no-index protocol. Password protecting is usually recommended when developing a new page while adding a no-index tag is usually recommended when you do not want a live page crawled.

To learn more about technical SEO and schedule a free marketing consultation, call us at 206-787-0784. You can also reach out online via chat or contact form.