This post is part of our blogging series in which we show new users and newbies how to setup and configure an online blog using WordPress. We’re have written a lot about blogging on this site. You may use the search box to find more content on blogging.
In this post, we’re going to describe and show you how to use Robots.txt file on your websites. The Robots.txt file doesn’t come with WordPress so you must create it yourself. It should be in the root directory of your WordPress site.
Robots.txt file contains sets of instructions to help website crawling bots on how to archive and categorize your website’s content. By default, search engines bots are configured to crawl and index all your website’s content. Content that contains sensitive information will also be indexed unless you specifically instruct the bots not to crawl or index these content.
Most search engines crawlers search for robots.txt file for instructions on how to crawl the site they are tying to index. If you don’t want a file or folder indexed by search engines, you must use disallowed attribute before each file or folder don’t want indexed.
So, what WordPress directories should you exclude? How do you determine which files to exclude from being indexed?
If you check the robots.txt file on this site, you’ll see I am excluding few directories. My robots.txt file is at https://www.liberiangeek.net/robots.txt
You can copy, modify and use it as you wish. If you need more tips on what to exclude when using WordPress, check out some of the bigger sites that are using WordPress.
Techcrunch’s robots.txe file is at http://techcrunch.com/robots.txt
WordPress.com’s robots.txt file is at http://wordpress.com/robots.txt
Go and find more websites that are using WordPress and copy that they have for. You may be able configure the perfect robots.txt file for your website.
Hope this helps.