Do you use your Robots.txt file effectively?
11/11/2015 6:55 AM by
If you decide to you use the redirect method when linking to external pages from your site, it's a good idea to have a robots.txt file in your root directory. In the robots.txt file, you specify that the robot (or spider as they're sometimes known) should not look in the redirects folder. This will avoid the problem that can sometimes occur, where a Google search will incorrectly display content from another site as if it was from your site.
Also, this avoids incorrect hits on your redirects, mucking up your statistics which is one of the main reasons you would use redirects in the first place!
Do you feel this rule needs an update?