A Good WordPress Robots.txt

By May 14, 2016 March 20th, 2017 Blog, Snippets, Wordpress
Allure Web Solutions Code Snippet

Download

Download robots.txt for WordPress Sites
Download robots.txt for Any Staging Sites

Robots.txt

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes

Cheat Sheet

Block all web crawlers from all content (necessary for public-facing staging sites)

User-agent: *
Disallow: /

Block a specific web crawler from a specific folder

User-agent: Googlebot
Disallow: /no-google/

Block a specific web crawler from a specific web page

User-agent: Googlebot
Disallow: /no-google/blocked-page.html

Sitemap Parameter

User-agent: *
Disallow:
Sitemap: http://www.example.com/none-standard-location/sitemap.xml

Optimal Format

Robots.txt needs to be placed in the top-level directory of a web server in order to be useful. Example: http://www.example.com/robots.txt

Mike Doubintchik

Author Mike Doubintchik

More posts by Mike Doubintchik

Leave a Reply