A Good WordPress Robots.txt

Allure Web Solutions Code Snippet

Download

Download robots.txt for WordPress Sites
Download robots.txt for Any Staging Sites

Robots.txt

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes

Cheat Sheet

Block all web crawlers from all content (necessary for public-facing staging sites)

User-agent: *
Disallow: /

Block a specific web crawler from a specific folder

User-agent: Googlebot
Disallow: /no-google/

Block a specific web crawler from a specific web page

User-agent: Googlebot
Disallow: /no-google/blocked-page.html

Sitemap Parameter

User-agent: *
Disallow:
Sitemap: http://www.example.com/none-standard-location/sitemap.xml

Optimal Format

Robots.txt needs to be placed in the top-level directory of a web server in order to be useful. Example: http://www.example.com/robots.txt

Mike Doubintchik

Author Mike Doubintchik

More posts by Mike Doubintchik

Leave a Reply