Skip to main content

A Good WordPress Robots.txt

By May 14, 2016Blog, Snippets, Wordpress
Allure Web Solutions Code Snippet

Download

Download robots.txt for WordPress Sites
Download robots.txt for Any Staging Sites

Robots.txt

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes

Cheat Sheet

Block all web crawlers from all content (necessary for public-facing staging sites)

User-agent: *
Disallow: /

Block a specific web crawler from a specific folder

User-agent: Googlebot
Disallow: /no-google/

Block a specific web crawler from a specific web page

User-agent: Googlebot
Disallow: /no-google/blocked-page.html

Sitemap Parameter

User-agent: *
Disallow:
Sitemap: http://www.example.com/none-standard-location/sitemap.xml

Optimal Format

Robots.txt needs to be placed in the top-level directory of a web server in order to be useful. Example: http://www.example.com/robots.txt

allure

Author allure

More posts by allure

Leave a Reply

Designed by

best down free | web phu nu so | toc dep 2017