Every WordPress site must have a robots.txt file.
That file tells to the search engine robots to index or not index some parts of your site.
In a WordPress site we want to index all post but we don't want to index the admin section (wp-admin) or core files (wp-includes).
So How to optimize your robots.txt file in WordPress?
1. Create the robots.txt file
In your root installation folder / if you don't have it already.
2. Fill the robots.txt file with this lines
Please notice the last line is the URL to your XML Sitemap. Create it if you don't have it.
User-agent: *
Disallow: /wp-includes/
Disallow: /wp-admin/
Sitemap: https://ricard.dev/sitemap.xml.gz
Code language: HTTP (http)
This lines will allow to index all your site but will disallow to index the wp-includes and the wp-admin folders.
Some WordPress developers also block the wp-content folder. It's up to you to index that folder.
Here is my robots.txt https://ricard.dev/robots.txt
Do you have a robots.txt file?
It very easy robots.txt code. and very easy to use.
Hi Quicoto, I was gladly surprised to find that you’ve become a web developer, I’m a fan of your photography blog π and it’s nice to see that you’ve chosen to walk this path.
I use a robots.txt file a little bit more complex; it’s nothing but restrictions to google so it doesn’t index dupe content (pagination links, author and category pagination too). You’re free to check it out anytime you want π
Hope that helps.
Hey Alan,
Thank you for your words π
Would be nice to see to arround here, an other expert eye to learn from π
I’m linking your robots here interesting example
http://tecnokun.com/robots.txt
Hi,
I’m not an expert, not even close to that but I love web design/development. I’m a huge fan of blogs like this and be sure that I will be checking out new content very often π
Thanks for the kind words.
Regards
I don’t have robot.txt but will include it now thanks for info.
You’re welcome.
The most important part is where you tell add the Sitemap url.
Thanks for your comment Ashutosh.
Regards