WordPress site robots file how to write?

In order for a website to write a good WordPress robots file, there are still some WordPress pages to study.

Here are the categories that must be banned:

Disallow the inclusion of WordPress system files:
Disallow:/wp-admin/
Disallow:/wp-content/
Disallow:/wp-includes/
Do not include repeat content, author pages, dynamic pages and other WordPress spam pages:Disallow: /trackback/
Disallow: /author/
Disallow: //feed Disallow: /function.
Disallow: /?s=*
Disallow: //?s=\
Disallow: /wp-*.php

Combining the above, we can write a complete WordPress site robots file like this:

User-agent: *
Disallow:/wp-admin/
Disallow:/wp-content/
Disallow:/wp-includes/
Disallow: /trackback/
Disallow: /author/
Disallow: //feed Disallow: /function.
Disallow: /?s=*
Disallow: //?s=\
Disallow: /wp-*.php

Disallow:/DotBot
Disallow:/SemrushBot
Disallow:/Uptimebot
Disallow:/MJ12bot
Disallow:/MegaIndex.ru
Disallow:/ZoominfoBot
Disallow:/Mail.Ru
Disallow:/SeznamBot
Disallow:/BLEXBot
Disallow:/ExtLinksBot
Disallow:/aiHitBot
Disallow:/Researchscan
Disallow:/DnyzBot
Disallow:/spbot
Disallow:/YandexBot
Disallow:/404.html

sitemap:https://bkzzz.com/sitemap.xml

Do not allow dotbot to grab yandexbot, these are unnecessary spiders.

Leave a Comment