robots.txt для CMS на 2019 год

В последнее время часто приходится сталкиваться с разными системами управления и для них искать robots.txt или модифицировать стандартные.

  • Но и тот, и другой путь не очень хорош, так как найти можно устаревшую информацию, а самим модифицировать — занимает время.

Поэтому я собрал в этой теме основные примеры актуальных на 2019 год robots.txt.

Но прежде чем их представить вам, хочу описать главные аспекты и критерии по которым различаются инструкции для разных поисковых систем.

Для Яндекса это прежде всего параметр Clean-Param, он может защитить от дублей страниц, которые образуются в результате url содержащих GET — параметры.

А директиву Host уже можно не использовать, она ушла в прошлое, т.к. 301-й редирект её полностью заменил.

Ну и самая популярная ошибка, это закрытие css и js для Яндекса, многие вебмастера как буд-то проигнорировали эту статью на официальном блоге Яндекса.

Переходя к Google могу только сказать, что скоро перестанет учитываться crawl-delay.

Вступление на этом закончено, давайте перейдём к примерам для популярных CMS, которые я использую в своих роботах тхт.

robots.txt для Joomla

User-agent: Yandex
Allow: /*.css
Allow: /*.js
Allow: /*.jpg
Allow: /*.png
Allow: /*.gif
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type
Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket

User-agent: Googlebot
Allow: /*.css
Allow: /*.js
Allow: /*.jpg
Allow: /*.png
Allow: /*.gif
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
Disallow:*utm*=
Disallow:*openstat= 

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow:*utm*=
Disallow:*openstat= 

Sitemap: https://site.ru/sitemap.xml

robots.txt для WordPress

User-agent:Yandex
Allow:*/uploads
Allow:/*/*.js
Allow:/*/*.css
Allow:/wp-*.png
Allow:/wp-*.jpg
Allow:/wp-*.jpeg
Allow:/wp-*.gif
Allow:/wp-admin/admin-ajax.php
Disallow:/cgi-bin
Disallow:/?
Disallow:/wp-
Disallow:/wp/
Disallow:*?s=
Disallow:*&s=
Disallow:/search/
Disallow:/author/
Disallow:/users/
Disallow:*/trackback
Disallow:*/feed
Disallow:*/rss
Disallow:*/embed
Disallow:*/wlwmanifest.xml
Disallow:/xmlrpc.php
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type
Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket

User-agent:GoogleBot
Allow:*/uploads
Allow:/*/*.js
Allow:/*/*.css
Allow:/wp-*.png
Allow:/wp-*.jpg
Allow:/wp-*.jpeg
Allow:/wp-*.gif
Allow:/wp-admin/admin-ajax.php
Disallow:/cgi-bin
Disallow:/?
Disallow:/wp-
Disallow:/wp/
Disallow:*?s=
Disallow:*&s=
Disallow:/search/
Disallow:/author/
Disallow:/users/
Disallow:*/trackback
Disallow:*/feed
Disallow:*/rss
Disallow:*/embed
Disallow:*/wlwmanifest.xml
Disallow:/xmlrpc.php
Disallow:*utm*=
Disallow:*openstat=

User-agent:*
Disallow:/cgi-bin
Disallow:/?
Disallow:/wp-
Disallow:/wp/
Disallow:*?s=
Disallow:*&s=
Disallow:/search/
Disallow:/author/
Disallow:/users/
Disallow:*/trackback
Disallow:*/feed
Disallow:*/rss
Disallow:*/embed
Disallow:*/wlwmanifest.xml
Disallow:/xmlrpc.php
Disallow:*utm*=
Disallow:*openstat=

Sitemap:https://site.ru/sitemap.xml

robots.txt для Битрикс

User-Agent: Yandex
Allow: /*.js
Allow: /*.css
Allow: /local/*.js
Allow: /local/*.css
Allow: /upload/*.png$
Allow: /upload/*.jpg$
Disallow: */index.php
Disallow: /bitrix/
Disallow: /*?q=
Disallow: /*show_include_exec_time=
Disallow: /*show_page_exec_time=
Disallow: /*show_sql_stat=
Disallow: /*bitrix_include_areas=
Disallow: /*clear_cache=
Disallow: /*clear_cache_session=
Disallow: /*ADD_TO_COMPARE_LIST
Disallow: /*ORDER_BY
Disallow: /*PAGEN
Disallow: /*?print=
Disallow: /*&print=
Disallow: /*print_course=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*register=
Disallow: /*forgot_password=
Disallow: /*change_password=
Disallow: /*login=
Disallow: /*logout=
Disallow: /*auth=
Disallow: /*backurl=
Disallow: /*back_url=
Disallow: /*BACKURL=
Disallow: /*BACK_URL=
Disallow: /*back_url_admin=
Disallow: /*?utm_source=
Disallow: /*?bxajaxid=
Disallow: /*&bxajaxid=
Disallow: /*?view_result=
Disallow: /*&view_result=
Disallow: /cgi-bin/
Disallow: /search/
Disallow: /user/
Disallow: /local/
Disallow: /login/
Disallow: /*?gpage
Disallow: /*?action
Disallow: /*&
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type
Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket

User-Agent: Googlebot
Allow: /*.js
Allow: /*.css
Allow: /local/*.js
Allow: /local/*.css
Allow: /upload/*.png$
Allow: /upload/*.jpg$
Disallow: */index.php
Disallow: /bitrix/
Disallow: /*?q=
Disallow: /*show_include_exec_time=
Disallow: /*show_page_exec_time=
Disallow: /*show_sql_stat=
Disallow: /*bitrix_include_areas=
Disallow: /*clear_cache=
Disallow: /*clear_cache_session=
Disallow: /*ADD_TO_COMPARE_LIST
Disallow: /*ORDER_BY
Disallow: /*PAGEN
Disallow: /*?print=
Disallow: /*&print=
Disallow: /*print_course=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*register=
Disallow: /*forgot_password=
Disallow: /*change_password=
Disallow: /*login=
Disallow: /*logout=
Disallow: /*auth=
Disallow: /*backurl=
Disallow: /*back_url=
Disallow: /*BACKURL=
Disallow: /*BACK_URL=
Disallow: /*back_url_admin=
Disallow: /*?utm_source=
Disallow: /*?bxajaxid=
Disallow: /*&bxajaxid=
Disallow: /*?view_result=
Disallow: /*&view_result=
Disallow: /cgi-bin/
Disallow: /search/
Disallow: /user/
Disallow: /local/
Disallow: /login/
Disallow: /*?gpage
Disallow: /*?action
Disallow: /*&
Disallow:*utm*=
Disallow:*openstat= 

User-Agent: *
Disallow: */index.php
Disallow: /bitrix/
Disallow: /*?q=
Disallow: /*show_include_exec_time=
Disallow: /*show_page_exec_time=
Disallow: /*show_sql_stat=
Disallow: /*bitrix_include_areas=
Disallow: /*clear_cache=
Disallow: /*clear_cache_session=
Disallow: /*ADD_TO_COMPARE_LIST
Disallow: /*ORDER_BY
Disallow: /*PAGEN
Disallow: /*?print=
Disallow: /*&print=
Disallow: /*print_course=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*register=
Disallow: /*forgot_password=
Disallow: /*change_password=
Disallow: /*login=
Disallow: /*logout=
Disallow: /*auth=
Disallow: /*backurl=
Disallow: /*back_url=
Disallow: /*BACKURL=
Disallow: /*BACK_URL=
Disallow: /*back_url_admin=
Disallow: /*?utm_source=
Disallow: /*?bxajaxid=
Disallow: /*&bxajaxid=
Disallow: /*?view_result=
Disallow: /*&view_result=
Disallow: /*?_openstat
Disallow: /*?gclid=
Disallow: /*?yclid=
Disallow: /*?ymclid=
Disallow: /*utm_source=
Disallow: /*from=adwords
Disallow: /*?cid=
Disallow: /*?frommarket=
Disallow: /*?gpage
Disallow: /*?action
Disallow: /*&
Disallow:*utm*=
Disallow:*openstat= 

Sitemap:https://site.ru/sitemap.xml

robots.txt для OpenCart

User-agent: Yandex
Allow: /*.css
Allow: /*.js
Allow: /*.jpg
Allow: /*.png
Allow: /*.gif
Disallow: /amp/
Disallow: /amp/*/
Disallow: /*route=product/amp_product
Disallow: /index.php?route=product/amp_product
Disallow: /index.php?route=product/amp_product&product_id=
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /*compare-products
Disallow: /*search
Disallow: /*cart
Disallow: /*checkout
Disallow: /*login
Disallow: /*logout
Disallow: /*vouchers
Disallow: /*wishlist
Disallow: /*my-account
Disallow: /*order-history
Disallow: /*newsletter
Disallow: /*return-add
Disallow: /*forgot-password
Disallow: /*downloads
Disallow: /*returns
Disallow: /*transactions
Disallow: /*create-account
Disallow: /*recurring
Disallow: /*address-book
Disallow: /*reward-points
Disallow: /*affiliate-forgot-password
Disallow: /*create-affiliate-account
Disallow: /*affiliate-login
Disallow: /*affiliates
Disallow: /*?page=-
Allow: /catalog/view/javascript/
Allow: /catalog/view/theme/*/
Allow: /catalog/view/stylesheet/pdqo/
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type
Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket

User-agent: Googlebot
Allow: /*.css
Allow: /*.js
Allow: /*.jpg
Allow: /*.png
Allow: /*.gif
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /*?tracking=
Disallow: /*&tracking=
Disallow: /*compare-products
Disallow: /*search
Disallow: /*cart
Disallow: /*checkout
Disallow: /*login
Disallow: /*logout
Disallow: /*vouchers
Disallow: /*wishlist
Disallow: /*my-account
Disallow: /*order-history
Disallow: /*newsletter
Disallow: /*return-add
Disallow: /*forgot-password
Disallow: /*downloads
Disallow: /*returns
Disallow: /*transactions
Disallow: /*create-account
Disallow: /*recurring
Disallow: /*address-book
Disallow: /*reward-points
Disallow: /*affiliate-forgot-password
Disallow: /*create-affiliate-account
Disallow: /*affiliate-login
Disallow: /*affiliates
Disallow: /*?page=-
Allow: /catalog/view/javascript/
Allow: /catalog/view/theme/*/
Allow: /catalog/view/stylesheet/pdqo/
Disallow:*utm*=
Disallow:*openstat=

User-agent: *
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /*?tracking=
Disallow: /*&tracking=
Disallow: /*compare-products
Disallow: /*search
Disallow: /*cart
Disallow: /*checkout
Disallow: /*login
Disallow: /*logout
Disallow: /*vouchers
Disallow: /*wishlist
Disallow: /*my-account
Disallow: /*order-history
Disallow: /*newsletter
Disallow: /*return-add
Disallow: /*forgot-password
Disallow: /*downloads
Disallow: /*returns
Disallow: /*transactions
Disallow: /*create-account
Disallow: /*recurring
Disallow: /*address-book
Disallow: /*reward-points
Disallow: /*affiliate-forgot-password
Disallow: /*create-affiliate-account
Disallow: /*affiliate-login
Disallow: /*affiliates
Disallow: /*?page=-
Allow: /catalog/view/javascript/
Allow: /catalog/view/theme/*/
Allow: /catalog/view/stylesheet/pdqo/
Disallow:*utm*=
Disallow:*openstat=

Sitemap:https://site.ru/sitemap.xml
© 2019 artur2k.ru
top