В последнее время часто приходится сталкиваться с разными системами управления и для них искать robots.txt или модифицировать стандартные.
- Но и тот, и другой путь не очень хорош, так как найти можно устаревшую информацию, а самим модифицировать — занимает время.
Поэтому я собрал в этой теме основные примеры актуальных на 2019 год robots.txt.
Но прежде чем их представить вам, хочу описать главные аспекты и критерии по которым различаются инструкции для разных поисковых систем.
Для Яндекса это прежде всего параметр Clean-Param, он может защитить от дублей страниц, которые образуются в результате url содержащих GET — параметры.
А директиву Host уже можно не использовать, она ушла в прошлое, т.к. 301-й редирект её полностью заменил.
Ну и самая популярная ошибка, это закрытие css и js для Яндекса, многие вебмастера как буд-то проигнорировали эту статью на официальном блоге Яндекса.
Ещё стоит отметить, что Google и Яндекс уже не учитывают директиву crawl-delay, её можно задать через вебмастер и консоль.
Вступление на этом закончено, давайте перейдём к примерам для популярных CMS, которые я использую в своих роботах тхт.
robots.txt для Joomla
User-agent: Yandex Allow: /*.css Allow: /*.js Allow: /*.jpg Allow: /*.png Allow: /*.gif Disallow: /administrator/ Disallow: /bin/ Disallow: /cache/ Disallow: /cli/ Disallow: /components/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /layouts/ Disallow: /libraries/ Disallow: /logs/ Disallow: /modules/ Disallow: /plugins/ Disallow: /tmp/ Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket User-agent: Googlebot Allow: /*.css Allow: /*.js Allow: /*.jpg Allow: /*.png Allow: /*.gif Disallow: /administrator/ Disallow: /bin/ Disallow: /cache/ Disallow: /cli/ Disallow: /components/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /layouts/ Disallow: /libraries/ Disallow: /logs/ Disallow: /modules/ Disallow: /plugins/ Disallow: /tmp/ Disallow:*utm*= Disallow:*openstat= User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /cli/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /logs/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow:*utm*= Disallow:*openstat= Sitemap: https://site.ru/sitemap.xml
robots.txt для WordPress
User-agent:Yandex Allow:*/uploads Allow:/*/*.js Allow:/*.css Allow:/wp-*.png Allow:/wp-*.jpg Allow:/wp-*.jpeg Allow:/wp-*.gif Allow:/wp-admin/admin-ajax.php Disallow:/cgi-bin Disallow:*/? Disallow:/wp- Disallow:/wp/ Disallow:*?s= Disallow:*&s= Disallow:/search/ Disallow:/author/ Disallow:/users/ Disallow:*/trackback Disallow:*/feed Disallow:*/rss Disallow:*/embed Disallow: */wp-json/ Disallow:*/wlwmanifest.xml Disallow:/xmlrpc.php Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket User-agent:GoogleBot Allow:*/uploads Allow:/*/*.js Allow:/*.css Allow:/wp-*.png Allow:/wp-*.jpg Allow:/wp-*.jpeg Allow:/wp-*.gif Allow:/wp-admin/admin-ajax.php Disallow:/cgi-bin Disallow:/? Disallow:/wp- Disallow:/wp/ Disallow:*?s= Disallow:*&s= Disallow:/search/ Disallow:/author/ Disallow:/users/ Disallow:*/trackback Disallow:*/feed Disallow:*/rss Disallow:*/embed Disallow: */wp-json/ Disallow:*/wlwmanifest.xml Disallow:/xmlrpc.php Disallow:*utm*= Disallow:*openstat= User-agent:* Disallow:/cgi-bin Disallow:/? Disallow:/wp- Disallow:/wp/ Disallow:*?s= Disallow:*&s= Disallow:/search/ Disallow:/author/ Disallow:/users/ Disallow:*/trackback Disallow:*/feed Disallow:*/rss Disallow:*/embed Disallow: */wp-json/ Disallow:*/wlwmanifest.xml Disallow:/xmlrpc.php Disallow:*utm*= Disallow:*openstat= Sitemap:https://site.ru/sitemap.xml
robots.txt для Битрикс
User-Agent: Yandex Allow: /*.js Allow: /*.css Allow: /local/*.js Allow: /local/*.css Allow: /upload/*.png$ Allow: /upload/*.jpg$ Disallow: */index.php Disallow: /bitrix/ Disallow: /*?q= Disallow: /*show_include_exec_time= Disallow: /*show_page_exec_time= Disallow: /*show_sql_stat= Disallow: /*bitrix_include_areas= Disallow: /*clear_cache= Disallow: /*clear_cache_session= Disallow: /*ADD_TO_COMPARE_LIST Disallow: /*ORDER_BY Disallow: /*PAGEN Disallow: /*?print= Disallow: /*&print= Disallow: /*print_course= Disallow: /*?action= Disallow: /*&action= Disallow: /*register= Disallow: /*forgot_password= Disallow: /*change_password= Disallow: /*login= Disallow: /*logout= Disallow: /*auth= Disallow: /*backurl= Disallow: /*back_url= Disallow: /*BACKURL= Disallow: /*BACK_URL= Disallow: /*back_url_admin= Disallow: /*?utm_source= Disallow: /*?bxajaxid= Disallow: /*&bxajaxid= Disallow: /*?view_result= Disallow: /*&view_result= Disallow: /cgi-bin/ Disallow: /search/ Disallow: /user/ Disallow: /local/ Disallow: /login/ Disallow: /*?gpage Disallow: /*?action Disallow: /*& Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket User-Agent: Googlebot Allow: /*.js Allow: /*.css Allow: /local/*.js Allow: /local/*.css Allow: /upload/*.png$ Allow: /upload/*.jpg$ Disallow: */index.php Disallow: /bitrix/ Disallow: /*?q= Disallow: /*show_include_exec_time= Disallow: /*show_page_exec_time= Disallow: /*show_sql_stat= Disallow: /*bitrix_include_areas= Disallow: /*clear_cache= Disallow: /*clear_cache_session= Disallow: /*ADD_TO_COMPARE_LIST Disallow: /*ORDER_BY Disallow: /*PAGEN Disallow: /*?print= Disallow: /*&print= Disallow: /*print_course= Disallow: /*?action= Disallow: /*&action= Disallow: /*register= Disallow: /*forgot_password= Disallow: /*change_password= Disallow: /*login= Disallow: /*logout= Disallow: /*auth= Disallow: /*backurl= Disallow: /*back_url= Disallow: /*BACKURL= Disallow: /*BACK_URL= Disallow: /*back_url_admin= Disallow: /*?utm_source= Disallow: /*?bxajaxid= Disallow: /*&bxajaxid= Disallow: /*?view_result= Disallow: /*&view_result= Disallow: /cgi-bin/ Disallow: /search/ Disallow: /user/ Disallow: /local/ Disallow: /login/ Disallow: /*?gpage Disallow: /*?action Disallow: /*& Disallow:*utm*= Disallow:*openstat= User-Agent: * Disallow: */index.php Disallow: /bitrix/ Disallow: /*?q= Disallow: /*show_include_exec_time= Disallow: /*show_page_exec_time= Disallow: /*show_sql_stat= Disallow: /*bitrix_include_areas= Disallow: /*clear_cache= Disallow: /*clear_cache_session= Disallow: /*ADD_TO_COMPARE_LIST Disallow: /*ORDER_BY Disallow: /*PAGEN Disallow: /*?print= Disallow: /*&print= Disallow: /*print_course= Disallow: /*?action= Disallow: /*&action= Disallow: /*register= Disallow: /*forgot_password= Disallow: /*change_password= Disallow: /*login= Disallow: /*logout= Disallow: /*auth= Disallow: /*backurl= Disallow: /*back_url= Disallow: /*BACKURL= Disallow: /*BACK_URL= Disallow: /*back_url_admin= Disallow: /*?utm_source= Disallow: /*?bxajaxid= Disallow: /*&bxajaxid= Disallow: /*?view_result= Disallow: /*&view_result= Disallow: /*?_openstat Disallow: /*?gclid= Disallow: /*?yclid= Disallow: /*?ymclid= Disallow: /*utm_source= Disallow: /*from=adwords Disallow: /*?cid= Disallow: /*?frommarket= Disallow: /*?gpage Disallow: /*?action Disallow: /*& Disallow:*utm*= Disallow:*openstat= Sitemap:https://site.ru/sitemap.xml
robots.txt для OpenCart
User-agent: Yandex Allow: /*.css Allow: /*.js Allow: /*.jpg Allow: /*.png Allow: /*.gif Disallow: /amp/ Disallow: /amp/*/ Disallow: /*route=product/amp_product Disallow: /index.php?route=product/amp_product Disallow: /index.php?route=product/amp_product&product_id= Disallow: /*route=account/ Disallow: /*route=affiliate/ Disallow: /*route=checkout/ Disallow: /*route=product/search Disallow: /index.php?route=product/product*&manufacturer_id= Disallow: /admin # Disallow: /catalog # если у вас настроены ЧПУ для категорий, то это правило нужно открыть Disallow: /system Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?order= Disallow: /*&order= Disallow: /*?limit= Disallow: /*&limit= Disallow: /*?filter_name= Disallow: /*&filter_name= Disallow: /*?filter_sub_category= Disallow: /*&filter_sub_category= Disallow: /*?filter_description= Disallow: /*&filter_description= Disallow: /*compare-products Disallow: /*search Disallow: /*cart Disallow: /*checkout Disallow: /*login Disallow: /*logout Disallow: /*vouchers Disallow: /*wishlist Disallow: /*my-account Disallow: /*order-history Disallow: /*newsletter Disallow: /*return-add Disallow: /*forgot-password Disallow: /*downloads Disallow: /*returns Disallow: /*transactions Disallow: /*create-account Disallow: /*recurring Disallow: /*address-book Disallow: /*reward-points Disallow: /*affiliate-forgot-password Disallow: /*create-affiliate-account Disallow: /*affiliate-login Disallow: /*affiliates Disallow: /*?page=- Allow: /catalog/view/javascript/ Allow: /catalog/view/theme/*/ Allow: /catalog/view/stylesheet/pdqo/ Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_type Clean-param: _openstat&gclid&yclid&ymclid&from&cid&frommarket User-agent: Googlebot Allow: /*.css Allow: /*.js Allow: /*.jpg Allow: /*.png Allow: /*.gif Disallow: /*route=account/ Disallow: /*route=affiliate/ Disallow: /*route=checkout/ Disallow: /*route=product/search Disallow: /index.php?route=product/product*&manufacturer_id= Disallow: /admin # Disallow: /catalog # если у вас настроены ЧПУ для категорий, то это правило нужно открыть Disallow: /system Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?order= Disallow: /*&order= Disallow: /*?limit= Disallow: /*&limit= Disallow: /*?filter_name= Disallow: /*&filter_name= Disallow: /*?filter_sub_category= Disallow: /*&filter_sub_category= Disallow: /*?filter_description= Disallow: /*&filter_description= Disallow: /*?tracking= Disallow: /*&tracking= Disallow: /*compare-products Disallow: /*search Disallow: /*cart Disallow: /*checkout Disallow: /*login Disallow: /*logout Disallow: /*vouchers Disallow: /*wishlist Disallow: /*my-account Disallow: /*order-history Disallow: /*newsletter Disallow: /*return-add Disallow: /*forgot-password Disallow: /*downloads Disallow: /*returns Disallow: /*transactions Disallow: /*create-account Disallow: /*recurring Disallow: /*address-book Disallow: /*reward-points Disallow: /*affiliate-forgot-password Disallow: /*create-affiliate-account Disallow: /*affiliate-login Disallow: /*affiliates Disallow: /*?page=- Allow: /catalog/view/javascript/ Allow: /catalog/view/theme/*/ Allow: /catalog/view/stylesheet/pdqo/ Disallow:*utm*= Disallow:*openstat= User-agent: * Disallow: /*route=account/ Disallow: /*route=affiliate/ Disallow: /*route=checkout/ Disallow: /*route=product/search Disallow: /index.php?route=product/product*&manufacturer_id= Disallow: /admin # Disallow: /catalog # если у вас настроены ЧПУ для категорий, то это правило нужно открыть Disallow: /system Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?order= Disallow: /*&order= Disallow: /*?limit= Disallow: /*&limit= Disallow: /*?filter_name= Disallow: /*&filter_name= Disallow: /*?filter_sub_category= Disallow: /*&filter_sub_category= Disallow: /*?filter_description= Disallow: /*&filter_description= Disallow: /*?tracking= Disallow: /*&tracking= Disallow: /*compare-products Disallow: /*search Disallow: /*cart Disallow: /*checkout Disallow: /*login Disallow: /*logout Disallow: /*vouchers Disallow: /*wishlist Disallow: /*my-account Disallow: /*order-history Disallow: /*newsletter Disallow: /*return-add Disallow: /*forgot-password Disallow: /*downloads Disallow: /*returns Disallow: /*transactions Disallow: /*create-account Disallow: /*recurring Disallow: /*address-book Disallow: /*reward-points Disallow: /*affiliate-forgot-password Disallow: /*create-affiliate-account Disallow: /*affiliate-login Disallow: /*affiliates Disallow: /*?page=- Allow: /catalog/view/javascript/ Allow: /catalog/view/theme/*/ Allow: /catalog/view/stylesheet/pdqo/ Disallow:*utm*= Disallow:*openstat= Sitemap:https://site.ru/sitemap.xml