{"id":10600,"date":"2024-10-16T00:34:12","date_gmt":"2024-10-16T00:34:12","guid":{"rendered":"https:\/\/lite14.net\/blog\/?p=10600"},"modified":"2024-10-16T00:34:12","modified_gmt":"2024-10-16T00:34:12","slug":"how-to-use-robots-txt-to-control-search-engine-crawling","status":"publish","type":"post","link":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/","title":{"rendered":"How to use robots.txt to control search engine crawling"},"content":{"rendered":"<p class=\"mb-2 last:mb-0\">Robots.txt is a text file that website owners can use to communicate with web crawlers and other web robots. It&#8217;s a simple, yet powerful way to control how search engines like Google, Bing, and others crawl and index your website&#8217;s content. By specifying rules in the robots.txt file, you can influence how crawlers interact with your website, which can have a significant impact on your website&#8217;s visibility, crawl rates, and overall search engine optimization (SEO).<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#How_robotstxt_works\" >How robots.txt works<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#Basic_syntax\" >Basic syntax<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#Best_practices\" >Best practices<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#Common_use_cases\" >Common use cases<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#Advanced_robotstxt_techniques\" >Advanced robots.txt techniques<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#Common_mistakes_to_avoid\" >Common mistakes to avoid<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"How_robotstxt_works\"><\/span><strong>How robots.txt works<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">When a search engine&#8217;s crawler (e.g., Googlebot) visits your website, it checks for the presence of a robots.txt file in the website&#8217;s root directory (e.g., <a>www.example.com\/robots.txt<\/a>). If the file exists, the crawler reads its contents to understand what it&#8217;s allowed to crawl and what it should avoid. This file is not a guarantee that crawlers will obey the rules, but it&#8217;s a widely adopted standard that most search engines respect.<\/p>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"Basic_syntax\"><\/span><strong>Basic syntax<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">The robots.txt file uses a simple syntax to specify rules for crawlers. Here&#8217;s a breakdown:<\/p>\n<ol>\n<li><strong>User -agent<\/strong>: specifies the crawler or robot to which the rule applies (e.g., Googlebot, Bingbot, etc.). You can use an asterisk (*) as a wildcard to apply the rule to all crawlers.<\/li>\n<li><strong>Disallow<\/strong>: specifies a URL or pattern that the crawler should not access. This can be a specific page, directory, or even a pattern (e.g., <code>\/private*<\/code>).<\/li>\n<li><strong>Allow<\/strong>: specifies a URL or pattern that the crawler is allowed to access (optional). This is useful when you want to override a previous Disallow rule.<\/li>\n<\/ol>\n<p class=\"mb-2 last:mb-0\"><strong>Examples<\/strong><\/p>\n<p class=\"mb-2 last:mb-0\">Here are some examples of robots.txt rules:<\/p>\n<ol>\n<li><strong>Disallow all crawlers from crawling a specific directory<\/strong>: <code>User -agent: * Disallow: \/private<\/code>\n<ul>\n<li>This rule tells all crawlers to avoid crawling the <code>\/private<\/code>directory and any subdirectories within it.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Allow Googlebot to crawl a specific directory<\/strong>: <code>User -agent: Googlebot Allow: \/public<\/code>\n<ul>\n<li>This rule allows Googlebot to crawl the <code>\/public<\/code> directory, even if there&#8217;s a previous Disallow rule that would otherwise block it.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Disallow all crawlers from crawling a specific page<\/strong>: <code>User -agent: * Disallow: \/private\/page.html<\/code>\n<ul>\n<li>This rule tells all crawlers to avoid crawling the specific page <code>\/private\/page.html<\/code>.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"Best_practices\"><\/span><strong>Best practices<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">When creating a robots.txt file, keep the following best practices in mind:<\/p>\n<ol>\n<li><strong>Keep it simple<\/strong>: use simple, clear rules to avoid confusion. Avoid using complex patterns or regex expressions that might be difficult for crawlers to interpret.<\/li>\n<li><strong>Test your rules<\/strong>: use tools like Google&#8217;s Robots.txt Tester to ensure your rules are working as intended. This can help you catch errors or unintended consequences.<\/li>\n<li><strong>Be mindful of crawl rates<\/strong>: don&#8217;t block crawlers entirely, as this can negatively impact your website&#8217;s visibility in search results. Instead, use robots.txt to control crawl rates and prioritize crawling of important pages.<\/li>\n<li><strong>Use robots.txt in conjunction with other SEO strategies<\/strong>: robots.txt is just one tool in your SEO toolkit. Make sure to combine it with other strategies, such as meta tags, header tags, and high-quality content, to maximize your website&#8217;s visibility.<\/li>\n<\/ol>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"Common_use_cases\"><\/span><strong>Common use cases<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">Robots.txt is commonly used to:<\/p>\n<ol>\n<li><strong>Protect sensitive content<\/strong>: prevent crawlers from accessing private or sensitive areas of your website, such as login pages, admin areas, or confidential data.<\/li>\n<li><strong>Prevent duplicate content issues<\/strong>: block crawlers from crawling duplicate or variant pages, which can help prevent duplicate content penalties and improve your website&#8217;s overall SEO.<\/li>\n<li><strong>Optimize crawl rates<\/strong>: control how frequently crawlers visit your website to prevent overload, reduce server load, and improve crawl efficiency.<\/li>\n<li><strong>Improve website performance<\/strong>: by blocking crawlers from crawling unnecessary pages or resources, you can reduce the load on your website and improve its overall performance.<\/li>\n<li><strong>Enhance user experience<\/strong>: by controlling what crawlers can access, you can ensure that users see the most relevant and up-to-date content, which can improve their overall experience on your website.<\/li>\n<\/ol>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"Advanced_robotstxt_techniques\"><\/span><strong>Advanced robots.txt techniques<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">While the basic syntax and examples above provide a solid foundation, there are some advanced techniques you can use to further customize your robots.txt file:<\/p>\n<ol>\n<li><strong>Using crawl delay<\/strong>: you can specify a crawl delay to control how frequently crawlers visit your website. For example, <code>Crawl-delay: 10<\/code> would tell crawlers to wait 10 seconds between requests.<\/li>\n<li><strong>Using sitemap directives<\/strong>: you can use robots.txt to specify sitemap locations, which can help crawlers discover new content and improve your website&#8217;s crawl coverage.<\/li>\n<li><strong>Using robots meta tags<\/strong>: you can use robots meta tags in conjunction with robots.txt to provide additional instructions to crawlers. For\u00a0example, <code>&lt;meta name=\"robots\" content=\"noindex, nofollow \"&gt;<\/code>would tell crawlers not to index a specific page and not to follow any links on that page.<\/li>\n<\/ol>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"Common_mistakes_to_avoid\"><\/span><strong>Common mistakes to avoid<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">When creating a robots.txt file, be sure to avoid the following common mistakes:<\/p>\n<ol>\n<li><strong>Blocking entire websites<\/strong>: avoid blocking entire websites or large sections of your website, as this can negatively impact your website&#8217;s visibility in search results.<\/li>\n<li><strong>Using overly broad rules<\/strong>: avoid using overly broad rules that might block crawlers from accessing important pages or resources.<\/li>\n<li><strong>Not testing your rules<\/strong>: failing to test your robots.txt rules can lead to unintended consequences, such as blocking crawlers from accessing important pages or resources.<\/li>\n<li><strong>Not keeping your robots.txt file up-to-date<\/strong>: failing to update your robots.txt file can lead to crawl errors, reduced crawl rates, and negatively impact your website&#8217;s SEO.<\/li>\n<\/ol>\n<h3 class=\"mb-2 last:mb-0\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span><strong>Conclusion<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"mb-2 last:mb-0\">Robots.txt is a powerful tool that can help you control how search engines crawl and index your website&#8217;s content. By understanding the basic syntax, best practices, and common use cases, you can create an effective robots.txt file that improves your website&#8217;s visibility, crawl rates, and overall SEO. Remember to keep your robots.txt file simple, test your rules, and avoid common mistakes to get the most out of this powerful tool.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt is a text file that website owners can use to communicate with web crawlers and other web robots. It&#8217;s a simple, yet powerful way&#8230;<\/p>\n","protected":false},"author":210,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[270],"tags":[],"class_list":["post-10600","post","type-post","status-publish","format-standard","hentry","category-digital-marketing"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to use robots.txt to control search engine crawling - Lite14 Tools &amp; Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to use robots.txt to control search engine crawling - Lite14 Tools &amp; Blog\" \/>\n<meta property=\"og:description\" content=\"Robots.txt is a text file that website owners can use to communicate with web crawlers and other web robots. It&#8217;s a simple, yet powerful way...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\" \/>\n<meta property=\"og:site_name\" content=\"Lite14 Tools &amp; Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-10-16T00:34:12+00:00\" \/>\n<meta name=\"author\" content=\"admin3\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin3\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\"},\"author\":{\"name\":\"admin3\",\"@id\":\"https:\/\/lite14.net\/blog\/#\/schema\/person\/3fcb0f4c8bd53264d6447eb6a6d36e81\"},\"headline\":\"How to use robots.txt to control search engine crawling\",\"datePublished\":\"2024-10-16T00:34:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\"},\"wordCount\":934,\"publisher\":{\"@id\":\"https:\/\/lite14.net\/blog\/#organization\"},\"articleSection\":[\"Digital Marketing\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\",\"url\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\",\"name\":\"How to use robots.txt to control search engine crawling - Lite14 Tools &amp; Blog\",\"isPartOf\":{\"@id\":\"https:\/\/lite14.net\/blog\/#website\"},\"datePublished\":\"2024-10-16T00:34:12+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/lite14.net\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to use robots.txt to control search engine crawling\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/lite14.net\/blog\/#website\",\"url\":\"https:\/\/lite14.net\/blog\/\",\"name\":\"Lite14 Tools &amp; Blog\",\"description\":\"Email Marketing Tools &amp; Digital Marketing Updates\",\"publisher\":{\"@id\":\"https:\/\/lite14.net\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/lite14.net\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/lite14.net\/blog\/#organization\",\"name\":\"Lite14 Tools &amp; Blog\",\"url\":\"https:\/\/lite14.net\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lite14.net\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/lite14.net\/blog\/wp-content\/uploads\/2025\/09\/cropped-lite-logo.png\",\"contentUrl\":\"https:\/\/lite14.net\/blog\/wp-content\/uploads\/2025\/09\/cropped-lite-logo.png\",\"width\":191,\"height\":178,\"caption\":\"Lite14 Tools &amp; Blog\"},\"image\":{\"@id\":\"https:\/\/lite14.net\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/lite14.net\/blog\/#\/schema\/person\/3fcb0f4c8bd53264d6447eb6a6d36e81\",\"name\":\"admin3\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lite14.net\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6a47a48d3f454a7aa919c16c5d03a6b63ff4ecdd1d2405bfba02f09414226574?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/6a47a48d3f454a7aa919c16c5d03a6b63ff4ecdd1d2405bfba02f09414226574?s=96&d=mm&r=g\",\"caption\":\"admin3\"},\"url\":\"https:\/\/lite14.net\/blog\/author\/adeyemoemmanuel001\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to use robots.txt to control search engine crawling - Lite14 Tools &amp; Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/","og_locale":"en_US","og_type":"article","og_title":"How to use robots.txt to control search engine crawling - Lite14 Tools &amp; Blog","og_description":"Robots.txt is a text file that website owners can use to communicate with web crawlers and other web robots. It&#8217;s a simple, yet powerful way...","og_url":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/","og_site_name":"Lite14 Tools &amp; Blog","article_published_time":"2024-10-16T00:34:12+00:00","author":"admin3","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin3","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#article","isPartOf":{"@id":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/"},"author":{"name":"admin3","@id":"https:\/\/lite14.net\/blog\/#\/schema\/person\/3fcb0f4c8bd53264d6447eb6a6d36e81"},"headline":"How to use robots.txt to control search engine crawling","datePublished":"2024-10-16T00:34:12+00:00","mainEntityOfPage":{"@id":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/"},"wordCount":934,"publisher":{"@id":"https:\/\/lite14.net\/blog\/#organization"},"articleSection":["Digital Marketing"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/","url":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/","name":"How to use robots.txt to control search engine crawling - Lite14 Tools &amp; Blog","isPartOf":{"@id":"https:\/\/lite14.net\/blog\/#website"},"datePublished":"2024-10-16T00:34:12+00:00","breadcrumb":{"@id":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/lite14.net\/blog\/2024\/10\/16\/how-to-use-robots-txt-to-control-search-engine-crawling\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/lite14.net\/blog\/"},{"@type":"ListItem","position":2,"name":"How to use robots.txt to control search engine crawling"}]},{"@type":"WebSite","@id":"https:\/\/lite14.net\/blog\/#website","url":"https:\/\/lite14.net\/blog\/","name":"Lite14 Tools &amp; Blog","description":"Email Marketing Tools &amp; Digital Marketing Updates","publisher":{"@id":"https:\/\/lite14.net\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/lite14.net\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/lite14.net\/blog\/#organization","name":"Lite14 Tools &amp; Blog","url":"https:\/\/lite14.net\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lite14.net\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/lite14.net\/blog\/wp-content\/uploads\/2025\/09\/cropped-lite-logo.png","contentUrl":"https:\/\/lite14.net\/blog\/wp-content\/uploads\/2025\/09\/cropped-lite-logo.png","width":191,"height":178,"caption":"Lite14 Tools &amp; Blog"},"image":{"@id":"https:\/\/lite14.net\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/lite14.net\/blog\/#\/schema\/person\/3fcb0f4c8bd53264d6447eb6a6d36e81","name":"admin3","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lite14.net\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/6a47a48d3f454a7aa919c16c5d03a6b63ff4ecdd1d2405bfba02f09414226574?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/6a47a48d3f454a7aa919c16c5d03a6b63ff4ecdd1d2405bfba02f09414226574?s=96&d=mm&r=g","caption":"admin3"},"url":"https:\/\/lite14.net\/blog\/author\/adeyemoemmanuel001\/"}]}},"_links":{"self":[{"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/posts\/10600","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/users\/210"}],"replies":[{"embeddable":true,"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/comments?post=10600"}],"version-history":[{"count":1,"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/posts\/10600\/revisions"}],"predecessor-version":[{"id":10601,"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/posts\/10600\/revisions\/10601"}],"wp:attachment":[{"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/media?parent=10600"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/categories?post=10600"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lite14.net\/blog\/wp-json\/wp\/v2\/tags?post=10600"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}