robots.txt
Embed This Widget
Add the script tag and a data attribute to embed this widget.
Embed via iframe for maximum compatibility.
<iframe src="https://tldfyi.com/iframe/glossary/robots-txt/" width="420" height="400" frameborder="0" style="border:0;border-radius:10px;max-width:100%" loading="lazy"></iframe>Paste this URL in WordPress, Medium, or any oEmbed-compatible platform.
https://tldfyi.com/glossary/robots-txt/Add a dynamic SVG badge to your README or docs.
[](https://tldfyi.com/glossary/robots-txt/)Use the native HTML custom element.
robots.txt是放置在域名根目录（如example.com/robots.txt）的纯文本文件，告知合规的网络爬虫网站哪些部分可以访问，哪些不能访问。它使用机器人排除协议指令，如User-agent、Disallow和Allow。通过robots.txt屏蔽某些部分可以为重要页面节省爬取预算，但被屏蔽的页面即使有[[backlinks|反向链接]]也无法被收录——过度屏蔽是常见的SEO失误。该文件还应声明[[sitemap-domain|XML网站地图]]的位置以便爬虫发现。
示例
Adding 'Disallow: /admin/' to robots.txt prevents Google from crawling internal admin pages, while 'Sitemap: https://example.com/sitemap.xml' advertises the sitemap location to all crawlers.