Having a robots.txt is extremely important for your XenForo forum. It tells search engines which pages to crawl and which pages to skip. This is your chance to build out a proper structure for the robots.txt file so that it will exclude non-important pages.
When you install XenForo onto your hosting server, robots.txt is not be included in the package. However, creating one takes less than 30 seconds.
Here is our recommended robots.txt file for XenForo.
User-agent: * Disallow: /find-new/ Disallow: /account/ Disallow: /login/ Disallow: /admin.php Disallow: /search/ Disallow: /whats-new/ Allow: / Sitemap: https://domain.com/sitemap.xml
We also recommend including a sitemap location of your XenForo forum in the robots.txt file. This helps search engines to locate the contents of your site.
Robots.txt and NoIndex Rule
One misconception that we see often is the use of robots.txt for noindexing of content.
Disallowing search engines to crawl a specific directory on your site does not mean search engines will not index it. The best practice for this case is to use the meta robot tags.
For example, a forum owner would like Google not to index any of their XenForo member’s pages. This can be achieved by placing the following meta tag right before the closing
</head> tag inside of the member_view template.
<xf:head option="metaNoindex"><meta name="robots" content="noindex" /></xf:head>
If you are not a technical person, you can use our XenForo add-on to automate this process. It also provides additional control over which pages on your forum search engines can index and follow.