Print this page

How to index specific pages on a site.com site for search engines

Knowledge Article Number 000159800
Description You can index specific pages on a Site.com site for search engines by adding the robots.txt file.
Resolution

Simply upload a robots.txt file as an asset to your Site.com site. 
As long as robots.txt exists on the root directory, crawlers that check for it will find it.

By default, crawlers will try to index all the pages in a site. You can specify exclusions in the robots.txt file.

For more information on the protocol used in the robots.txt file, please review the Wikipedia page on the Robots Exclusion Standard





promote demote