Best Practices For Configuring Robots.txt in B2C Commerce

You can edit the robots.txt file in your B2C Commerce Production environment by logging into Business Manager and going to ​Merchant Tools > SEO > Robots​.

We recommend using a  custom robots.txt definition, as with this option you can include any desired definition in the text field.

• Add an asterisk to User-Agent to identify a wild card, meaning that these instructions are applicable to all spiders.
• Add individual disallow statements to instruct search engines not to index specific pages. Choose to disallow account, cart or other pages that don't have much value in being indexed by search engines.
• Reference the location of the sitemap here. Below is a sample Robots.txt definition, but you can create your own:

User-Agent: *
Disallow: /account*
Disallow: /cart*
Disallow: /checkout*
Disallow: /order-history *
Disallow: /search*
Disallow: /wishlist*

Sitemap: http://www.yoursite.com/sitemap_index.xml 

In order to prevent crawlers from discovering these pages, URLs containing search refinements should be disallowed. See below:

• Disallow: /*pmin*
• Disallow: /*pmax*
• Disallow: /*prefn1*
• Disallow: /*prefn2*
• Disallow: /*prefn3*
• Disallow: /*prefn4*
• Disallow: /*prefv1*
• Disallow: /*prefv2*
• Disallow: /*prefv3*
• Disallow: /*prefv4*