Also known as search robots, spiders, crawlers, and bots, these web tools systematically browse the internet in order to index sites for search engines primarily. Maintaining search engine listings is vital to the success of any website. Search engine links are gathered by various search engine providers (Google, Bing) by way of sending these ‘bots’ to your website to crawl each of the pages and report back to the search engine provider about the available content.

These bots are doing important work – but they can consume quite a lot of bandwidth when left unmanaged. If the timing of these crawls coincides with a busy site (high activity level), this activity can result in degraded performance, making it difficult for visitors to browse the site and place orders. We’re seeing an overwhelming amount of traffic from these bots which is directly contributing to degraded performance on stores, so we’d like your help to manage the traffic and keep your site as responsive as possible.

Two of the most important crawlers to mitigate, which allow you some control over their actions, are Google and Bing. Reducing the speed of their crawls will help prevent the activity from overwhelming the site, and allow the bandwidth to be better ‘shared’ with your visitor traffic without negatively impacting page load speeds while a crawl is in progress.

For Google, you can sign in or sign up for Google Search Console :
- Follow the instructions in this article to reduce the Google crawl rate to a minimum.

For Bing, you can sign in or sign up for Bing webmaster tools :   
- Follow the instructions in this article to manage the timing and the speed of Bing crawls.