New bot for Bing Search Crawler

Bing now has a new crawler. The new Bing crawler user agent string listed as:

msnbot/2.0b (+http://search.msn.com/msnbot.htm).

Note that the bot’s name for Bing didn’t change. You’ll also see the previous version of the bot as well:

msnbot/1.1 (+http://search.msn.com/msnbot.htm)

— which is still in use, as of today, as Bing’s primary user agent.

Here are some Question and Answers that probably can help you!

Q: Bing is crawling my site, so why is it not indexing my pages?

A: Make sure your content is viewable by MSNbot.

• Ensure there are text navigation links if main navigation is pictorial-based
• Ensure there is plenty of indexable content outside of <script> </script>tags
• Make sure each page has unique <title> and <meta> description tags
• If possible, avoid frames and meta tag refresh
• If your site uses session IDs or cookies, make sure they are not needed for crawling
• Use friendly URLs whenever possible
• Review our Webmaster Center blog post, I’m not ranking in Bing, what can I do?

Q: My site is moving. How do I set up a redirect?

A: In most instances, you will want to use a 301 redirect . There are many different ways to institute these redirects, based on your preference and the technologies your site is using.

Q: Why isn’t Bing crawling or is only partially crawling my site?

A: Depth of crawl is based on a lot of factors, including how your site ranks and the architecture of your site.  However, here are some steps you can take to help us crawl your site more effectively:

• Verify that you are not blocked and there are no site crawling errors in Webmaster Center tools
• Make sure you are not blocking access to the Bing web crawler, MSNbot, in your Robots.txt file
• Make sure you are not blocking MSNbot’s IP range
• Ensure you have no site navigation issues that might prevent MSNbot from crawling your site:
  • Invalid page coding (HTML, XHTML, etc.)

Note: You can validate your page source code at www.w3c.org.

• • Site navigation links that are only accessible by JavaScript
  • Password protected pages
• Submit a sitemap in the Sitemaps tool of Webmaster Center tools and provide the path to the sitemap.xml file(s) in a Robots.txt file