How The Bing Toolbar May Accidentally Submit Private Pages & Ads

Microsoft has confirmed that they do discover and index URLs that they find through users surfing the Internet with the Bing Toolbar installed. This can be a concern for those who know certain URLs are only accessible if the individual knows about it and navigates to the URL with the Bing Toolbar installed.

For example, YouTube has a setting where you can share a video privately with someone else by giving them a special URL. If the user they are sharing that URL with has the Bing Toolbar installed, Bing can technically discover the URL, assuming the page is not blocked via the NOINDEX meta tag or other means.

Duane Forrester, a Senior Product Manager at Microsoft, told us:

Yes, as with some other toolbars, the Bing toolbar (when permitted by the user) may record the open (not https) web sites that a user visited to add to our knowledge of the internet in order to improve the search results we provide to users. This in turn (if the site permits) may indeed lead to our crawling links we have discovered. As noted, any site can control the indexing of pages via the NOINDEX meta tag.

Recently one Google advertiser complained that Bing had discovered, crawled and indexed their AdWords tracking URLs. This caused several of their special ad URLs to be listed in Bing, leading to inflated AdWords clicks and conversions.

Duane said, “for the case of Advertisers, they can (and often do) notify crawlers not to crawl their click tracking URLs by using robots.txt disallow rules, Bingbot honors these rules and will not index the URL nor follow the redirection.”

To some, this might not come as a surprise. Remember our story, Google: Bing Is Cheating, Copying Our Search Results? Bing was discovering data within Google’s search queries through their toolbar. So one might assume that Bing also used the Bing Toolbar to discover other URLs.

It is important for webmasters and users to know, Bing may use the toolbar to discover URLs that one might not expect them to discover. Such URLs include YouTube private videos, specialized ad URLs that you do not want indexed, documents on private servers and many other cases.