Drop In Bing? Moving To HTTPS (SNI) Would Have Caused It

Posted On 17 Dec 2020

Comment: Off

Many webmasters have been pressured to switch from HTTP to HTTPS, but some are finding that this is resulting in a loss of traffic from Bing. Columnist Ben Goodsell explains the issue and offers solutions.

Last August, Google announced they’d be giving a small ranking boost for serving secured (HTTPS) content, in an effort to “encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web.” As more and more sites migrate, Bing seems to have had trouble keeping up with uprising security methods and standards.

Bing’s Trouble Crawling TLS SNI

If your site has moved to HTTPS using an extension of TLS called SNI, there’s a good chance you’ve seen a drop in Bing crawling and rankings. This is due to, prior to this month, Bing’s inability to crawl sites with this specific implementation of site security. In fact, they’ve only been able to manually “whitelist” sites with this type of implementation. Unfortunately, this issue seems to prevent the transfer of equity to HTTPS URLs, even with proper one to one 301 redirects.

As shown below, the end result is a severe drop in crawling, ranking, and visits from Bing search results. To the left you can see a rank based traffic estimation from SEMrush, and to the right what can happen to the number of pages crawled in Bing’s Reports & Data report.

So, What Is SNI Again?

SNI stands for Server Name Indication, and it helps companies save money by allowing one IP to handle the security certificates for multiple HTTPS sites.

F5 has an excellent writeup on the technical details. In essence, it explains how one IP is needed to identify the security certificate needed for the requests of multiple sites using SNI. This prevents the need to have individual IP’s for each secure site, as shown below. One can imagine the benefit of only needing one VS several IPs and the costs associated with them.

Bing Webmaster Support

Bing’s support on the issue seems to have been a frustration within the webmaster community for some time. In this thread, after over a month, Bing finally responded with the following.

We would like to inform you that as of today, Bing only supports a limited number of SNI websites through a whitelist. We understand that more and more websites are switching to SNI so we plan to expand our support for all SNI hosts in the next 6 months.

As a short term mitigation, we can whitelist about 10 – 20 SNI hosts for you, if needed.

Even more recently, Bing has stated that the issue has been resolved, but that a fix has still yet to be applied to the XML Sitemap submission tool. It’s recommended to submit XML Sitemaps from an HTTP host.

The SNI issues were not specific to this site. Also, they have already been solved for Bingbot at large…

…That said, sitemap fetcher still is using our legacy MSNBot but will be updated soon. Until then SNI sites could choose to submit Sitemaps from non-SNI endpoints.

What Can You Do?

First, if you or your client moved to HTTPS using SNI in the last 6 months, take a good look at Reports & Data in Bing Webmaster Tools. This is available for free for anyone who verifies site ownership with Bing.

This gives insight into Bing’s backend, and shows the number of pages crawled overtime as shown below.

Second, compare before and after the move to help gain insight into any potential loss. To make this easy for Google Analytics users, I’ve created this Custom Report for easy import. Don’t forget, keyword data is still available for Bing, so be sure to create a secondary dimension to get landing page and keyword level data!

As you can see in this image, it’s clear Bing Organic visits was no longer maintaining previous levels after November 24th. Being armed with this information you can make a clear argument about the cause of the issue. Scrolling further into the report gives even more detail into which landing pages and keyword combinations were effected the most.

Final Takeaways

As mentioned previously, Bing has stated that a fix for this issue and has been pushed live, but we have yet to confirm.

Always be sure to submit perfect, error free, XML Sitemaps to Bing. For sites using SNI these need to be hosted on the non-HTTPS version of the site. At least until the XML Sitemap submission tool is fixed.

Keep an eye out for trouble using a combination of the Reports & Data report in Bing Webmaster Tools and Bing Organic traffic in your analytics platform.

Finally, as a last resort, the Bing Webmaster team can be emailed by using this form.