Mobile site indexing in a nutshell, Optimal Mobile Site Indexing

by Vrushali Dandekar on 09-21-2018 in Web Design, Development, Mobile Marketing

Optimal Mobile Site Indexing

It’s probably a sign of the times, but lately I have been fielding a lot of questions regarding SEO for mobile websites. Because there are so many ways to build a mobile site and because there are such a variety of mobile devices, one area that really piques my interest is mobile site indexing.

There are several tips and varied opinions on mobile SEO out there, and until recently, neither Google nor Bing have been very clear about mobile indexing best practices, especially regarding mobile indexing and smartphones. My goal is to offer up some of the most current best practices and research from the world of mobile SEO.

Judging by some of the forums and discussions out there, this topic is anything but resolved, and there are some heated opinions. Please let me know in the comments if I have missed any techniques and/or tips that have worked for you.First, we need to discuss how mobile sites are served to mobile searchers. One of the most popular methods is to create a user-agent detection at the server level.

If the user-agent is a desktop computer, serve the desktop version of your site. If the user-agent identifies itself as coming from a phone, a tablet, an iPad or anything that is mobile, serve that user the mobile version of your site.

I know what you’re thinking: Isn’t that cloaking? Nope. Don’t believe me? Just ask Matt Cutts. In this video, he’s pretty clear about it not being a cloaking issue (read: if you do it correctly).

The most common way of implementing user-agent detection is to place the mobile version of your website on a subdomain such as m.domain.com. Whenever a mobile searcher is directed to one of the pages from www.domain.com, a server-side 301 redirect will send them to the corresponding page at m.domain.com. (By the way, in the video I referenced earlier, Matt Cutts says that it’s a good idea to use this type of setup.)

Now, that doesn’t mean that you can’t do it other ways. Here is a list of other options I have seen companies use for mobile websites:

Subdomains :

m.domain.com

mobile.domain.com

touch.domain.com

wap.domain.com

Other TLDs :

domain.mobi

domainmobile.com

Directories

www.domain.com/m

www.domain.com/mobile

Websites detects the user-agent and if it matches a mobile user-agent, the visitor will see a mobile-themed version of the site – but the URLs don’t change. The URLs are the same for desktop visitors and mobile visitors, and at the bottom of each page, the visitor can choose between viewing the regular version of the site or the mobile version.

This strategy is also great for indexing and PageRank, as there are no other unique URLs for the mobile version of the site. 

The only thing you need to be clear on is which method is best for you, in terms of technical feasibility and even in terms of analytics and tracking. There is no *right* way to do it, and there are pros and cons for each method. Pick the one that works best for your company’s situation, and don’t look back.

The next question is: How do you let Google and Bing know about the mobile version of your site?

Verify Your Mobile Site In Google & Bing Webmaster Tools

This works for mobile sites on their own TLD or subdomain, and it’s very helpful for getting data specifically about the mobile site.

Build & Submit A Sitemap Of Your Mobile Website’s URLs To Google & Bing

Google has specific criteria for mobile sitemaps:

<?xml version="1.0" encoding="UTF-8" ?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0">
<url>
<loc>http://mobile.example.com/article100.html</loc>
<mobile:mobile/>
</url>
</urlset>

The only real distinction here is the addition of the tag. Otherwise it is very similar to a typical XML sitemap. For more information on mobile sitemaps, please check out this post from Google’s WebmasterCentral blog.

Mobile Searchbots & Robots.txt File Considerations

Some webmasters and SEOs are noticing their desktop sites URLs in the mobile SERPs and their mobile URLs in the desktop SERPs. It is not happening all the time, and on the mobile side of things, it’s pretty standard nowadays.

However, for SEOs who want complete control over user-experience and/or branding, these blended search results are a headache, especially if they don’t have user-agent detection and a redirect configured.

Some of these SEOs go as far as blocking Googlebot from their mobile site and blocking Googlebot-Mobile from their desktop site. Here’s how that looks in the robots.txt files:

Desktop site: http://www.domain.com/robots.txt

User-agent: Googlebot
User-agent: Slurp
User-agent: bingbot
Allow: /

User-agent: Googlebot-Mobile
User-Agent: YahooSeeker/M1A1-R2D2
User-Agent: MSNBOT_Mobile
Disallow: /

Mobile site: http://m.domain.com/robots.txt

User-agent: Googlebot
User-agent: Slurp
User-agent: bingbot
Disallow: /

User-agent: Googlebot-Mobile
User-Agent: YahooSeeker/M1A1-R2D2
User-Agent: MSNBOT_Mobile
Allow: /

First off, how many of you knew that each search engine has its own mobile bot? Pretty cool, right?

To be clear, I do not recommend using this strategy of restricting bots. It is important that Googlebot has access to your mobile site and that Googlebot-Mobile has access to your main site.

My main reasoning: it’s probably not a good idea to block Googlebot from anything that you want indexed, even if it is a mobile site and even if the mobile SERPs are determined by Googlebot-Mobile. And it’s definitely not a good idea to block Googlebot-Mobile from your main site, as desktop URLs are still showing up a lot for searches on mobile devices.

Furthermore, trying to tell Google how to do their job (i.e. which results to show for any given search in any given setting) is a little arrogant, and ultimately it’s a waste of time.It reminds me of back in the day when people used a meta tag to try to tell Google when to re-visit and/or re-index a URL. Yeah. Good luck with that.
Also, if you block Googlebot, get ready for an ‘unhealthy site status’ message in your Google Webmaster Central profile.

Finally, I’d like to point out two great quotes on this topic:

Googlebot, our crawler, must crawl your site before it can be included in our search index. – Jun Mukai, Software Engineer, Mobile Search Team at Google (source)

Don’t block the mobile site from being returned in desktop or smartphone results, as it’s relevant to more than just the search engine’s mobile index. – Bryson Meunier (source)

Recent News Regarding Mobile Searchbots

Google recently announced additional smartphone user-agent strings for Googlebot-Mobile:

Feature phones Googlebot-Mobile:

  • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Smartphone Googlebot-Mobile:

  • Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

It appears Google is aiming to improve user experience on mobile search. I am very happy to see them doing something like this because there are a lot of mobile search marketers out there with specific needs and questions regarding mobile SEO, and some of the more specific issues are still a little cloudy. I don’t think this is the end of the discussion. In fact, in many ways, it is just the beginning.

Link To Your Mobile Site From Your Main Site & Vice VersaFor my penultimate mobile site indexing tip, I’m going to go old school on you. If you want URLs from a subdomain indexed, link to them from the main site. Linking between your main site and your mobile site will help with the crawling and indexing of the mobile site.

Begin A Link Building Campaign…But For Your Mobile SiteBecause Google dominates market share for mobile search and because Google makes links such an important factor in their search algorithms, it’s a good idea to get external links to your mobile site. I’d start with mobile directories. Bryson Meunier has a list of mobile directories that will help get you started.

In your mobile SEO journey, be sure to check to see if your mobile site is being indexed and/or showing up in Google Mobile Search. I like to check to see what types of results are showing up for various search keywords and phrases at http://www.google.com/m/.

It’s a good idea to get familiar with the layout of mobile SERPs, and it will give you the chance to see how Google displays your site.

Please share your thoughts and questions in the comments section.

Log in to post a comment