Google has introduced some changes to Webmaster Tools – in particular, handling of URLs with parameters.
“URL Parameters helps you control which URLs on your site should be crawled by Googlebot, depending on the parameters that appear in these URLs,” explains Kamila Primke, Software Engineer with the Google Webmaster Tools Team. “This functionality provides a simple way to prevent crawling duplicate content on your site. Now, your site can be crawled more effectively, reducing your bandwidth usage and likely allowing more unique content from your site to be indexed. If you suspect that Googlebot’s crawl coverage of the content on your site could be improved, using this feature can be a good idea. But with great power comes great responsibility! You should only use this feature if you’re sure about the behavior of URL parameters on your site. Otherwise you might mistakenly prevent some URLs from being crawled, making their content no longer accessible to Googlebot.”
Google is now letting users describe the behavior of parameters. For example, you can let Google know if a parameter changes the actual content of the page.
“If the parameter doesn’t affect the page’s content then your work is done; Googlebot will choose URLs with a representative value of this parameter and will crawl the URLs with this value,” says Primke. “Since the parameter doesn’t change the content, any value chosen is equally good. However, if the parameter does change the content of a page, you can now assign one of four possible ways for Google to crawl URLs with this parameter.”
Those would be: let Googlebot decide, every URL, only crawl URLS with value or no URLs.
Users can tell Google if a parameter sorts, paginates, determines content, or other things that it might do. For each parameter, Google will also “try” to show you a sample of example URLs from your site that it has already crawled that contain a given parameter.
To bring up the use of caution again, Primke warns about the responsibilities that come with using the No URLs option. “This option is the most restrictive and, for any given URL, takes precedence over settings of other parameters in that URL. This means that if the URL contains a parameter that is set to the ‘No URLs’ option, this URL will never be crawled, even if other parameters in the URL are set to ‘Every URL.’ You should be careful when using this option. The second most restrictive setting is ‘Only URLs with value=x.’”
She runs through some examples in this blog post, and there is more related information in Google’s Webmaster Help forum.
Be Careful About Selling the Same Stuff From Multiple Domains
As long as we’re discussing webmaster issues for Google, I’ll also point to the latest Webmaster Help video from Matt Cutts, who discusses selling products on multiple domains. The user question he sought to answer was:
“I manage 3 websites that sell the same products across 3 domains. Each site has a different selling approach, price structure, target audience, etc. Does Google see this as spumy or black hat?”
Cutts says, “On one hand, if the domains are radically different lay-out, different selling approach, different structure – like, essentially completely different, and especially the fact that you said it’s only 3 domains, that might not be so bad. Clearly if it were 300 domains or 3,000 domains – you can quickly get to a fairly large number of domains that can be crowding up the search results and creating a bad user experience…by the time you get to a relatively medium-sized number of sites.”
“The thing that was interesting about the question is that you said it’s the same products, as in identical. So it’s a little weird if you’re selling identical products across 3 domains. If you were selling like men’s sweaters on one, and women’s sweaters on another, and shoes on a third….I’ve said before, there’s no problem with having different domains for each product, and a small number of domains (2, 3, or 4) for very normally separable reasons can make perfect sense, but it is a little strange to sell the same products, so if they’re really identical, that starts to look a little bit strange – especially if you start to get more than 3 domains.”
“Definitely, I have found that if you have one domain, you’ve got the time to build it up – to build the reputation for that domain…in my experience, when someone has 50 or 100 domains, they tend not to put as much work – as much love into each individual domain, and whether they intend to or not, that tends to show after a while. People have the temptation to auto-generate content or they just try to syndicate a bunch of feeds, and then you land on one domain vs. another domain, and it really looks incredibly cookie cutter – comparing the two domains, and that’s when users start to complain.
