12 Ways Webmasters Create Duplicate Content

[pow-wow]

Here are 12 ways people unintentionally create dupe content:

1) Build a site for the sole purpose of promoting affiliate offers, and use the canned text supplied by the agency managing the affiliate program.

2) Generate lots of pages with little unique text. Weak directory sites could be an example of this.

3) Use a CMS that allows multiple URLs to refer to the same content. For example, do you have a dynamic site where E-Commerce Hosting pulls up the exact same content as E-Commerce Hosting If so, you have duplicate content. This is made worse if your site actually refers to these pages using multiple methods. A surprising number of large sites do this.

4) Use a CMS that resolves sub domains to your main domain. As with the prior point, a surprising number of large sites have this problem as well.

5) Generate pages that differ only by simple word substitutions. The classic example of this is to generate pages for blue widgets for each state where the only difference between the pages is a simple word substitution (e.g. Alabama Blue Widgets, Arizona Blue Widgets, …).

6) Forget to implement a canonical redirect. For example, not 301 redirecting .com to .com (or vice versa) for all the pages on your site. Regardless of which form you pick to be the preferred form of URL for your site, someone out there will link to the other form, so implementing the 301 redirect will eliminate that duplicate content problem for you, as well as consolidate all the page rank from your inbound links.

7) Having your on site links back to your home page link to .com/index.html (or index.htm, or index.shtml, or …). Since most of the rest of the world will link to .com, you now have created duplicate content, and divided your page rank, if you have done this.

8) Implement printer pages, but not using robots.txt to keep them from being crawled.

9) Implement archive pages, but not using robots.txt to keep them from being crawled.

10) Using Session ID parameters on your URLs. This means every time the crawler comes to your site it thinks it is seeing different pages.

11) Implement parameters on your URLs for other tracking related purposes. One of the most popular is to implement an affiliate program. The search engine will see .com?affid=1234 as a duplicate of .com. This is made worse if you leave the “affid” on the URL throughout the user’s visit to your site. A better solution is to remove the ID when they arrive at the site, after storing the affiliate information in a cookie. Note that I have seen a case where an affiliate had a strong enough site that .com?affid=1234 started showing up in the search engines rather than .com (NOT good).

12) Implement a site where parameters on URLs are ignored. If you, or someone else, links to your site with a parameter on the URL, it will look like dupe content.
There are many ways that people intentionally create duplicate content, by various scraping techniques, but there is no need to cover that here.

Source: Ramblings About SEO Blog Archive 12 Ways Webmasters Create Duplicate Content

[Gorkfu]

This is a great list! I had something similar to number 11 happen in the past. I just want to share the issue in case anyone ever has the same problem.

Google crawled a new site I created, that I had yet to link to from anywhere. They were able to crawl it because my host automatically had directory list on as default in apache. I turned it off of course and I recommend anyone running apache do the same. You can do it by adding this to a .htaccess file.
Options -Indexes

Google ended up making like 10 links like this: sitename.com/?id=3 etc. The only way I found Google would get rid of it from their index was by making disallows in my robots.txt for each link. Deleting the URLs in Google's webmaster tools will simply not work.

[geme]

nice article pow-wow... keep up the good work on this forum!

[tb987]

This is a great post - I have made and still continue to make all/some of these mistakes.

There is a 14th, if you use url rewrites make sure you exclude the none rewritten URLS and redirect them back to the friendly ones. I have noticed a site with /index.php?ref=blah in the index but it should be /blah/

[Tom_Sean]

This is a great list pow-wow! Thanks a lot. It's very useful.

[dyamni]

Originally Posted by tb987

This is a great post - I have made and still continue to make all/some of these mistakes.

There is a 14th, if you use url rewrites make sure you exclude the none rewritten URLS and redirect them back to the friendly ones. I have noticed a site with /index.php?ref=blah in the index but it should be /blah/

Hi

I agree with you. This point must be included in this. I aslo noticed these types of sites.

thanks

[happyman]

Good articles.Thank for sharing.