Duplicate Pages and Search Engines

The buzz on duplicate content penalties is almost deafening. Some people think it's a myth while others strongly believe that search engines are out to hunt down these so-called posers and give them the worst punishment possible. Regardless of their accurate definition, duplicate content penalties do occur. The bottom line is that search engines aren't big fans of duplicate content at all, so why even have it on your website?

The last thing any search engine would want is to give its users an unsatisfying search experience. They are doing everything in their power to provide optimum search results. By constantly improving their algorithms and filtering duplicate content, they are presenting their users with the most relevant and unique listings for search results. This is the main reason you use search engines in the first place. For them to work to your advantage as a website owner or blogger, you will need high-quality content that is both unique and informative. This way, search engine results related to your niche pull up your page as a primary valid listing.

How do search engines deal with duplicate content exactly, you ask? Google, for instance, uses a supplemental index found within its database that acts as a filtering mechanism. Basically, it weeds out websites and blogs that have duplicate content. They use spiders called Googlebots to collect and analyze similar content found in different web pages. They select a few of these web pages and present them in related searches. Meanwhile, those that are disregarded are placed in Google's supplemental index. This doesn't mean your site is thrown into the void, never to be found again; it is merely positioned at the end of search listings, which makes it almost impossible for search engine users to stumble upon your site.

Duplicate content doesn't do you or your site any good at all. You want significant traffic to pour into your site. The best solution to boost traffic for your site with SEO is to create original content. Writing unique content to your readers is like coming up with a remedy for a particular disease. People are always looking for something that would satisfy their curiosity, but if you give them information that they've already been hearing a thousand times over, then you are not really offering anything new to the table. A good website or blog thrives on well-written and originative content -- that is a fact. By providing original content, you are giving search engine users a pretty good reason to visit your site.

It isn't easy to come up with purely original content all the time. You do your best to write original content, but sometimes it still isn't enough. The good news is that there are tools available for you to maximize your original text output. The best of the lot, I would say, is a duplicate content checker. This tried-and-tested tool analyzes and checks your articles for duplicate texts. A duplicate content checker basically goes over your own material, checks it against other available web content, and hits you with a red flag if matching texts are detected.

All in all, without original content, your site could just be as good as invisible. Be seen and be a valuable source of online content. Write unique copies and use a duplicate content checker every chance you get. By doing so, you're sure to get some Google-love and, ultimately, a decent amount of traffic into your site.

Introduction

Search engines have become the gateway to information in the Internet. Search engines are so important that websites find that they need to rank well in search engine results pages (SERPs) in order to get noticed. With the numerous websites vying to get into the coveted position of the top 30 results listed in SERPs more and more website owners are using search engine optimization (SEO) techniques to improve their rankings. People who use SEO know that there are certain factors that can affect your ranking positively and of course negatively. Of the negative factors one of the most well-known is duplicate content.

Search engines are biased against duplicate content. As a matter of fact some sites do not get listed in SERPs because of this factor. This happens when crawlers do not index sites which they have previously determined to be a duplicate site of another site. The crawlers skip the duplicate site to be more efficient and save time. Crawler also do this for another reason – to avoid listing duplicate pages in SERPs and thus point users to different sites containing just the same information. Search engines do not like that to happen because it would be irritating for users who expect to see different sites for the different links they click. For similar sites, search engines also usually just list one of the sites and relegate the others under a link that says See related pages. For those that get manage to be listed in the SERPs the page rank is still usually affected and so affects the sites standing.

Where Search Engines See Duplicate Content

So where do crawlers see this duplicate content. And what are the possible content that they would interpret as duplicate? According to an article by William Slawski on Duplicate Content Issues and Search Engines, search engines see duplicate content from the following kind of web pages:

1. Product descriptions from manufacturers, publishers, and producers reproduced by a number of different distributors in large ecommerce sites.

2. Alternative print pages – This happens when website owners who are user friendly offer copies of the same documents in different formats for a varied printing options. Although helpful to users it might actually indexed by crawlers as duplicate pages.

3. Pages that reproduce syndicated RSS feeds through a server side script.

4. Canonicalization issues, where a search engine may see the same page as different pages with different URLs.

5. Pages that serve session IDs to search engines, so that they try to crawl and index the same page under different URLs.

6. Pages that serve multiple data variables through URLs, so that they crawl and index the same page under different URLs.

7. Pages that share too many common elements, or where those are very similar from one page to another, including title, meta descriptions, headings, navigation, and text that is shared globally. – This is common for company websites that insist on having their logo, description, etc put on every page of their website.

8. Copyright infringement – Plagiarism is of course a good reason for not being indexed. The problem is that crawlers cannot distinguish the original from the duplicate and might mistakenly filter out the original instead.

9. Use of the same or very similar pages on different subdomains or different country top level domains (TLDs).

10. Article syndication – Some writer allow their articles to be published in other websites as long as they are given credit for their work. The problem arises when the crawler sees the original article as the duplicate and opts to index duplicate page or at least give it a higher rating.

11. Mirrored sites – Mirrored sites are used to handle the traffic of a very popular site. Mirror sites have a good chance of being ignored by web crawlers and so won’t be indexed.

How Search Engines See Duplicate Content

There are many methods employed by different search engines to determine pages with duplicate content. The methods in many ways, from the concept, to the algorithms, and of course their effectiveness. Search engines are, however, all finding new ways to improve their methods for searching duplicate content as seen by the patents filed by different search engines companies like AltaVista, Microsoft Corporation, Google, and other bodies like the company Digital Equipment Corporation and even the Regents of the University of California.

The different patents include methods for Detecting query-specific duplicate documents, Detecting duplicate and near-duplicate files, clustering closely resembling data objects, identifying near duplicate pages in a hyperlinked database, indexing duplicate database records using a full-record fingerprint, indexing duplicate records of information of a database, utilizing information redundancy to improve text searches and methods and apparatus for detecting and summarizing document similarity within large document sets, and for finding mirrored hosts by analyzing URLs.

Each method is unique and is interesting in its approach. The methods vary greatly from generating fingerprints for records to using query-relevant information to limit the portion of the documents to be compared. Discussing each method would be interesting and would shed light as to how different search engines approach the problem. The new methods are all innovative and if some of them are used in concert with each other, it would surely improve the search engine’s ability to detect duplicate documents. However, since the patent holders are competing companies, it is unlikely that there would be collaboration between them.

Conclusion

As search engines further refine their methods for detecting duplicate content it would be harder for plagiarists to get away with what they do. However, web pages containing duplicate content for a good reason could suffer as well. Furthermore since none of the published patents tackled the issue of differentiating the original content from the duplicate ones refinement in the search engine’s methods might mean further trouble for the website owners of original content. Because of this search engines ought to find ways and invent new methods for identifying original content from duplicate ones as well as valid duplicate content.

Creating good, unique content for your blog or website can be intensive and time consuming, and you are the person who should reap the full rewards of your efforts. There is, however, a growing trend in internet marketing referred to as 'splogging' (a combination of spam and blogging) that can sometimes inadvertently punish honest webmasters, even if they create unique content from scratch.

Splogging is a practice that the search engines look down upon, and it consists of creating a free blog and using automated software programs to "scrape" together content from various sources. These automated programs are web robots that scour the internet looking for information and articles related to a certain topic, and they will copy various pieces of content from different sources to create a new page.

This scaped content is put into the blog and surrounded with ads, and the owner will then use spam or other dishonest promotional practices to try and drive traffic to his blog. None of the content on this blog or "splog" is original, it is simply articles from different websites that are copied and posted.

When it comes to the merchants of information such as Google and Yahoo, they are constantly monitoring the status of the internet and working to improve their search engine algorithms so that they can deliver relevant and useful information to people. One of the latest developments in search algorithms has been the addition of a "duplicate content filter" which will penalize the ranking of certain webpages if they contain information that is identical to some other webpage.

The search engines look down on this type of behavior because their goal is to provide people with information that is relevant to their search, and they do not want these people going to a page that is poorly designed, stuffed with ads, and filled with unoriginal and useless content. So in order to combat this, the search engines implemented a filter that will block all webpages containing the same content except for one of them.

Even if you are the most honest and genuine webmaster around and the thought of doing something like this would make it hard for you to sleep at night, there is still a chance that you and your online business could be penalized by the duplicate content filter. The way this would happen is that a web robot or content scaper will happen across your webpage and copy its content before it is indexed by the search engines, and then when your content (which may be original and have been painstakingly and laboriously crafted) is posted on the "splog" it will get indexed before your own page does.

Then when the search engines finally discover your page, they will already have a saved copy of a separate page that already contains this exact content and will penalize the rankings of your webpage instead of the unoriginal blog which should be penalized.

Another way that your business could be negatively affected by the dupicate content penalty is if you look to submitting articles as a method to garner traffic and publicity. Many webmasters will choose to create articles for their website or blog and then submit those articles to different article directories.

While this practice in and of itself is not a bad idea, one of the consequences to submitting the same articles that you put on your own website is that the search engines might index the version on the article directory first before indexing the version on your website, and they will slam you with the penalty instead of the directory. This is especially likely if you submit your articles to big, popular content directories such as Ezinearticles and ArticlesBase.

The way to avoid this is to write your articles and publish them on your own website, and then wait about a week or so before you submit them to article directories. Ideally you would want to wait until you can find your article in Google so that you know that it is indexed and that it counts as the original version (meaning all other versions will be penalized).

SEO Deadly Sins
Mistakes that reduce web page ranking the following are a list of SEO mistakes can ensure that your site maintains a low ranking with the search engines. Avoid at all costs. * Specifying no title for your page * Excessive use of images or Flash animation on a page * complicated menu systems specifying no title for your page: I cannot stress how important the title of a web page is. Failing to specify a descriptive, keyword optimized title will do untold damage to your ranking with the search engines. It is the equivalent to owning a shop and boarding up its windows. Ideally each page on your site should have a unique, content-specific title following the guidelines specified in the Title Tag Optimization article. Back to top
Excessive use of images or Flash animation on a page:

If your web page has plenty of nice-looking graphics and eye-popping Flash animation and not a lot of textual content it may indeed look nice but have you ever considered what how the search engines might see it. Search engines thrive on textual content, scavenging as much text as they can but unfortunately they cannot understand images or Flash animations like we can and so will find nothing of real value on your page. Try to balance your page so that the textual content is given priority and that any images or animations are used only when needed. Also it is a good idea to attach some text to an image by using its ALT tag as search engines use this text when determining rank. Back to top
Complicated menu systems:

Search engines spiders that crawl through our pages are a relatively primitive bunch. They find in hard to navigate complicated menu systems implemented for example in JavaScript or as a Java applet. Just because it is easy for a human to navigate through the site never assume it will be as easy for a search engine spider. A menu system using simple textual links will be easier for a spider to understand and it will be able to successfully navigate your site. A lot of the time complicated menu systems can be replicated using textual links and CSS. If you must use a complicated menu system be sure to provide a site map that is clearly accessible from the homepage of your site and contains only textual links to your pages. This ensures that even if the spider cannot understand your menu system that it will be able to find the pages on your site....
Link Popularity - Search Engines Page Rank Technology
What is Link Popularity?
Link popularity literally means the popularity of links to your site on other web sites. The more popular your site is - especially with well ranked sites, the higher will your site be ranked. For this reason we prefer to call link popularity as link quality. Most search engines decide the popularity of your individual site pages so you need to build up links to all your major pages. Thus page ranking will vary for your site pages and keywords.
Search Engine Optimization
To optimize your site for search engines, your pages should have relevant content with all related keywords appearing as valid reading matter. If your site doesn't have quality content or you have just repeated your keywords without any sense, your page rank will be affected or worse you could be black listed by the search engines.

Take a look at your website. How much of your content might be considered as duplicate by a search engine algorithm? Even though you never copy anyone you can't answer 'none' because someone can be copying you. Duplicate content is one of the biggest issues both for search engines trying to keep their results' relevancy high, and webmasters trying to avoid search engine penalties.

Penalties for having duplicate content can be really harmful. This is not just a downgrade in rankings but a move to supplementary results which are hardly visible to the most of the web users. Normally it is expected that Google would select one URL over another to display in SERPs, while duplicates could be found in supplemental results. Unfortunately this is not always so. In the thread "Duplicate content observation" in the WebmasterWorld.com forum you can read about a case when an original high quality and authoritative page was removed from Google's index together with its duplicates. Considering that this can happen even to the most honest webmaster, one can imagine the amount of attention this issue gets on any SEO forum.

Types of Duplicate Content

Duplicate content has a wider definition than the 'copy-paste' plagiarism; it is not just content scrapped from a competitor's site, a SERP or a RSS feed. Apart from this there are few more aspects that are generally referred to as duplicate content.

Circular Navigation

Jake Baille from TrueLocal vaguely defines circular navigation as having multiple paths across website. This can be understood as the same content being accessible via different URLs. An example of the circular navigation could be an article that is retrieved by links like - example.com/articles/1/ , - mysite.com/article1/ - mysite.com/articles.php?id=1

Another legitimate use of multiple URLs is forum threads. Each thread can be accessible by a link like myforum.com/index.php/topic.1201.html , and each message within the tread has a URL like myforum.com/index.php/topic.1201.msg.01.html . In the eyes of a search engine all the links lead to different pages with identical content. Solution? Think of a consistent way of linking, or apply robot.txt exclusion rules.

This can also be the case when other people link to you using differently looking URLs. Since these external links are out of your control, you should create a 301 redirect to the canonical URL you choose to be displayed.

Printer-Friendly Versions

Making a printer friendly version is a common practice and it adds value to the visitors. But printer-friendly version is also a prominent example of duplicate content! Fortunately a simple solution like adding a 'noindex' meta tag to your print pages solves the issue.

Product-Only Pages

Product pages looking similar are common among online stores. Typically they are created using a single template. Often two different product pages share a description that varies in just few words or numbers, which causes them to be filtered out as duplicate content. This issue has no easy solution. Either you rewrite robot.txt to allow only one product description to be crawled and lose SE traffic to the rest of them, or you roll up your sleeves and add something different to each product page, like testimonials, which is time consuming or nearly impossible depending on the number of product types in your stock.

How Do Duplicate Content Filters Work?

There are several algorithms in data mining aiming to detect similar text passages. The one claimed to be used by search engines is w-shingling. Each document has a unique fingerprint or shinglings - the contiguous subsequences of tokens (blocks of text). The ratio of magnitude of union and intersection of two documents' shinglings can be used to determine their resemblance. Another algorithm that can be used for duplicates detection is Levenshtein's distance

It is naturally to expect from a duplicate content filter to be able to discover the origin and rank it higher. The simplest way to detect the origin would be comparing the date of indexing implying that the original source is uploaded and crawled earlier than its copies. But with the advent of the RSS feeds the new content can be distributed instantaneously and this approach is no longer valid.

Concerning the origin's right to be ranked higher - this is not always implemented. J.S.Cassidy in her article 'Duplicate Content Penalties Problems with Googles Filter' published at SEOChat.com tells about an experiment of an article distribution. An article was syndicated twice scoring as many as 19000 copies. After some time Google, Yahoo and MSN have purged their indices leaving just few of the duplicates. MSN's filter managed not only to discover the origin but also put it to the top of the search results. Yahoo has also discovered the origin, but in the results page to the title of the article, the origin's position fluctuated obviously responding to the way Yahoo counts relevancy and authority.

To the tester's amusement Google's refined index did not include the original at all! Evidently Google featured only those pages with copies of the same article which it considered relevant and authoritative with no regard to the original source of the content! I've already mentioned a thread where a similar problem is discussed. The both stories took place in 2005 and early 2006 and so far I found no evidence that this issue is resolved.

Introduction

Search engines have become the gateway to information in the Internet. Search engines are so important that websites find that they need to rank well in search engine results pages (SERPs) in order to get noticed. With the numerous websites vying to get into the coveted position of the top 30 results listed in SERPs more and more website owners are using search engine optimization (SEO) techniques to improve their rankings. People who use SEO know that there are certain factors that can affect your ranking positively and of course negatively. Of the negative factors one of the most well-known is duplicate content.

Search engines are biased against duplicate content. As a matter of fact some sites do not get listed in SERPs because of this factor. This happens when crawlers do not index sites which they have previously determined to be a duplicate site of another site. The crawlers skip the duplicate site to be more efficient and save time. Crawler also do this for another reason - to avoid listing duplicate pages in SERPs and thus point users to different sites containing just the same information. Search engines do not like that to happen because it would be irritating for users who expect to see different sites for the different links they click. For similar sites, search engines also usually just list one of the sites and relegate the others under a link that says See related pages. For those that get manage to be listed in the SERPs the page rank is still usually affected and so affects the sites standing.

Where Search Engines See Duplicate Content

So where do crawlers see this duplicate content. And what are the possible content that they would interpret as duplicate? According to an article by William Slawski on Duplicate Content Issues and Search Engines, search engines see duplicate content from the following kind of web pages:

1. Product descriptions from manufacturers, publishers, and producers reproduced by a number of different distributors in large ecommerce sites.

2. Alternative print pages - This happens when website owners who are user friendly offer copies of the same documents in different formats for a varied printing options. Although helpful to users it might actually indexed by crawlers as duplicate pages.

3. Pages that reproduce syndicated RSS feeds through a server side script.

4. Canonicalization issues, where a search engine may see the same page as different pages with different URLs.

5. Pages that serve session IDs to search engines, so that they try to crawl and index the same page under different URLs.

6. Pages that serve multiple data variables through URLs, so that they crawl and index the same page under different URLs.

7. Pages that share too many common elements, or where those are very similar from one page to another, including title, meta descriptions, headings, navigation, and text that is shared globally. - This is common for company websites that insist on having their logo, description, etc put on every page of their website.

8. Copyright infringement - Plagiarism is of course a good reason for not being indexed. The problem is that crawlers cannot distinguish the original from the duplicate and might mistakenly filter out the original instead.

9. Use of the same or very similar pages on different subdomains or different country top level domains (TLDs).

10. Article syndication - Some writer allow their articles to be published in other websites as long as they are given credit for their work. The problem arises when the crawler sees the original article as the duplicate and opts to index duplicate page or at least give it a higher rating.

11. Mirrored sites - Mirrored sites are used to handle the traffic of a very popular site. Mirror sites have a good chance of being ignored by web crawlers and so won't be indexed.

How Search Engines See Duplicate Content

There are many methods employed by different search engines to determine pages with duplicate content. The methods in many ways, from the concept, to the algorithms, and of course their effectiveness. Search engines are, however, all finding new ways to improve their methods for searching duplicate content as seen by the patents filed by different search engines companies like AltaVista, Microsoft Corporation, Google, and other bodies like the company Digital Equipment Corporation and even the Regents of the University of California.

The different patents include methods for Detecting query-specific duplicate documents, Detecting duplicate and near-duplicate files, clustering closely resembling data objects, identifying near duplicate pages in a hyperlinked database, indexing duplicate database records using a full-record fingerprint, indexing duplicate records of information of a database, utilizing information redundancy to improve text searches and methods and apparatus for detecting and summarizing document similarity within large document sets, and for finding mirrored hosts by analyzing URLs.

Each method is unique and is interesting in its approach. The methods vary greatly from generating fingerprints for records to using query-relevant information to limit the portion of the documents to be compared. Discussing each method would be interesting and would shed light as to how different search engines approach the problem. The new methods are all innovative and if some of them are used in concert with each other, it would surely improve the search engine's ability to detect duplicate documents. However, since the patent holders are competing companies, it is unlikely that there would be collaboration between them.

Conclusion

As search engines further refine their methods for detecting duplicate content it would be harder for plagiarists to get away with what they do. However, web pages containing duplicate content for a good reason could suffer as well. Furthermore since none of the published patents tackled the issue of differentiating the original content from the duplicate ones refinement in the search engine's methods might mean further trouble for the website owners of original content. Because of this search engines ought to find ways and invent new methods for identifying original content from duplicate ones as well as valid duplicate content.

Many times, the so-called web marketing experts warn us about duplicate contents. They have stressed the fact that duplicate content will trigger a red flag from the search engines like Google. Though it is true that duplicate content is one of the many factors that the search engines abhor, it is equally true that there are cases where duplicate content becomes acceptable.

When does a duplicate content becomes acceptable? When is it legitimate to have duplicate content? These kinds of questions and other similar issues regarding duplicate content have caused several issues to arise. Here are some of the instances where duplicate content can somehow be considered as acceptable.

When Is Duplicate Content Acceptable?

• The same product listings on two different sites. If you want to include a product listing on two sites that you both own, the search engines may be able to tolerate the case.

• If a particular site has reprinted or copied a particular content from another site, this will be tolerated as long as the copying site has the right to do it with author credits.

• There are webmasters and website owners who would like to create two pages for the same item. One page would be the standard site page and the other one would be a printer friendly page. This would mean that the two pages would have the same content but it is acceptable.

• For reasons that you may not be able to explain, there are cases wherein there appears an odd duplicated page on your site. This usually happens to some sites and this is an honest error.

• Duplicate content may be a product of some errors on the site. By errors, I mean the unintentional ones.

Reasons Why Search Engines Prevent Duplicate Content

Search engines like Google try to avoid sites with duplicate content - in fact, they don't just avoid them, they want to get rid of them. This is the reason why having sites with duplicate content is something to worry about. Why do the search engines detest sites with duplicate contents?

• To prevent duplicate content sites on the internet. Online users will not benefit from browsing different sites with exactly the same content. Replicate sites have exactly the same content including titles and even codes.

• To avoid what others called scraping - a method used by others to duplicate a particular site. This arouses the issue of copyrights.

• To avoid impassive PLR articles for replicated sites.
Knowing the duplicate content will cause your site be penalized by the search engines, you must strive to provide your site with unique data and content. This is important not only to avoid penalties but also to build your site's credibility and image.