Researching the Validity of Online Competitive Intelligence Tools – Introduction

Jan 07, 2010

18 Comments

Researching the Validity of Online Competitive Intelligence Tools – Introduction

The Question

I embarked upon my study of online competitive intelligence tools in late June of 2009. My muse for this project was a study by SEOmoz comparing the website analytics of 25 Internet marketing blogs with estimates made by intelligence tools such as Compete Site Profile and Alex Site Information. SEOmoz’s conclusion was that:

Based on the evidence we’ve gathered here, it’s safe to say that no external metric, traffic prediction service or ranking system available on the web today provides any accuracy when compared with real numbers.

I found the study exciting, but the results surprising and somewhat counterintuitive. Because research thrills me, I began pondering ways I could replicate their results with a larger and more diverse sample. I immediately saw an opportunity in a relative newcomer to the competitive intelligence industry: Quantcast.

Competitive Intelligence

Companies such as Quantcast that offer web analytics, competitive intelligence, and/or market research services generally gather information from one or more of  sources:

  • Panels of Internet users
  • Aggregate ISP data
  • On-site direct measurement

Each of these methods has strengths and weaknesses. The current trend in the world of competitive intelligence is  to eschew relying on a single data source. Instead, companies are choosing to integrate two or more. Hypothetically, this allows them to compensate for the weaknesses of a single method.

Quantcast

Quantcast could be considered the pioneer of direct measurement in the competitive intelligence industry. Quantcast even compensates for the major weakness of direct measurement by employing cookie corrected audience data, taking into account …numerous factors including the frequency of visitation and the respective balance between work and home access to build a translation of cookies to people that is unique to each digital media property.

The Study

I began this study with the assumption that directly measured traffic data from Quantified websites are 100% accurate. I now know that reaching 100% accuracy with web analytics is virtually impossible. However, it is likely that a metric such as page views, which is not dependent on tracking cookies or JavaScript is as close to canon as is possible under current technological constraints.

Under this assumption, I have compared Quantcast’s direct measurements of websites to monthly traffic estimates given by services such as Alexa, Compete, and Google Ad Planner. Since June, I have conducted several pretests using smaller samples, culminating in the current analysis of more than 1,350 root domains. My correlations have remained fairly consistent regardless of sample size and month of data collection.

Further Exploration

I have gathered data on my sample from many other sources including search engines, social media services, and miscellaneous third-party tools. These additional variables give insight into the factors that mediate intelligence tool estimates. They also present the opportunity to conduct future analyses by identifying the factors that correlate with website traffic and user engagement.

I will be posting the results of my research this month, mediated only by the pace at which I can write about and display them. I strongly encourage you to leave questions and requests as comments on this post. I will make sure to address them in my analyses.

Special Thanks

I would like to thank Aaron Prebluda of Compete.com, and Danny Dover of SEOmoz for their support and patience. I truly appreciate your aid and advocacy. I am confident that you will find it was well worth your time.

How You Can Help

Interested in contributing to this study? I would greatly appreciate your taking my survey on the topic. It should take you less than a minute to complete. Make sure to send this post to your friends too. Thanks!

I Love Sharing Too!

  • Twitter
  • Facebook
  • LinkedIn
  • StumbleUpon
  • Digg
  • del.icio.us
  • Technorati
  • Sphinn

Related Posts:

  • http://topsy.com/tb/bit.ly/84pU8g Tweets that mention Researching the Validity of Online Competitive Intelligence Tools – Introduction | WoT — Topsy.com

    [...] This post was mentioned on Twitter by Sean W Ferguson, Sean W Ferguson. Sean W Ferguson said: Researching the Validity of Online Competitive Intelligence Tools – Introduction http://su.pr/6sJJt3 [...]

  • http://www.tgseo.co.uk/seomoz/what-is-pagerank-good-for-anyway-statistics-galore/ TG SEO » What is PageRank Good for Anyway? (Statistics Galore)

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://seoplans.net/seo-news/what-is-pagerank-good-for-anyway-statistics-galore/ What is PageRank Good for Anyway? (Statistics Galore) | SEO Plans

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.webulartech.com/what-is-pagerank-good-for-anyway-statistics-galore What is PageRank Good for Anyway? (Statistics Galore) – Webular Technologies

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://googleadsense.us/06/what-is-pagerank-good-for-anyway-statistics-galore.html What is PageRank Good for Anyway? (Statistics Galore) | Google Adsense

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.earnmoneynews.com/index.php/search-marketing/what-is-pagerank-good-for-anyway-statistics-galore/ What is PageRank Good for Anyway? (Statistics Galore) : Earn Money News

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://besttraffictips.com/what-is-pagerank-good-for-anyway-statistics-galore/ What is PageRank Good for Anyway? (Statistics Galore) | Best Traffic Tips

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.this-works.com/what-is-pagerank-good-for-anyway-statistics-galore/ Finally… This Works » Blog Archive » What is PageRank Good for Anyway? (Statistics Galore)

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.searchingsolutions.com/smithteamtest/?p=606 What is PageRank Good for Anyway? (Statistics Galore) | SmithTeam.com

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.social-webnet.com/?p=85 What is PageRank Good for Anyway? (Statistics Galore)

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.googleseoguide.tk/2010/06/30/what-is-pagerank-good-for-anyway-statistics-galore/ What is PageRank Good for Anyway? (Statistics Galore) | Google Seo Guide

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://digitalmarketingtalk.co.uk/?p=195 What is PageRank Good for Anyway? (Statistics Galore) | digitalmarketingtalk.co.uk

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://www.taoofseo.com/what-is-pagerank-good-for-anyway-statistics-galore/ What is PageRank Good for Anyway? (Statistics Galore) | Tao of SEO

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • http://seo.warezavr.com/what-is-pagerank-good-for-anyway-statistics-galore What is PageRank Good for Anyway? (Statistics Galore) | SEOBLOG

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]

  • Mike Roberts

    Hey Sean,

    We just did a similar study to help calibrate our search traffic estimates for SpyFu SEO Recon Files. I initially made the same assumption you make here about Quantcast Quantified data. So, we pulled the Quantified numbers for about 10k websites.

    Turns out, the Quantcast tracking pixels aren't necessarily deployed on an entire website — so there's often significant under reporting. When I thought about it, it made sense, because Quantified sites often do so to sell ads — and so they don't put the pixels where they don't sell ads. Mistakes and laziness is another possible explanation ;)

    But, what we did was pull data for the same domains from Compete and from Alexa. When all three numbers were within a certain range (I think we used 50%), we considered the data valid. So, that brought the list down to like 4500 domains.

    Anyway, I'm happy to send you some data if it helps.

  • http://wellontop.com/ Sean Weigold Ferguson

    Mike,

    That's a great point, and one I hadn't given enough thought. I'm currently revisiting this project, and would love to see what you came up with. You can reach me at: Sean at wellontop.com

  • http://www.email-direct-marketing-tool.org Sheena Harries

    how do you effectively use tools?

  • http://prosearchengineoptimization.com/what-is-pagerank-good-for-anyway/ prosearchengineoptimization.com | What is PageRank Good for Anyway

    [...] was intrigued by the study, and vowed to investigate the metric using my own data set. Because all of my data are at the root domain level, I chose to focus on the homepage PageRank of [...]