Virtual Press Room
POVs
Share |

UPDATED

Uncovering the Discrepancies in Unique Measurement Data Generated by Panel and Census-Based Online Measurement Techniques

A Point of View By Bill Perry, Director, Digital Auditing Services, Audit Bureau of Circulations, Schaumburg, Ill.

Part two in a three-part series.
download pdf This article was recently updated to incorporate the new audience measurement guidelines issued by the IAB. It is the second in a three-part series intended to address the confusion surrounding online audience and traffic metrics, including differing methodologies and the future of the online measurement industry. The first white paper in the series examined the role of third-party auditing in providing independent verification for either method.

Because ABC audits both panel and census-based data for the U.S. daily newspaper initiative Audience-FAX, I’d like to shed some light on how each of these measurement methods can generate different numbers for two of the most common metrics—unique user data and page impressions.

Defining and Calculating Unique Users

One of the more popular Web metrics is commonly referred to as “unique users” or “unique visitors” or just “uniques.” For years, these terms were interchangeable and used to describe any unique data generated by either panel-based methodologies (Nielsen Online, ComScore) or census-based methodologies (Omniture, Google Analytics). Using these terms interchangeably, with no regard to the source of the data, created a lot of confusion in the marketplace. A panel could produce one number for a website’s unique users and the census method could generate a wildly different number for that same website for the same time period, even though both numbers may have been calculated correctly based on the underlying methodologies.

An industry committee hosted by the Interactive Advertising Bureau (IAB) was formed to address these discrepancies and provide greater transparency in audience counts and terminologies. As a participant on that committee, I’m proud to announce that, after many laborious discussions, we have concluded our work and the new audience measurement guidelines are final and available to the industry. One of the key provisions of the guidelines is a redefinition of unique data as measured by a panel and by census. To eliminate marketplace confusion and provide clear notice as to the source of the measurement, unique data generated by a panel will now be referred to as unique users or visitors while unique data generated using the census method will most commonly be labeled as unique cookies.

This move is a giant step forward for online audience measurement. Both the terminology clarification and the requirement to provide additional disclosures about measurement methodology will go a long way in quelling the debate surrounding the two types of online audience reporting. Even though both panel and census-based methods may report a unique data measurement, the techniques used to arrive at the data are so fundamentally different that the resulting statistics can’t be considered equivalent.

The Panel Method

The panel-based measurement method recruits a sample of Internet users and applies their habits and behaviors to the larger Internet population. Participants on the panel are typically required to install software on their computer (preferably on each computer they use) and are provided a unique login to use each time they access the computer. This helps ensure that all of the Web activity is tied to an individual. Because the panels track the habits of actual people participating on the panel, not machines, they also have the advantage of providing user demographic data.

In an interview published in Editor & Publisher, Nielsen Online’s president said that their random-digit-dial panel (RDD) was comprised of approximately 30,000 people in the U.S. Of those 30,000 people, about 3,000 are part of the at-work panel that measures the panelists’ Internet usage while at work. For the panel companies to generate representative numbers for your website, the Internet usage habits of those 30,000 panelists need to be typical of your website’s visitors. If not, the measurements can skew greatly from month to month.

Let’s examine a couple of different scenarios that may affect panel measurements.

Minimal Panel Coverage for Regional and Niche websites
websites that are targeted to a unique or regional market and those that diverge from the broad Internet population may not be accurately represented on a panel. This is especially relevant for ABC’s business-to-business publication websites and regional and community newspapers. If there are only a few panel participants who visit your website, then the panel may not accurately represent your typical traffic. This lack of representation may result in large swings in unique user and page impression counts from month to month.

Large Volume of At-Work Traffic
One of the most controversial elements surrounding the panel companies is their ability to accurately track the Internet usage habits of the at-work population. The at-work market segment is an active and important part of the Internet community. Industry estimates project that the average at-work user generates 11 sessions per person and 26 domain visits per week. This is why website publishers are so eager to receive precise numbers for their traffic activity during normal business hours. If a significant amount of your website traffic is generated during work hours, there is the potential that the panels are not accurately representing your website’s unique users.

URLs and Domains Included in Your Website Definition
This third scenario is not influenced by the composition of the panel, but it is something that affects a significant number of our members. When panel companies gather data for a website, they must decide on what domains to include in the final overall number for that website. Upon first glance, this may seem simple; the panel companies should include the homepage of the website and any other URLs that contain the homepage domain. But in reality, many websites use third-party companies to deliver content throughout their sites. Often, these third-party companies design the pages to mimic the overall “look and feel” of the website’s homepage and the visitors never realize they’ve left their original destination. But if the panel company doesn’t include the third-party vendor’s site as part of the website definition, that traffic is credited to another site. In the newspaper industry, this is especially relevant for websites that use third-party vendors to deliver obituaries, classifieds, etc.

Let’s use a fictitious website as an example—www.anytownnews.com. This is the main homepage of the newspaper and is how the majority of the visitors access the site. Now let’s theorize that the Anytown News uses a popular service to deliver its obituaries to readers via www.obits.com/anytownnews. When the visitor of www.anytownnews.com clicks on the link to access the obituaries, they are seamlessly transferred to www.obtis.com/anytownnews. The obituary page has the same branding as the Anytown News’ website, but the domain is different. Even though the traffic should arguably be credited to the Anytown News, the panel company may not include www.obits.com/anytownnews in the website’s definition and instead may credit www.obits.com.

The Census Method

While the census method does not provide any demographic information for your site, it does have the distinct advantage of tracking almost 100 percent of the traffic on your website, regardless of visitors’ locations, including at-home, at-work and mobile devices. The census method however, does not actually track the Internet usage of a human like the panel method; instead its unique data is attributed to a unique cookie set on a browser. If the device used to access the Internet changes or if the cookie set is deleted, a new unique cookie is issued and added to the unique data count.

Change in Unique Cookie Set
Cookies are small text files stored on a computer that collect a variety of data to identify the user. Some people manually delete their cookies on a regular basis as a security or privacy measure. When they visit your website again, new cookies are issued that identify them as a new visitor. In this case, someone that deletes cookies every day and visits your site every day would be counted as 30 unique cookies in a 30-day period.

The rate of cookie deletion is debatable, although some research suggests it may be as high as 30 percent. Others argue that users who completely reject cookies or multiple users accessing the same computer may actually offset the inflation caused by cookies.

Change in Unique Device
For many of us, a change in the device we use to access the Internet is a daily occurrence—home computers, work computers, Blackberries, etc. Because the census method recognizes computers and mobile devices based on cookie sets, not the person using them, this event increases the unique cookie tally because you are counted as one unique cookie on your work computer, another unique cookie on your home computer and yet another unique cookie on your mobile device.

On the other hand, multiple people accessing the same computer may not be counted as individual unique cookies. For example, if you, your spouse and your two children all use the same browser on the same computer, the four of you may only count as one unique cookie.

Calculating Page Impressions

Despite the challenges of defining a page impression in the Web 2.0 world of Ajax, widgets and other evolving technologies, many ABC members still find this metric relevant and include it as part of their audited Web metrics. Page impression data is important because it represents the interaction of a visitor with a website. This interaction is the foundation of all other Web metrics, including unique users, unique cookies, time spent, and visits.

The controversy surrounding page impression data is far more muted than that surrounding unique data measurements. Why? Because by eliminating the “unique” factor, you’ve effectively removed the angst of much of the unique data debate. Page impressions counted by a panel are counted just like a unique user. The same panel and the same methodology used to generate unique user data are also used to generate page impression data. This is also true for census data. The log files that generate that unique cookie data generate the page impression data. The only difference on the census side is that the “unique” element is removed, rendering it irrelevant if cookies are deleted or if the same person accesses the page on several computers.

Which Methodology is Better?

Panel measurements and census measurements have their own distinct advantages and disadvantages. An independent third-party audit can help limit the discrepancies associated with each type of measurement.

Since both methodologies have strengths and weaknesses, it’s impossible to endorse one over the other; nor is it ABC’s role as an independent auditor to do so. Rather it is important that both publishers and advertisers understand the inherent differences of the two methodologies and also understand how to best apply the data generated from either measurement method to buy and sell online advertising. Like I mentioned earlier, I think the new guidelines issued by the IAB, with separate “unique” labels for each type of measurement will go a long way toward marketplace understanding of each of the companies’ underlying methodologies.

Next in the Series

In the third and final article in this series, we’ll discuss the hybrid measurement approach. This is a relatively new type of online audience measurement that combines both panel and census-based technologies. Many are heralding the “hybrids” as the Web measurement of the future.

For More Information

For more information on our interactive audit services, members are encouraged to submit an online request form or contact ABC’s manager, marketing and sales, at 416-962-5840 ext. 226.

About ABC

With nearly 4,000 members in North America, ABC is a forum of the world’s leading magazine and newspaper publishers, advertisers and advertising agencies. The organization provides credible, verified information essential to the media buying and selling process. ABC maintains the world’s foremost online database of audited-circulation information and a growing array of readership, audience and website usage data. To learn more, visit http://www.accessabc.ca.

Bill Perry joined ABC in 1985 as a newspaper field auditor. He joined ABC Interactive (ABCi) in 1997 as manager of interactive auditing and was promoted to director, special projects in 2000. As director, digital auditing services, Perry’s current responsibilities include leading a team of auditors in a variety of non-traditional auditing projects including Web activity, readership verification, insert verification and centralized audits. Perry earned his bachelor’s degree in accounting from Indiana University.