Advertising testing

Advertising testing is one of the staples of market research as it directly appeals to measuring and improving marketing effectivenss. Ad testing itself comes in a variety of types depending on where the advert is in development and implementation. At the start, advertising research can include creative development via concept testing and selection of candidates from mock-ups. It can include pre-testing to validate and estimate the potential impact of the advert before it goes live, or it might cover a full pre- and post- test, possibly with test and control areas to measure advertising effectiveness and to show how the adverts or commercials affect brand consideration. In addition, it's possible to use 'buzz metrics' or social media monitoring to measure how responses and campaigns spread across the internet and so measure viral effectiveness. And these are all in addition to standard measurement of response - for instance counting replies to a direct mail campaign, or measuring click-thrus and conversion rates on online media advertising.

Advertising basics and direct response measurement

The purpose of an advert is to create sales, but good advertising does more than just raise sales, it alerts consumers to the brand and imparts meaning to that brand. If all you want is for advertising to create sales then the most effective measures of advertising performance are to measure response and sales. In direct marketing and online marketing these are fundamental tests of the advertising execution. Unfortunately for direct marketing, actual response rates can be lower than 1% for a mailing and less than 0.1% for email or web-based marketing campaigns. That doesn't mean the advertising hasn't had an effect, just that response is not always the easiest or best measure and there are other important elements that advertising delivers and it may be worth looking at other metrics too.

For any form of advertising, much of the cost comes from the delivery mechanism - the type of media used, the number of times the advert is displayed and the specifics of the audience in question. For this reason, it's common to spend time optimising the advertising with small samples before unleashing the full campaign. For direct marketing, this is part and parcel of the process of delivering the advertising. For instance, alternate adverts can be shown or sent out and the best one then used for the full campaign (known as A/B testing). Or even more carefully, tailored messages are sent to individuals based on known information about them, such as shopping habits from a loyalty card, or web-tracking and by monitoring responses the return on marketing investment (RMI) can be optimised.

Mass media advertising measurement

In direct marketing, or online-marketing you can relate the person who was sent the advert to their direct response and so conclude one followed the other. For mass-marketing advertising such as posters, television commercials or radio advertising (or PR or sponsorship) the targeting of the message is much broader and so it can be difficult to separate out practical day-to-day changes in response such as seasonality, from real changes caused by the presence of the advert. The costs of television and other mass media advertising also tend to be high, so getting the advertising right before the media money is spent is extremely important.

Advertising testing therefore mostly starts at the creative end of the scale looking at concept testing using qualitative research. Various concepts are drawn up and respondents, often in focus groups, but also in direct sensory-emotional depths, describe what they take out of the advert, what they like or don't like about it and how they think it would affect their behaviour. Naturally it's very difficult for someone to say how they would respond to advertising, or which advert they would find most memorable, so researchers take care to introduce the advertising carefully and subtlely. For instance hiding the test ad among others, changing the order in which the adverts are shown, giving respondents dials to play with to show interest, or play games like a post-test after the respondents think the testing has finished.

At an initial level, these concept tests can screen out poorly thought through adverts, but they are often testing adverts before they are fully finished and if can be difficult for respondents to fully imagine the final version. An extension of this type of qualitative testing is qualitative concept development. That is where the research is used iteratively with the creative team to define and refine the ideas. It might start very open and then the design team works up concepts to test, placing them in front of respondents to see how individuals respond neurologically or psychologically to the concepts, then slowing refining and picking winners. This type of iterative development is rare, but is being used more often. With online research it can also be slotted into fast-testing to ensure the qual is reflected by small sample quant tests.


The formal testing of advertising which is practically finished is known as pre-testing. This is typically a more quantitative process to evaluate the potential success of the campaign. For broadcast advertising, much of the cost is in buying media space so in an advanced form of pre-testing the advertising is tested in a smaller region or area prior to rolling out more widely. In this way, the advertising would only be rolled out if it meets certain metrics.

Pre- Post- Test and Control testing

The main testing of advertising is done through a formal statistical test. It is possible for overt recollection of advertising to be quite poor, but for the advertising itself to have an effect on brand recognition and consideration and other market metrics, almost at a subconscious level, and secondly there is usually an amount of false recognition (around 3-4% in the UK, and up to 5-6% in the US). So to formally measure effectiveness it's not sufficient to rely on post-advertising recollection as reported by respondents. Instead measurement is done by a pre- and post- measurement using matched samples. The pre- measurement takes place before the advertising and sets a benchmark. It's normally constructed carefully to ensure that a range of different awareness and consideration measures are captured firstly without the respondent knowing which company is sponsoring the research, then with prompting to capture additional recollection. The post- measurement then re-measures these details among a sample matched to the pre- sample (matched samples) to ensure statistical replicability. Changes are then assumed to be attributable directly to the advertising campaign and any other news or information that the advertising generates.

In practice this still might not be sufficient to measure a true effect. Changes to the market, or an economic or political event or even simple seasonality can cause the post- measurement to change even without an advertising effect. So to control for this a full pre- post- test and control trial can be run. In this design the pre- and post- measures are divided into two areas (typically geographic, such as different cities) - one larger area, the test area where people get to see or hear the advertising and a smaller area - the control - where the advertising is not shown. From this it becomes possible to isolate out the advertising effectiveness from other factors by looking at how measurements changed in the control area compared to how they changed in the test area.

To make this even more sophisticated you can look at test and control areas for different media - eg some with radio, some with radio plus poster and so on, so you can start to try to isolate out media effects (generally media has a cumulative effect - that is combined has a bigger effect than either one thing or another separately). Even where there is no formal demarkation it can be possible to infer effectiveness by looking at groups that listen to the radio compared to those who didn't.

Key elements

The key elements of advertising research for mass media are therefore controlled sampling and a properly structured questionnaire. Sample size becomes important because to measure small effects a bigger sample size is needed. It is also possible to underestimate just how effective an advertising campaign needs to be to become statistically measureable. In the UK with an adult population of 60 million, if your ad campaign starts with brand awareness at 20% and your pre- and post- sample sizes are 400, then you need to raise awareness to 26% to have a significant result - it doesn't seem much, but it does mean the advert would need to have an effect on 3.6 million people which can be a tall, and expensive, order. At pre- and post- samples of 1000, the sensitivity of the test improves and you'd only need to be affecting 2.0 million people.

Social media metrics

An addition to formal statistical testing are social media metrics (also known as buzz metrics). The aim of social media measurement is to measure mentions, likes, links, tweets and other forms of interaction that occur during the advertising period and so can be associated with the advertising. Obviously it's useful to have baselines in place before the research starts. It may also be useful to combine social metrics with traditional ad measurement as, for some ads, they might not be directed towards internet type traffic and build other aspects of the brand.

For help and advice on ad testing and developing ads with research contact


How can we help?

Help with a query Site feedback Contact me