Advertising Insights

September 21, 2016

Putting Creative Pre-Testing to the Test

Understand how evaluating ad creative before it goes live can predict ad success and help build a better campaign.

Advertising InsightsCreative EffectivenessNorth America


    Watching TV on a laptop, streaming live video on a phone and reading news headlines on a tablet are just a few of the ways people consume media these days. And no matter the channel, marketers want ad campaigns that reach their audience and make an impact.

    About the Study

    About the Study

    As advertisers continue to shift more budgets to digital, and to mobile specifically, there is a need for rigorous creative evaluation. According to an Ipsos study, creative quality determines 75% of impact as measured by brand and ad recall.1

    Advertisers have long used creative pre-testing research to optimize their ads before launching a campaign. Facebook Marketing Science wanted to understand if the leading industry creative pre-testing solutions in the US could successfully be applied in a digital, feed-based environment.

    In June 2016, we tested four ad creatives from two completed US campaigns from Frito-Lay North America brands. We used three creative pre-testing solution providers, Ad-Vance by MetrixLab, LinkNow for Digital by Millward Brown and ASI Connect by IPSOS.

    We wanted to understand if:

    • Testing digital creative could give insight into how campaigns perform in-market
    • Different testing solutions would provide consistent recommendations
    • Digital creative testing results would hold up in Facebook News Feed



    Working closely with Frito-Lay North America, we identified campaigns from Doritos and Ruffles that ran on Facebook during the second half of 2015. Each campaign ran for a similar time period, used comparable targeting strategies and had the goals of increasing awareness of a new flavor and driving trial.

    Despite having the same goals, each campaign had different outcomes. The Doritos campaign had much stronger sales and brand lift than the Ruffles campaign.

    With the campaign performance measured, we could see whether pre-testing was a good determinant of a campaign’s ability to deliver on business objectives. We selected two representative videos from the Doritos and Ruffles campaigns that ran on Facebook to compare the three digital pre-testing solutions. The pre-testing solution providers were not given any background on how the ads had performed in-market.

    Pre-testing MethodologiesPre-testing Methodologies

    “The goal of this study was to quantify creative pre-testing and its ability to predict whether an ad will be effective. Determining which ad creatives have the most potential to deliver on business objectives, such as in-store sales, will help drive more impactful campaigns.” – Shawn Baron, Measurement Lead, Marketing Science, Facebook

    Chips all in for consistent results

    Chips all in for consistent results

    Results were consistent across the three pre-testing solutions, which all selected the ad creative that performed best in-market in 2015 as the strongest ad, but gave mixed results for the remaining three ads.

    The Doritos ad (below) was the strongest overall performer on both breakthrough and response. Compared to the lowest-scoring ad, the Doritos ad scored 3x higher on Ipsos’s Copy Effect Index, 8% higher on Millward Brown’s Impact Score and 38% higher for MetrixLab’s Ad Recognition. The brand and the product benefit are well integrated into the storyline, leading to strong brand recall. The video both stands out and grabs attention—and consumers are interested in the featured brand after watching the video.

    Breaking through the screen

    Breaking through the screen

    A feed environment requires attention-grabbing ad creative. And our research has shown that every second matters. People are consuming and recalling more content faster than ever before. To determine if pre-testing results will hold up in Facebook News Feed, we need to look at the different testing environments.

    Testing environments vary in how realistic they are for people viewing content.

    Testing EnvironmentsTesting Environments

    Each of the three creative pre-testing solution used a different testing environment. All three solutions measure brand presence as a factor to predict if an ad will break through. We saw that as the testing environments began to mirror how people expect to experience an ad within Facebook News Feed, it became more difficult for otherwise successful ad creative to stand out. The more relevant and personalized the surrounding content was, the more difficult it was for the ad to break out.

    This study suggests that creative has to work harder in a personalized feed environment than it does in an environment with content that isn’t as relevant. We have seen that when advertisers optimize for the screen where the ad is being delivered, creative is able to break through more effectively.

    Factors to consider

    Factors to consider

    Based on this analysis, there are three factors to consider when implementing a creative pre-test for a campaign. Consider where the ad creative will appear when selecting a testing environment (desktop vs mobile) and how that will impact your results. Look at the reporting metrics of each pre-testing solution and determine which best aligns with your business objectives. Remember that ads are rarely flighted in isolation on Facebook or other digital platforms. Current pre-testing solutions are designed to evaluate a single ad, which could be a limitation in digital testing.

    While the tests were able to consistently pick the top-performing ad, they were less clear on predicting the in-market success of the remaining three ads. The ads all tested average or below average compared to each solution’s copy-testing norms. This suggests that if a creative execution is average, the campaign flighting is going to have a larger impact on the overall campaign success.

    What it means for marketers

    What it means for marketers

    As Atin Kulkarni, Senior Director, Portfolio Analytics, of Frito-Lay North America explained, “Creative development is an important focus for us to maximize the effectiveness of our marketing efforts and drive sales. Digital creative pre-testing provides us with a new way to think about evaluating creative before we go to market.”

    While all three pre-testing solutions provided similar results for the best ad, none were able to predict the in-market outcomes of the other three ads. Broadly, there were things that we learned that can influence how marketers should evaluate their creative.

    • Put your creative to the test:

      Testing digital ad creative before a campaign goes live will help advertisers launch higher-impact campaigns in-market and spend more efficiently. Based on results from creative pre-testing, marketers can confidently decide which assets to support with media spend.

    • Think holistically:

      Playing with creative can help identify the best overall combination of ad creative assets for each campaign. While testing solutions in our study were consistent in predicting the top-performing ad, when creative is average, the other factors of the campaign become even more important. Consider how the creative fits with targeting, timing and other campaign goals.

    • Capture attention in-feed:

      While creative pre-testing worked in a feed-based environment, ads viewed in-feed require more breakthrough creative to capture attention. Facebook News Feed is fast, frequent and personalized. Advertisers should leverage iconography and branding cues strategically to captivate consumers early and get people to stop scrolling.

    Source 2: Oracle Data Cloud-measured Facebook campaigns, May–Oct 2015.

    Source unless otherwise specified: 4 Ad-Vance studies by MetrixLab, 4 LinkNow studies by Millward Brown Digital and 4 ASI Connect studies by Ipsos (Facebook-commissioned studies of people ages 18+ in the US), May 2016.

    Advertising InsightsCreative EffectivenessNorth America

    Next in Advertising Insights

    Stay in the know

    Get timely insights from Facebook IQ delivered to your inbox.