Maybe Panama Doesn't Rock For Clients?

Following up on my breathless musically-lyrical treatment of early results from Yahoo's Panama launch, this story in Mediapost (free registration required) suggests that clients may not be as thrilled about early performance as are paid search providers.

Avenue A reports that, across their 33 clients, click-through rate is definitely up, by +10%, but conversions are down an average of -5%. The story contradicts itself on cost-per-click, claiming it dropped early in the story but then referring to a CPC rise in the next paragraph.

CPA, the third most important individual metric to clients (after margin and then revenue), is up +6%.

This might mean something -- or it might not. A key question I have here is about the integrity of the study. Testing best practices state that you hold all ancillary variables in control while you test one or just a few key variables at a time, with everything else running exactly the same on a simultaneous basis (in this case, same bids, same keywords, same keyword rotation, same offer, same copy, same copy rotation, same site split within the content distribution network, etc.) If you can't manage that, then you need to apply some sophisticated regression analysis to tease out the noise in order to bring everything onto the same playing field.

In this case, the thing that exacerbates the analysis even more is that they are comparing on a before-and-after basis, in which changing competitive conditions over time can also create a challenge in determining the validity of the results.

I'm not saying that Avenue A hasn't applied all the statistical filters required to the study in order to eliminate the noise, and their conclusions may be totally valid on an objective basis. But that's the question that I as a second-hand observer have.

FOLLOW UP: I just ran across this story on BizReport, which confirms my earlier post regarding Yahoo's happiness about performance, and also cites the Mediapost story above. But this is interesting -- the BizReport story notes Avenue A's clients seeing a click increase and CPC decrease -- BUT no mention on CPA increase and conversion drops! I guess including that sort of information would have put an unwelcome and unhappy spin on the story, huh?