Talk:Fundraising 2012/We Need A Breakthrough

some questions to latest tests[edit]

Hi Guys, we have some questions regarding the fascinating tests you conducted during the last weeks:

According the test results the facts banner wins against the personal appeal banner. But we did not find a test comparing the facts banner with the very succesful banner layout from last year's campaign (green background photo!). Do you plan to compare these two banner layouts in a future test? Or have we overseen that one?
As we understand the new banner layouts you are trying a new approach - persuading readers to donate directly by means of banners instead of landing page appeals. We think this is very interesting and the results you've got are promising. On the other hand this approach is kind of straight forward (especially banners including donation form). We are wondering if you received certian reactions by readers or community. Have you received any feedback?
You tested the new banner positions on top and inline. Did you do a comparison to the old position as well?
Since it is not possible to see the landing pages you used - do you work with identical landing pages in a banner a/b test (e.g. July 19 Part 2 & July 26)
Would it be possible that you publish the number of banner impressions as well to analyse statistical significance (in case it's a banner and not a lp test)?

Thanks for your help, Till and Tobias

--TobiasSchumann(WMDE) (talk) 11:45, 1 August 2012 (UTC)[reply]

Correction of what I wrote below: We DID in fact test top banners vs inline banners. We just didn't post those results. We just went back through them. Big blue top facts banners beat the line white Jimmy banners by a big margin. But we've never tested the new smaller top grey banners (that some people think are less annoying) against the inline Jimmy. Zackexley (talk) 19:04, 1 August 2012 (UTC)[reply]

I'm happy that this page has been useful to someone! I'll answer each point individually:

We've compared a bigger top facts banner with the "inline banner" (that's what we've been calling the traditional banner in the page) -- and the top banner won. That top banner was actually smaller vertically than traditional inline banner, but it looked much bigger because it takes the full width of the page. So we've been using a shorter banner on top. We haven't tested that against inline Jimmy, but we're going to get to that, probably this week. I know it seems strange that we don't always test everything against that control banner. But there's a reason: we have such a short time to test, so we can only test a few things, therefore when we're going in a new direction we try to first find the best example of that new direction and then finally come back and see how it does against the control. We're hoping that the new top facts banners beat the old control banners.
The new facts banners still rely very heavily on the appeals. You saw that we tested banners with the donation form inside that bypassed the appeals. They lost. The facts banners have much lower click rates than the "personal appeal" banners. That's because we're asking for money in the banner. We've tested a lot of donation asks in banners and they always lose by donations. This is the first time we've found a way to include a donation ask that makes more money. I think it's because the facts convince a lot of people to give, and then donation ask brings them in.

we can only support that. We saw in our "5 Euro-Banner" from last year, that those kind of asks can result in a lot more productive clicks (lower click rate, higher donation/impressions). We seem to convince the people to give before they click on the banner. To the donation form inside the banner: Maybe we oversaw another test but here the inline form seem to work pretty good or at least further testable. Or do you have any other information?

I think that the current inline banners that we're testing are the same position as last year. Do they look different?
Our system that counts banner impressions is broken right now. Our tiny dev team has higher priority things that they're working on. The reason it's not a high priority to fix it is: (1) we have all the data, so we can reconstruct the banner counts later if we want to, (2) we know the approximate banner impression counts for these tests. And we find in calculating confidence that nothing changes whether we approximate for example 250,000 impressions per banner or 1,000,000 banner impressions. I'm not 100% sure that we're doing it right, but I'm 99% sure. ;-) So those confidence levels that you see on this page are calculated with some kind of estimate. You can go to the thumbtack.com tool that we're using and change the banner impression numbers yourself to see the effect. Also note that click rate is not relevant in a lot of these tests. The top facts has a much lower click rate than inline appeal. The number to look at is donations per banner impressions.
Since we would like to start testing at the end of this month, do you already know when the impression counting software will be repaired?

Zackexley (talk) 15:42, 1 August 2012 (UTC)[reply]

Could you also answer question No. 4?

Thanks. Till Mletzko (WMDE) (talk) 08:58, 2 August 2012 (UTC)[reply]

Sugeestions for better banners[edit]

Hello, folks! I have some suggestions to improve the banners.

Three columns of text is too much, I think. Not just for users with narrow computer screens or navigator windows, but it's boring to read three pieces. You should try with two columns instead, which mean it's more likely that people actualy read the text rather than just glance at it.

Also, I tend to igonre text on dark background. You should try non-saturated blue, or even better, light blue with black text. Good luck! --NaBUru38 (talk) 18:30, 3 November 2012 (UTC)[reply]

Glossary for the Data Table Please[edit]

Hi! Thank you for posting all the banner testing data. A lot of us who are advocating for non-profit causes can learn a lot (and have learned a lot) from your rigorous testing.

In the data table for this year, there are a number of new terms (compared to 2011), like amt18, amt50, ccpct, cccnv, ppcnv, etc. Could a staffer briefly explain the terms that are used in the tables? Thank you!