Jump to content

Fundraising/Chapters/WMDE 2013

From Meta, a Wikimedia project coordination wiki

This page is for discussion and informational purposes about fundraising tests from Wikimedia Germany. Discussion should occur on the discussion page. If you want to leave a note in German, the German portal site might be a better place.

Tests on de.wikipedia.org

[edit]

Test 1: facts banner: appeal vs. no appeal

[edit]

Tuesday, 5.8.2013, 10:00 UTC - Friday, 9.8. 2013, 9:00 UTC (only 1% of page views)

Last year we saw that the success of the facts banner is not much affected by changes of the content inside the drop down. Because of that finding, WMF has reduced additional content inside the drop down facts banner. So to start this years campaign prepaperation we tested two versions of the facts banner: one with the well performing appeal of Dr. Cueppers inside the drop down against one with solely the donation form inside. The results suggest that the text of the facts banner (before clicking) is the key element in convincing readers to donate: Both banner versions perform almost on the same level.

Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
with_appeal 5f 1082000 18500 1113 828 14190.77 17.14 0.017098 0.001029 0.06016 0.0007652 0.04476 0.74394 0.0131153 19.34
form_only 5f 1055500 18485 1184 865 14185.39 16.40 0.017513 0.001122 0.06405 0.0008195 0.04679 0.73057 0.0134395 18.31
  • donation/click: +4,55% (p-value = 0.3704)


--Tobias Schumann (WMDE) (talk) 10:42, 21 August 2013 (UTC)[reply]


Test 2: facts banner vs. 5€ banner

[edit]

Tuesday, 27.8.2013, 14:15 UTC - Thursday, 29.8.2013, 17:00 UTC

Sometimes less is more. Last year's best performing banner only featured one sentence: the 5€ slogan. Yet, it was not clear to us if the reason for the success was related to the end time of the fundraising campaign when readers assumingly already had recognized the facts banners with it's important information. So this year we wanted to check the performance of the 5€ banner right at the beginning of our campaign testing. The results show us that the success of the banner is not due to the point in time of the campaign. A 45% increase in donations/impressions clearly shows the power to convince readers to support Wikipedia. Still there is a difference to the tests in 2012. While this year the success of the 5€ banner lies in a profound increase of click rate in last year's campaign this banner type received far less clicks than the facts banner but a much higher percentage of productive clicks. Maybe this difference is related to point in time of the campaign.

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
facts banner 379600 9725 493 418 7668.52 18.35 0.025619 0.001299 0.05069 0.0011012 0.04298 0.84787 0.0202016 15.06
5€ banner 378300 15576 698 604 9250.32 15.32 0.041174 0.001845 0.04481 0.0015966 0.03878 0.86533 0.0244523 20.67
  • click rate: p-value < 2.2e-16
  • donations/imp: p-value = 4.382e-09
  • donations/click: p-value = 0.1127

--Tobias Schumann (WMDE) (talk) 12:53, 3 September 2013 (UTC)[reply]


Test 3: dear readers

[edit]

Wednesday, 4.9., 10:00 UTC - Saturday, 7.9., 12:00 UTC

Last year's test results revealed the power of directly adressing Wikipedia readers. In the current test we wanted to see if we could use this effect to rise our constantly low female donor rate (roughly about 20 % of all donors) and by that the donation rate in general. So we modified the form of adress to "Liebe Leserinnen und Leser" (impossible to translate that directly into English due to missing grammtical gender in nouns - it could be translated as: "Dear female and male readers" though). The results show us that this modification has no impact on donation rate nor female quota.

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 1448000 19912 963 852 16496.29 19.36 0.013751 0.000665 0.04836 0.0005884 0.04279 0.88474 0.0113925 20.24
var 1452000 19784 1097 886 18534.21 20.92 0.013625 0.000756 0.05545 0.0006102 0.04478 0.80766 0.0127646 21.76
  • donations/imp: p-value = 0.4486

--Tobias Schumann (WMDE) (talk) 12:18, 12 September 2013 (UTC)[reply]


Test 4: 5€ banner: appeal vs. no appeal

[edit]

Wednesday, 11.9., 9:45 - Monday, 16.9., 9:30 UTC

As in this year's first test here we looked at the effect of showing a personal appeal inside the drop - this time in combination with the 5€ banner. And again there was no real difference in donation rate with and without the personal appeal.

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
appeal 1435400 12427 770 632 9813.79 15.53 0.008658 0.000536 0.06196 0.0004403 0.05086 0.82078 0.0068370 18.31
no_appeal 1460500 12299 776 636 9345.00 14.69 0.008421 0.000531 0.06309 0.0004355 0.05171 0.81959 0.0063985 16.73
  • donation/click: +1,7% (p-value = 0.7607)

--Tobias Schumann (WMDE) (talk) 09:49, 23 September 2013 (UTC)[reply]


Test 5: size of donation button

[edit]

Thursday, 19.9., 12:00 am - Thursday 24.9., 9:00 am UTC

In the coming weeks we will focus more on concept and layout of the donation process. As a first step this test evaluated four different sizes of the donation button. According to the results it looks like changing the size doesn't affect the conversion rate.

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 1178200 14922 789 667 12969.00 19.44 0.012665 0.000670 0.05287 0.0005661 0.04470 0.84537 0.0110075 22.12
big 1183200 15215 811 675 13449.48 19.93 0.012859 0.000685 0.05330 0.0005705 0.04436 0.83231 0.0113670 22.73
bigger 1188200 14854 803 675 12896.59 19.11 0.012501 0.000676 0.05406 0.0005681 0.04544 0.84060 0.0108539 22.39
biggest 1175600 15031 806 674 12664.31 18.79 0.012786 0.000686 0.05362 0.0005733 0.04484 0.83623 0.0107726 25.90


Test 6: rollin' rollin' rollin'

[edit]

Thursday, 26.9., 10:00 am - Friday, 27.9., 10:00 pm UTC

By evaluating last year's test results we have learned that one of the key elements for success is to win attention by readers. Many augmentations of the donation rate in 2012 were generated by modifications of banner layout focussing on this aspect. In this test we hoped to benefit from these findings trying a new layout where the banner is not simply displayed but rolls down kind of like a jalousie. The test results confirm our presumptions: the new layout increased the click rate profoundly and by that also the donation rate is raised by 22%.

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 848500 31211 1011 841 16243.62 19.31 0.036784 0.001192 0.03239 0.0009912 0.02695 0.83185 0.0191439 23.85
roll down 817500 35920 1175 988 19383.55 19.62 0.043939 0.001437 0.03271 0.0012086 0.02751 0.84085 0.0237108 21.71
  • click rate: +15,1% (p-Wert < 2.2e-16)
  • donation/imp: +21,9% (p-Wert = 2.275e-05)


Test 7, 8 and 9: small contribution, 1€, 5€ or 10€?

[edit]

Throughout October we ran several tests focussing on the amount mentioned in our most succesful call to action "If everybody donated x €, ...". We wanted to know which amount works better: 1€, 5€, 10€ or the more open wording "small contribution". The concept behind this: with a smaller amount, we assume to lower the threshold for the support of Wikipedia so that more people decide to donate. On the other hand there exists the risk of reducing the average amount too much. Indeed, the test results shows that the average amount correlates with the mentioned amount in the banner text and the highest average results from the abstract term "small contribution". Also, the donation rate (numbers/impression) changes clearly depending on the banner text. With the "1€"-variation so many readers are being convinced to donate that the smallest average amount is far compensated.


test 7, Tuesday, 1.10., 2:30 pm - Monday, 7.10., 4:00 pm UTC

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
5€ 2953100 20813 1413 1202 21055.43 17.52 0.007048 0.000478 0.06789 0.0004070 0.05775 0.85067 0.0071299 18.78
small contribution 2932200 22060 1052 832 16914.83 20.33 0.007523 0.000359 0.04769 0.0002837 0.03772 0.79087 0.0057686 21.38


test 8, Friday, 18.10., 10:15 am - Tuesday, 22.10., 2:00 pm UTC

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
5€ 2182600 15738 1030 823 12660.66 15.38 0.007211 0.000472 0.06545 0.0003771 0.05229 0.79903 0.0058007 20.18
1€ 2210700 18132 1438 1168 15061.13 12.89 0.008202 0.000650 0.07931 0.0005283 0.06442 0.81224 0.0068128 21.44


test 9, Wednesday, 23.10., 10:00 am . Tuesday, 29.10., 10:00 am UTC

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
1€ 3203800 23320 1801 1403 18152.33 12.94 0.007279 0.000562 0.07723 0.0004379 0.06016 0.77901 0.0056659 20.81
10€ 3262600 21032 1135 880 16387.12 18.62 0.006446 0.000348 0.05397 0.0002697 0.04184 0.77533 0.0050227 21.55

--Tobias Schumann (WMDE) (talk) 10:23, 4 November 2013 (UTC)[reply]


Test 10 und 11: form banner

[edit]

In this two tests we tried the new WMF banner concept of integrating the donation form into the banner. At first we wanted to see the effect of the form as isolated as possible. So we modified a facts banner (drop) to adapt it graphically to the form banner. As expacted the click rate decreased strongly. On the other hand the donation rate increased dramatically by 98%. In a following test we ran the form banner against our current best performing 1€-banner. Still the increase of donation rate added up to 79%.


test 10

Thursday 30.10., 13:00 - Friday 01.11., 13:30 Uhr

Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
no_form 5F 1034000 10862 663 513 9733.50 18.97 0.010505 0.000641 0.06104 0.0004961 0.04723 0.77376 0.0094134 23.04
form 5F 1023700 0 1904 1006 18746.33 18.63 0.000000 0.001860 0.00000 0.0009827 0.00000 0.52836 0.0183123 14.56


test 11

Friday 01.11. 1:30 pm - Thursday 05.11. 3:00 pm

Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
1€_no_form 5F 2276500 17836 1466 1131 14962.16 13.23 0.007835 0.000644 0.08219 0.0004968 0.06341 0.77149 0.0065724 19.41
form 5F 2276400 0 4515 2027 38018.39 18.76 0.000000 0.001983 0.00000 0.0008904 0.00000 0.44895 0.0167011 20.66
  • p-value donation/imp (chi-squared test) = 2,80648183729183E-57


Test 12: rolling banner II

[edit]

Friday 08.11., 10:30 am - 5:20 pm

This time we simply retested banner display with the new roll-effect of test 6. The roll-effect seems to increase attenion on the banners by readers. To be sure about the results we tried it on the form-banner resluting in similiar results. The click rate is increased by 17% (test 6: +19%) and the donation rate by 12% (test 6: 22%).


Banner Landing-
Page
Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
no_roll 5F 2396200 0 6640 3163 57011.12 18.02 0.000000 0.002771 0.00000 0.0013200 0.00000 0.47636 0.0237923 23.18
roll 5F 2368200 0 7697 3485 65047.40 18.66 0.000000 0.003250 0.00000 0.0014716 0.00000 0.45277 0.0274670 24.92
  • statistical significance don/imp: chi-squared-test p=0,00000936


[edit]

To highlight the important "If everybody..." we retested a successful design of 2012: formatting the sentence as a blue link. In test 39 we saw an increased donation/click while the click rate didn't change. This lead us to the assumption that the whole banner text gets more convincing by highlighting this so well-proven sentence.

In our current retest we saw totally different results: an increased click-rate (+49%) but a decrease of donation/lp visit on a similiar scale (-33%) so that the overall donation rate didn't change. It looks like the formatting as a link has a totally different impact in the context of the new banner concept (donation form, no drop with personal appeal). Obviously the landing page doesn't match the expactions of people clicking on the link and perhaps looking for more information. So here we see big potential in adjusting the landing page.


Monday 11.11., 11:45 am - 7:00 pm

Banner Landing-
Page
Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_link 5F 1043100 0 3189 1326 22973.51 17.33 0.003057 0.0012712 0.00000 0.41580 0.0220243 23.43
link 5F 1052400 0 4788 1332 24933.00 18.72 0.004550 0.0012657 0.00000 0.27820 0.0236916 24.69

--Tobias Schumann (WMDE) (talk) 14:12, 15 November 2013 (UTC)[reply]


Test 14: banner text form banner: facts vs. 1€

[edit]

Tuesday 12.11., 4:45 pm - Wednesday 13.11., 09:45 am

From our second test from this year we know that reducing the banner text to the "If everybody..." sentence can be very powerful. So we needed do try this too on our current form banner. However in the banner design of test 2 the facts text was shown inside the drop and still before filling out the donation form. From our current test we know now that this is crucial. Reducing the banner text of the form banner to the "If everybody..." sentence strongly reduced it's succes: the donation rate decreased by ~30%. Combined with a smaller donation average the success of the banner fell even more.

Banner Landing-
Page
Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
facts 5F 1195100 0 3835 1680 30328.84 18.05 0.000000 0.003209 0.00000 0.0014057 0.00000 0.43807 0.0253777 23.11
1€ 5F 1202200 0 3094 1190 14973.45 12.58 0.000000 0.002574 0.00000 0.0009899 0.00000 0.38462 0.0124550 19.26
  • significance donation/imp: p = 1,26892988247306E-20 (chi-squared-test)


--Tobias Schumann (WMDE) (talk) 14:24, 18 November 2013 (UTC)[reply]


Test 15, 16, 28, 30, 53: modifications facts text

[edit]

Throughout the campaign we tested several changes to the banner text. At first we tried a new version of the facts text inspired by the current WMF text (test 15). This new text version performed significantly better (donation rate e.g. donation/imp: +20%).

In test 16 we again examined the difference between "1€" and "small contribution". As always "1€" results in an increase of donations but a decrease in average amount. Unlike previous tests of the "If everybody"-sentence it looks like within the facts the "small contribution" text works better than "1€".

To get a shorter banner text in test 28 we cut out the text part about the temple of mind. Our assumption was that a shorter text would have a higher chance to be read by Wikipedia users. In contrast the shorter version performed slightly worse in the overall donation rate (-6,7%).

In test 30 we added some aspects we expected by experience to be convincing. In fact the text changes improved the donation rate, but on such a small scale (+2,7%) that the results were not statistically significant.

An appeal gets more compelling when it is concrete and refers to a clear occasion or cause. Thats why on christmas we tried a banner text which asks the readers to give something back "on this christmas day" (test 53). That small modification improved the donation rate by around 13% - a very instructive success which gives a clear hint to communicate more focussed on occasions or causes in the next fundraiser.


test 15, Friday 15.11., 9:00 am - 1:00 pm

Banner Landing
Page
Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 5F 1273900 1452 701 12162.50 17.35 0.001140 0.0005503 0.48278 0.0095475 24.50
text_new 5F 1282800 1667 866 18364.49 21.21 0.001300 0.0006751 0.51950 0.0143159 25.40

chi-squared-test donation/imp: p = 0,00005538


test 16, Friday 15.11., 4:00 pm - Sonntag 17.11., 5:30 pm

Banner Landing
Page
Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
small_contrib 5F 8055800 19299 10180 207503.16 20.38 0.002396 0.0012637 0.52749 0.0257582 24.50
1€ 5F 8065800 21927 11807 194619.86 16.48 0.002719 0.0014638 0.53847 0.0241290 23.96

chi-squared-test donation/imp: p = 0,00000000


test 28, Monday 02.12., 00:15 pm - Tuesday 03.12., 00:15 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 4182300 5391 2696 49653.99 18.42 0.001289 0.0006446 0.50009 0.0118724 29.08
no_park 4228700 5481 2652 49195.98 18.55 0.001296 0.0006271 0.48385 0.0116338 30.48

chi-squared-test donation/imp: p = 0,314689


test 30, Tuesday 05.12., 11 am - Friday 06.12., 5:30 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
text_ctrl 5221100 6015 2876 54755.30 19.04 0.001152 0.0005508 0.47814 0.0104873 28.41
text_var 5169400 6160 2925 57276.78 19.58 0.001192 0.0005658 0.47484 0.0110800 29.26

chi-squared-test donation/imp: p = 0,3065


test 53, Monday 23.12., 11 pm - Wednesday 25.12., 9 am

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 2042900 4126 2156 45632.07 21.17 0.002020 0.0010554 0.52254 0.0223369 19.59
xmas_textvar 2041600 4395 2439 51387.08 21.07 0.002153 0.0011947 0.55495 0.0251700 19.58

chi-squared-test donation/imp: p = 0,00003123




Test 17, 19, 20, 23, 41, 44, 55, 56: time delay

[edit]

During the Fundraiser we put some effort to understand the effect of a time delay of the banner display. Why did we do that? With our banner communication we interupt the reader in his search for information. Wouldn't it be much better if the banner is displayed after the reader found what he was looking for? He would probably be less distracted by the banner and much more thankful so that his disposedness to donate would grow. As plausible this sounds as difficult this is to implement. Different information needs go along with different durations of reading. And the average retention period on Wikipedia probably differs with the time of the day and the weekdays. In the end we had no other choice than to test different time delay periods by trial and error. After the total of eight tests we now can say that a time delay between 7,5 and 15 seconds improves the donation rate significantly.

We tested these time intervals: 60, 30, 15 and 7,5 seconds. The results suggest that the click rate drops with growing time delay. On the other hand, the conversion rate on the landing page improves with time delay, but seems to stagnate at 15 seconds. This goes along with our assumptions about the impact of the time at which the banner appears. And it is reflected on the impact on the overall donation rate (donation/imp): While a longer time delay is not successful a delay between 7,5 and 15 seconds improves the donation rate by up to 14%. Because of the expected variation of the average retention period on the different weekdays we retested some time intervals plus ran intervals against each other. In the end a time delay of 7,5 seconds performs best and was implemented most time during the campaign.

Test 17, Monday 18.11., 9:00 pm - Tuesday 19.11., 8:00 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_delay 3926400 6226 2873 58464.06 20.35 0.001586 0.0007317 0.46145 0.0148900 28.76
delay_30s 4006000 6018 3090 61968.11 20.05 0.001502 0.0007713 0.51346 0.0154688 29.14
  • chi-squared-test visitor/imp: p = 0,002766
  • chi-squared-test donation/visitor: p = 0,000000
  • chi-squared-test donation/imp: p = 0,041739


test 19 Friday 22.11., 11:45 am - 5:30 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_delay 1322800 1687 748 15063.53 20.14 0.001275 0.0005655 0.44339 0.0113876 25.33
delay_60 1399000 1308 643 13453.00 20.92 0.000935 0.0004596 0.49159 0.0096162 28.30
  • chi-squared-test visitor/imp: p = 0,000000
  • chi-squared-test donation/visitor: p = 0,008710
  • chi-squared-test donation/imp: p = 0,000112


test 20 Friday 22.11., 5:30 pm - Sunday 24.11., 1:00 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_delay 5090000 8329 4088 83313.43 20.38 0.001636 0.0008031 0.49082 0.0163681 25.05
delay_15 4990400 8549 4570 90724.19 19.85 0.001713 0.0009158 0.53457 0.0181797 24.90
  • chi-squared-test visitor/imp: p = 0,002885
  • chi-squared-test donation/visitor: p = 0,000000
  • chi-squared-test donation/imp: p = 0,000000


test 23 Thursday 28.11., 7:00 pm - Friday 29.11., 10:30 am

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_delay 1593500 1982 841 17522.50 20.84 0.001244 0.0005278 0.42432 0.0109962 27.12
delay_15 1602300 1961 964 20222.34 20.98 0.001224 0.0006016 0.49159 0.0126208 27.50
  • chi-squared-test visitor/imp: p = 0,611696
  • chi-squared-test donation/visitor: p = 0,000022
  • chi-squared-test donation/imp: p = 0,005454


test 41 , Sunday 15.12., 1:45 pm - Monday 16.12., 1:15 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
delay_15 3564200 4361 2062 43676.14 21.18 0.001224 0.0005785 0.47283 0.0122541 25.12
delay_7,5 3529100 4809 2233 45783.13 20.50 0.001363 0.0006327 0.46434 0.0129730 25.69
  • chi-squared-test visitor/imp: p = 0,000000
  • chi-squared-test donation/visitor: p = 0,415859
  • chi-squared-test donation/imp: p = 0,003341


test 44 , Monday 16.12., 10 pm - Tuesday 17.12., 4:30 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_delay 2474800 2840 1157 24763.58 21.40 0.001148 0.0004675 0.40739 0.0100063 27.41
delay_7,5 2385900 2513 1106 22826.46 20.64 0.001053 0.0004636 0.44011 0.0095672 25.34
  • chi-squared-test visitor/imp: p = 0,001727
  • chi-squared-test donation/visitor: p = 0,015594
  • chi-squared-test donation/imp: p = 0,839830


test 55, Saturday 28.12., 2:45 pm - Sunday 29.12., 6 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_delay 3356500 6884 3873 82483.04 21.30 0.002051 0.0011539 0.56261 0.0245741 23.45
delay_7,5 3318700 7114 4260 88047.13 20.67 0.002144 0.0012836 0.59882 0.0265306 23.98
  • chi-squared-test visitor/imp: p = 0,008877
  • chi-squared-test donation/visitor: p = 0,000014
  • chi-squared-test donation/imp: p = 0,000002


test 56, Sunday 29.12., 6 pm - Monday 30.12., 11 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
delay_15 3771900 6877 4290 90130.42 21.01 0.001823 0.0011374 0.62382 0.0238952 22.15
delay_7,5 3761100 7304 4344 91444.13 21.05 0.001942 0.0011550 0.59474 0.0243131 21.55
  • chi-squared-test visitor/imp: p = 0,000170
  • chi-squared-test donation/visitor: p = 0,000391
  • chi-squared-test donation/imp: p = 0,474747






Test 21, 31, 52: new landing page layout

[edit]

During each fundraiser one of our main concerns is the drop out rate on the landing page. Even now with the donation form in the banner, still about 50% of the people filling out the form break up the process on the landing page and evenutally don't donate. That's kind of strange. After all, filling out the form on the banner indicates a clear intention to donate.

In this light the approach for the 2013 fundraiser included a completely new design and relaunch of the landing page featuring well-known factors supporting online payment processes. For instance the layout should be clean and clear structured, the payment process should feature positive feedback (e.g. green check marks after filling out a form element) and informations should be provided right where they are needed. Being curious about of the outcome, we started to test the new landing page layout. A first test with a slightly negative result was not trustworthy because afterwards it appeared that a technical error occured on the new landing page (test 21). A second test afterwards featured a new designed donation form in the banner to provide more continuity in design to the landing page (test 31). This produced somewhat strange results: a 15% decrease in click rate and a 13% increase in landing page conversion with a the final effect that both test groups performed more or less equal in donation rate. We don't have an explanation for the decrease in click rate plus this test didn't bring more clarity about the impact of the new landing page layout. Finally, a third and fourth test on wikipedia.org and wikipedia.de over long testing periods brought certainty: the new landing page layout indeed decreases the drop out rate. An increase of landing page conversion of 6% and 5% is not so much as expected but still is a continous improvement.


test 21 , Wednesday 27.11., 1:00 pm - Thursday 28.11., 12:00 am

Banner Landing-Page Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl layout_old 3513000 4036 1897 37924.00 19.99 0.001149 0.0005400 0.47002 0.0107953 29.05
ctrl layout_new 3484600 4040 1851 37210.64 20.10 0.001159 0.0005312 0.45817 0.0106786 26.81

chi-squared-test donation/visitor: p = 0,285606

test 31 , Friday 06.12., 5:30 pm - Monday 09.12., 11:30 am

Banner Landing-Page Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
form_old lp_old 8482300 11442 5747 112457.79 19.57 0.001349 0.0006775 0.50227 0.0132579 26.39
form_new lp_new 8314500 9501 5392 108190.45 20.06 0.001143 0.0006485 0.56752 0.0130123 25.41

chi-squared-test donation/visitor: p = 0,000000


Test 52, Monday 23.12., 9 am - Friday 27.12., 00:15 pm

Banner Landing-Page Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl layout_old 6713400 13451 7437 157678.19 21.20 0.002004 0.0011078 0.55290 0.0234871 24.25
ctrl layout_new 6773400 13080 7662 160878.99 21.00 0.001931 0.0011312 0.58578 0.0237516 18.56

chi-squared-test donation/visitor: p = 0,00000006



Test 22, 34, 35, 40, 42: form banner <--> facts_drop/Jimmy_drop/5€_drop

[edit]

The integration of the donation form into the fundraising banners brought a huge boost in the number of donations. But we couldn't be sure that this banner concept, which gives relatively few informations, would work best over the whole course of the annual campaign. So throughout the fundraiser, we repeatedly tested the form banner against drop down banners which provide more information and a donation appeal. We used different drop down banners which featured the facts text, the "if everybodey would donate 5€,..." and the Jimmy appeal. But at no time the drop down banners outperformed the facts banner. Only the Jimmy appeal came close while the facts banner still had an increase in donation rate by +2%.

test 22 , Thursday 28.11., 12:00 am - 7:00 pm

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
facts_drop 1702100 16669 849 695 14382.10 20.69 0.009793 0.000499 0.05093 0.0004083 0.04169 0.81861 0.0084496 30.77
facts_form 1678600 0 1794 821 16807.50 20.47 0.000000 0.001069 0.00000 0.0004891 0.00000 0.45764 0.0100128 27.31

chi-squared-test donation/impression: p = 0,000452


test 34 , Tuesday 10.12., 4:00 pm - Wednesday 11.12., 9:00 am

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
facts_drop 2200900 20954 1302 1039 20584.55 19.81 0.009521 0.000592 0.06214 0.0004721 0.04958 0.79800 0.0093528 31.63
facts_form 2132200 0 2432 1140 23216.45 20.37 0.000000 0.001141 0.00000 0.0005347 0.00000 0.46875 0.0108885 31.32

chi-squared-test donation/impression: p = 0,003674


test 35 , Wednesday 11.12., 11:30 am - Wednesday 11.12., 5:00 pm

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
pa_Jimmy_drop 1452200 35205 751 569 11637.00 20.45 0.024243 0.000517 0.02133 0.0003918 0.01616 0.75766 0.0080134 35.42
facts_form 1420300 0 1349 567 11454.00 20.20 0.000000 0.000950 0.00000 0.0003992 0.00000 0.42031 0.0080645 30.40

chi-squared-test donation/impression: p = 0,752729


test 40 , Friday 13.12., 5:45 pm - Sunday 15.12., 00:00 am

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
5€_drop 4026900 70876 3517 2878 49334.77 17.14 0.017601 0.000873 0.04962 0.0007147 0.04061 0.81831 0.0122513 26.42
facts_form 3263200 0 5113 2581 49811.50 19.30 0.000000 0.001567 0.00000 0.0007909 0.00000 0.50479 0.0152646 24.87

chi-squared-test donation/imp: p = 0,000182


test 42 , Monday 16.12., 1:15 pm - 10:00 pm

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
5€_drop 2257200 42680 1444 1128 20286.29 17.98 0.018908 0.000640 0.03383 0.0004997 0.02643 0.78116 0.0089874 24.95
facts_form 1926400 0 2379 1053 22332.01 21.21 0.000000 0.001235 0.00000 0.0005466 0.00000 0.44262 0.0115926 26.10

chi-squared-test donation/imp: p = 0,036274




Test 24 and 38: banner layout: text highlighting

[edit]

Very often we could see in our tests how compelling the "If everybody" sentence is. For higher visibility of the sentence we tested the yellow text highlighting from the WMF banner layout. This was really successful: The donation rate improved by 46%. Thats why we wanted to use this effect also to highlight our bank account in the banner. We know from surveys that there are plenty of Wikipedia readers which want to donate but don't trust online payment methods in generell. During every fundraiser we receive tons of emails where readers ask for our bank account number to donate by bank transfer. That's why we had put these informations on our banners (see test 29), but even after that people still asked in emails for them. So we decided to highlight them. In test 38 we only wanted to see if this layout modification has an effect on the donation rate, but it hasn't much.


test 24, Friday 29.11., 12:00 am - 6:45 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_highlight 1320200 1461 733 15002.00 20.47 0.001107 0.0005552 0.50171 0.0113634 25.98
highlight 1288100 1894 1047 21336.95 20.38 0.001470 0.0008128 0.55280 0.0165647 24.62

chi-squared-test donation/imp: p = 0,00000000


test 38, Tuesday 12.12., 7:00 pm - Friday 13.12., 1:30 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_highlight 2105300 3183 1510 29505.31 19.54 0.001512 0.0007172 0.47440 0.0140148 27.17
highlight 2084700 2956 1403 26351.69 18.78 0.001418 0.0006730 0.47463 0.0126405 27.07

chi-squared-test donation/imp: p = 0,08583915




Test 25: salutation in capital letters

[edit]

Friday 29.11., 6:45 pm - Saturday 30.11., 5:00 pm

For the purpose of getting more attention on the important salutation we tried a version in capital letters following the layout of WMF. Unfortunately, it did not effect the conversion rates positively.

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 2250500 3848 2130 39670.52 18.62 0.001710 0.0009465 0.55353 0.0176274 23.60
caps 2254300 3873 2137 40256.00 18.84 0.001718 0.0009480 0.55177 0.0178574 24.13

chi-suared-test donation/imp: p=0,95846222






Test 26, 27, 29: hover element

[edit]

With the payment process starting directly in the banner, the form banner provides relatively few information before the decision to donate. We expected at least some people would like to have more information before donating. This is why we thought about a way to integrate some additional information into the form banner. So the hover element came to our mind. In test 26, we changed the content of this lower part of the banner, only visible when one moves with the mouse over it. In the space where prior only our account number had been shown, we now informed about tax deductibility, provided links to information about spending and about contact possibilities plus gave yet another payment possibility: 5€ via text message. This resulted in a certain amount of clicks on the additional information links so that the click rate of the test group improved by about 17%. However, only few of these eventually donated. Thus, the overall donation rate didn't change in test.

By changing the background colour of the hover element to blue, we aimed to put more attention to it (test 27). This did not really change the test results compared to the previous test. Because of this we started to wonder if the hover element would have any effect on reader's attention. Before, we had thought that the hover element supported the visibility of the banner in general. So now we wanted to evaluate this by testing banners with a hover against one without (test 29). An advantage of leaving the hover away was that the bank account now became visible independently of any mouse interaction. Also this test did not result in a change of donation rate. For the rest of the campaign we continued with non-hover banners beacause of the improved visibility of the account numbers.


test 26 , Saturday 30.11., 7:15 pm - Sunday 01.12., 5:30 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
hover_ctrl 2979400 4939 2699 50797.57 18.82 0.001658 0.0009059 0.54647 0.0170496 25.35
hover_var 2946200 4665 2525 49549.99 19.62 0.001583 0.0008570 0.54126 0.0168183 24.77
info_links - 1023 72 1472.01 20.44 0.000000 0.0000000 0.07038 0.0000000 27.42

chi-squared-test donation/imp: p = 0,320062


test 27 , Sunday 01.12., 5:30 pm - Monday 02.12., 00:15 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
hover_ctrl 2754100 3760 1901 37246.49 19.59 0.001365 0.0006902 0.50559 0.0135240 29.11
hover_blue 2754900 3736 1873 35349.27 18.87 0.001356 0.0006799 0.50134 0.0128314 28.64
info-links - 460 27 1000.00 37.04 0.000000 0.0000000 0.05870 0.0000000 23.81

chi-squared-test donation/imp: p = 0,992855

test 29 , Wednesday 04.12., 3:15 pm - Thursday 05.12., 11:00 am

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl_hover 2808400 3491 1777 34244.78 19.27 0.001243 0.0006327 0.50902 0.0121937 29.84
bnk_nohvover 2786500 3541 1748 34222.71 19.58 0.001271 0.0006273 0.49365 0.0122816 29.72

chi-squared-test donation/imp: p = 0,797848




Test 32: sticky on lower side of screen

[edit]

Monday 09.12., 11:30 am - Monday 09.12., 5:00 pm

In the hope of gaining more attention of Wikipedia readers, we tested a modified position of the banner on the screen. Assuming that users attention while reading would focus more on the on the lower part of the monitor, we tried a position on the lower edge of the screen. However, the results suppose the opposite: the test group with the banner display on the lower edge of the screen had a 12% lower click rate. The overall donation rate decreased by 10%.

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
sticky_top 1539100 1562 700 12760.50 18.23 0.001015 0.0004548 0.44814 0.0082909 30.45
sticky_bottom 1428100 1277 585 10972.30 18.76 0.000894 0.0004096 0.45810 0.0076831 26.97

chi-squared-test visitors/imp: p = 0,0007806 chi-squared-test donations/imp: p = 0,061650




Test 33: donor comments on landing page

[edit]

Monday 09.12., 10:00 pm - Tuesday 10.12., 4:00 pm

We still wonder why about 50% of the people filling out the form in the banner break up the process on the landing page and evenutally don't donate. After all, filling out the form on the banner indicates a clear intention to donate. To convince some of these 50% in this test we featured a box with the latest five donor comments on our landing page. In addition to the often very strong stories and compelling reasons to donate these comments entail, we tried to use them as a mechanism of social proof: to see that other people trust this site makes this site more trustworty for yourself. But unfortunately the comments didn't have a real influence and the donation rate didn't change during the campaign test.

Update: A similiar and simlutaneous test on wikipedia.de came to the same results.

Banner Landing-Page Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl no_comments 2610300 2704 1179 22559.77 19.13 0.001036 0.0004517 0.43602 0.0086426 28.57
ctrl comments 2552800 2709 1216 21889.62 18.00 0.001061 0.0004763 0.44887 0.0085747 31.27

chi-squared-test donation/visitor: p = 0,193106



Test 36: text and layout banner: short version in bigger font size

[edit]

Wednesday 11.12., 5 pm - Tuesday 12.12., 00:30 pm

Here we tested a shorter version of the facts text which allows an increased size of letters. This WMF change in banner layout seems to get more people reading the banner text. With an increase in click rate the overall donation rate in our test was increased by 10%.


Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
text_ctrl 2221500 2648 1244 26283.42 21.13 0.001192 0.0005600 0.46979 0.0118314 28.56
text_short_bold 2195500 2889 1355 28422.00 20.98 0.001316 0.0006172 0.46902 0.0129456 28.44

chi-squared-test donation/imp: p = 0,013206

--Tobias Schumann (WMDE) (talk) 14:07, 20 January 2014 (UTC)[reply]



Test 37 und 46: banner colors

[edit]

Changes in banner colour very often had a profound impact on donation rates in campaign tests during the last years. One possible reason could be an increased awareness of the banner by readers. During the 2013 campaign however we saw contradictory test results concerning banner colours. The change from green to yellow on a banner on wikipedia.de (the German search portal for de.wikipedia.org) brought a significant increase in donation rate by the signal colour yellow (test 2: donation/imp +22%). We evaluated another change in colour - yellow against blue - in three further tests. While the first one showed a great success for blue (test 37: donation/imp +28%), the second one resulted in practically no difference between the colours (test 46: donation/imp: +3%). Likewise an identical test on wikipedia.de saw no obvious winner (test 4: donation/imp: -3%). It remains unclear if the first, very positive result really represents a factual effect of the colour blue or if the result is not trustworthy despite its statistical significance.


test 37, Tuesday 12.12., 11:30 am - 7:00 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
yellow 1565900 1780 764 15324.00 20.06 0.001137 0.0004879 0.42921 0.0097861 23.60
blue 1561200 2189 977 20816.09 21.31 0.001402 0.0006258 0.44632 0.0133334 29.84

chi-squared-test donation/imp: p = 0,000000


test 46, Tuesday 17.12., 7:10 pm - Wednesday 18.12., 3:30 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
yellow 2674600 2947 1296 28011.05 21.61 0.001102 0.0004846 0.43977 0.0104730 27.08
blue 2684200 2955 1339 27263.11 20.36 0.001101 0.0004988 0.45313 0.0101569 26.40

chi-squared-test donation/imp: p = 0,455724



Test 39: dialogue banner

[edit]

Friday 13.12., 1:30 pm - 4:30 pm

In this test we tried a completey new approach. Our goal was to reach more Wikipedia readers by starting a dialogue with them. For this purpose we showed a banner which asks if a reader wants to help Wikipedia - without clarifiying what help that could be. Instead two buttons were displayed to allow for a decision to help or not to help Wikipedia. As the result of clicking on the not-help-button the banner disappeared. In the case of a desire to help the usual facts banner with the donation form included was displayed afterwards. Unfortunately, this new approach profoundly decreased the donation rate (donation/imp: -61%). Still, the test was instructive and perhaps shows possible further developments. Indeed, more readers reacted to the banner: 32% more people clicked on the button to help Wikipedia compared to the number of people which completed the donation form in the form banner.

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
facts_form 532500 0 594 275 5967.21 21.70 0.000000 0.001115 0.00000 0.0005164 0.00000 0.46296 0.0112060 23.79
ask_banner 585800 789 177 118 2335.11 19.79 0.001347 0.000302 0.22433 0.0002014 0.14956 0.66667 0.0039862 24.00

chi-squared-test donation/imp: p = 0






Test 47, 49, 57: donation progress bar

[edit]

Overlooking the past years the 2013 fundraiser in his course has much more similarity with the campaign in 2011 than the one in 2012. In the latter, we succeeded relatively early in finding improvements to compensate and reverse the notorious decline of donations after the usually fulminant beginning. That's why we were able to show banners, stories and appeals of Wikip/medians, which gained much attention by readers, allowed for a gaze inside the world of Wikipedia, but did not convince readers to donate as much as other banners. Instead, in 2011 it took some time until the last campaign weeks before we found a way to uplift the donation numbers to bring the campaign to a successful end. It was the new 5€ banner in the final part of the campaing which resulted in a nearly dramatic increase of donations.

Likewise, the 2013 campaign followed a difficult course. With small improvements we had been stabilizing the revenue on a certain scale throughout the weeks, but that wouldn't suffice at all to reach the campaign goal. So we thought: Why shouldn't we illustrate this situation in the banner? We returned to the old concept of an animated donation barometer, which shows how many donations came in already, how much money is still missing and what time remains to complete the goal. The old barometer went through a revision and got a new look by our web designer. At first, we tested it in a 5€ banner because it's layout provides more space (test 47). Compared to the facts banner (with donation form), the donation rate improved profoundly by 83%. Knowing that the facts banner had outperformed the 5€ banner in previous tests, we now did our best to integrate the barometer in it's densely filled banner space and compared it in test 49 with the 5€ banner (with barometer). By this, the donation rate grew by further +24%. However, the real uplift was even higher because of the clearly lower donation average of the 5€ banner (amount/imp: +49%). A third and last test of the barometer focussed on it's description text of the remaining campaign time. On the last day of the fundraiser we tested "Today our campaign ends" against an automatic countdown of the last 24 hours. Here we can say: Keep it simple. Counting down minutes and hours decreased the donation rate by 20%.

With the development of the donation barometer, we succeeded in significantly rising the donation numbers in the last weeks to a sustaining higher level. Only by this key improvement of the 2013 campaign we were able to finally reach the campaign goal and finish in time right before new years eve.


test 47, Wednesday 18.12., 3:30 pm - Thursday 19.12., 10:30 am

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
5€_drop_thermo 2316000 25184 2666 2227 39153.21 17.58 0.010874 0.001151 0.10586 0.0009616 0.08843 0.83533 0.0169055 25.40
facts_form 2165400 0 2486 1135 21990.00 19.37 0.000000 0.001148 0.00000 0.0005242 0.00000 0.45656 0.0101552 27.72

chi-squared-test donation/imp: p = 0


test 49, Thursday 19.12., 5:30 pm - Friday 10:30 am

Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
facts_form_thermo 1606600 0 3396 1992 43127.06 21.65 0.000000 0.002114 0.00000 0.0012399 0.00000 0.58657 0.0268437 23.79
5€_drop_thermo 1661000 16922 1925 1660 29984.78 18.06 0.010188 0.001159 0.11376 0.0009994 0.09810 0.86234 0.0180522 24.74

chi-squared-test donation/imp: p = 0


test 57, Monday 30.12., 11 pm - Tuesday 31.12., 1:45 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 1028100 2393 1597 35672.43 22.34 0.002328 0.0015534 0.66736 0.0346974 21.46
hour_countdown 1046000 2024 1302 27188.33 20.88 0.001935 0.0012447 0.64328 0.0259927 19.15
  • chi-squared-test visitor/imp: p = 0
  • chi-squared-test donation/visitor: p = 0,093128
  • chi-squared-test donation/imp: p = 0





Test 48: layout and text of facts inside drop banner

[edit]

Thursday 19.12., 11:45 am - 4:30 pm

For a short period during the fundraiser it appeared that the drop version of the banner would outperform the donation form banner. From last year we know that the facts text "inside" the drop also plays an important role. In this test we evaluated two versions of the facts text differing in layout and text length. A blue and shorter version came out ahead, but only with a minor advance (donation/click: +3,7%). Furthermore, the result was not statistically significant. That's why we assume that there is no real difference between the two versions of the facts text inside the drop banner.


Banner Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
% Women
red_and_long 986300 9574 950 786 15047.00 19.14 0.009707 0.000963 0.09923 0.0007969 0.08210 0.82737 0.0152560 24.77
blue_and_short 1042000 9934 984 846 14543.54 17.19 0.009534 0.000944 0.09905 0.0008119 0.08516 0.85976 0.0139573 24.89
  • chi-squared-test click/imp: p = 0,205996
  • chi-squared-test donation/click: p = 0,597
  • chi-squared-test donation/visitor: p = 0,049813
  • chi-squared-test donation/imp: p = 0,706822





[edit]

During the fundraiser we tried to improve the conversion rate on the landing page in several ways due to a drop out rate of 50%. One assumption was that people got irritated by the WMDE logo on the landing page because they had not known the organisational background, yet. Suspicious by the missing Wikipedia logo they break up the donation process. So in two tests we replaced the WMDE logo by the Wikipedia puzzle ball, one time on the old landing page layout, the second time on the new one. We were really surprised that this had absolutely no effect on the landing page conversion.

test 50 , Friday 20.12., 1:00 pm - Saturday 21.12., 7:00 pm

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
WMDE_Logo 2997700 7056 4097 87600.11 21.38 0.002354 0.0013667 0.58064 0.0292224 21.83
WP_logo 3008900 7118 4143 88668.95 21.40 0.002366 0.0013769 0.58205 0.0294689 22.15

chi-squared-test donation/visitor: p = 0,86538749


Test 54, Friday 27.12., 00:15 pm - Saturday 28.12., 2:45 pm

Banner Landing-Page Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl wmde_logo 3004100 6052 3618 75559.00 20.88 0.002015 0.0012044 0.59782 0.0251520 21.72
ctrl wp_logo 3003600 6021 3576 77287.15 21.61 0.002005 0.0011906 0.59392 0.0257315 21.57

chi-squared-test donation/visitor: p = 0,66257838





Test 51: donation form: preselected amount

[edit]

Saturday 21.12., 7 pm - Monday 23.12., 2 pm

In this test we simply analysed the effect of a preselected default amount in the donation form. It shows: No preselection of an amount has no real impact on the overall donation rate. On the other hand click rate (-15%) and landing page conversion (+20%) change on a statistically significant scale, but in such a manner that they equal each other out in their effect. We're puzzled by these results. One possible explanation could be that some users break up the process when they see a warn notification after not filling out an amount in the radio buttons and pressing the donation button. Perhaps these people just wanted to see more information before donating and because of the notification they think that pressing a donation button already initiates a financial transaction.

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl 4206500 9061 5214 112115.67 21.50 0.002154 0.0012395 0.57543 0.0266530 22.75
no_default 4248100 7778 5393 116839.26 21.66 0.001831 0.0012695 0.69337 0.0275039 23.67
  • chi-squared-test visitor/imp: p = 0
  • chi-squared-test donation/visitor: p = 0
  • chi-squared-test donation/imp: p = 0,217926


--Tobias Schumann (WMDE) (talk) 09:57, 9 April 2014 (UTC)[reply]



Tests on wikipedia.de

[edit]

Wikipedia.de is a German search portal for content in de.wikipedia.org. During the fundraiser we also display banners on this site.

Test 1: with vs. without donation form in banner

[edit]

Monday, 28.11.2013 - Wednesday, 30.11.2013

A lot of tests throughout the campaign documented the positive impact of integrating the donation form into the banner itself (here and here). Still, we didn't want to apply this change to wikipedia.de without retesting it here. Great to see: Also on this website the donation rate was clearly improved by using a form banner (donation/imp: +27%).


Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
no_form 172291 634 137 3122.00 22.79 0.003680 0.0007952 0.21609 0.0181205 21.00
form 172128 307 174 4069.00 23.39 0.001784 0.0010109 0.56678 0.0236394 18.18



Test 2: banner color - green vs. yellow

[edit]

Wednesday, 4.12.2013 - Saturday, 7.12.2013

During the 2013 campaign we ran several tests of banner colours. An evaluation summing up these tests you can find here.

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
green 290180 447 277 5817.00 21.00 0.001540 0.0009546 0.61969 0.0200462 22.43
yellow 396747 752 463 10122.70 21.86 0.001895 0.0011670 0.61569 0.0255142 21.23



Test 3: donor comments on landing page

[edit]

Monday, 9.12.2013 - Sunday, 15.12.2013

Parallel to a similiar test on de.wikipedia.org we tried to improve the drop out rate by showing the newest donation comments on the landing page. Similiar to his twin, the test did not result in any change of the donation rate.


Banner Landing-
Page
Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl no_comments 620875 936 584 13559.50 23.22 0.001508 0.0009406 0.62393 0.0218393 20.17
ctrl with_comments 619731 998 621 17048.05 27.45 0.001610 0.0010020 0.62224 0.0275088 16.42




Test 4: banner color - yellow vs. blue

[edit]

Sunday, 15.12.2013 - Tuesday, 19.12.2013

During the 2013 campaign we ran several tests of banner colours. An evaluation summing up these tests you can find here.

Banner Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
yellow 449946 647 419 13365.68 31.90 0.001438 0.0009312 0.64760 0.0297051 19.58
blue 450710 650 407 11335.00 27.85 0.001442 0.0009030 0.62615 0.0251492 16.92


--Tobias Schumann (WMDE) (talk) 11:48, 9 April 2014 (UTC)[reply]

Test 5: landing page layout: new vs. old

[edit]

Thursday, 19.12.2013 - Tuesday, 31.12.2013

Here we tested a new, in layout and concept completely revised landing page. The detailled analysis of the test results is posted together with test results of the new landing page on de.wikipedia.org.


Banner Landing-
Page
Impressions Visitors
LP
Donations Sum Average LP-
Visitor/
Imp
Donation/
Imp
Donation/
LP-Visitor
Amount/
Imp
% Women
ctrl old LP 1141958 3262 2321 67780.22 29.20 0.002856 0.0020325 0.71153 0.0593544 18.97
ctrl new LP 1142043 3291 2451 68929.41 28.12 0.002882 0.0021462 0.74476 0.0603562 15.25

chi-squared-test donation/visitor: p = 0,002499

--Tobias Schumann (WMDE) (talk) 14:18, 20 January 2014 (UTC)[reply]


mobile Tests on m.de.wikipedia.org

[edit]

During the 2013 campaign we made an important step forward in diversifying sources of donations by displaying banners on mobile wikipedia (m.de.wikipedia.org). Furthermore, we started testing in this new and interesting field and succeeded in some significant improvements. Due to the lower numbers of visits and clicks, tests needed a longer duration for trustable results. That's why we only accomlished six tests during the campaign.


Test 1: sticky function

[edit]

Wednesday, 4.12.2013, 9:15 pm - Thursday, 5.12.2013, 9:15 pm UTC

Due to the sometimes suboptimal internet connection on mobile devices, loading of banners takes longer. Because of that, a lot of Wikipedia readers probalby don't see banners because they've already scrolled down on the website before the banner is being displayed. So we began testing on the mobile site by evaluating the sticky function. As expected the donation rate improved, however on a surprising level: +220%.


Banner Impressions Donations Sum Average Donation/
Imp
Amount/
Imp
not_sticky 1510100 42 318.00 7.57 0.0000278 0.0002106
information links - 0 0.00 0.00 0.0000000 0.0000000
sticky 1237500 108 705.00 6.53 0.0000873 0.0005697
information links - 2 28.00 14.00 0.0000000 0.0000000
  • p-value donation/impression = 0,00000



Test 2: text layout in drop down - facts text yellow vs. red

[edit]

Saturday, 7.12.2013, 5:00 pm - Tuesday, 10.12.2013, 5:00 pm

On small sized smartphone screens we tried to gain more attention for the facts texts by changing the banner background inside the drop from yellow to red. However, this didn't effect the donation rate at all.

Banner Impressions Donations Sum Average Donation/
Imp
Amount/
Imp
facts_yellow 4285600 279 1944.00 6.97 0.0000651 0.0004536
information links - 0 0.00 0.00 0.0000000 0.0000000
facts_red 4203800 277 1741.00 6.29 0.0000659 0.0004141
information links - 0 0.00 0.00 0.0000000 0.0000000
  • p-value donation/impression = 0,88676786



[edit]

This two tests focused on the layout of the mobile banner. Instead of a "donate now" button in the banner we tried a WMF orientated layout version with a blue link and the highlighted "If everbody...". By that the click rate droped down by about 70%, but the conversion inside the drop improved significantly by 294%. All in all the donation rate increased by about 12%. Unfortunately, between the two test groups a difference in letter size in the drop down text slipped in by the layout change. For the purpose of really trustworthy results, we repeated the test with similiar letter sizes in both test groups. The outcome gave us a double checked certainty to use the new layout - the test results were practically identical.

Friday, 10.12.2013, 8:15 pm - Monday, 13.12.2013, 3:15 pm UTC

Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
button L10h16 2677600 15522 320 186 1129.00 6.07 0.005797 0.000120 0.02062 0.0000695 0.01198 0.58125 0.0004216
info links - 0 29 0 0.00 0.00 - 0.000000 0.00000 0.0000000 0.00000 0.00000 0.0000000
link L10h16 2641000 4362 355 206 1286.00 6.24 0.001652 0.000134 0.08138 0.0000780 0.04723 0.58028 0.0004869
info links - 0 21 0 0.00 0.00 - 0.000000 0.00000 0.0000000 0.00000 0.00000 0.0000000

p-value donation/impression = 0,25160119


Monday, 13.12.2013, 3:15 pm - Thursday, 16.12.2013, 2:15 pm UTC

Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
button L10h16 3600000 21112 511 300 2130.00 7.10 0.005864 0.000142 0.02420 0.0000833 0.01421 0.58708 0.0005917
info links - 0 40 1 5.00 5.00 0.000000 0.000000 0.00000 0.0000000 0.00000 0.02500 0.0000000
link L10h16 3652800 5922 546 347 2324.00 6.70 0.001621 0.000149 0.09220 0.0000950 0.05860 0.63553 0.0006362
info links - 0 31 0 0.00 0.00 0.000000 0.000000 0.00000 0.0000000 0.00000 0.00000 0.0000000

p-value donation/impression = 0,09637155

Test 5: quantity donation amounts

[edit]

Thursday, 16.12.2013, 2:15 pm - Monday, 20.12.2013, 14:15

From a lot of tests we know how important it is to find the optimal quantity and sum of selectable donation amounts in the donation form. A higher number of donation amounts to choose can increase usability but also complexity of the form. On small mobile screens this problem has to be readressed. So we began with testing four (3|5|10|20) against six (5|15|25|50|75|100) amounts. (Due to technical novelty of mobile banners we couldn't implement a free field for donation amounts, yet.) While tests on desktop banners showed an increased donation rate through more amount buttons in the form, in the current test the donation rate was not affected. A different relationship of usability and complexity on small screens may be an explanation. However, the modification still was a success. The form with six amounts performed better because of a higher donation average.


Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
four_amounts L10h16 3879100 74882 523 284 2020.00 7.11 0.019304 0.000135 0.00698 0.0000732 0.00379 0.54302 0.0005207
information links - - 61 2 15.00 7.50 - 0.000000 0.00000 0.0000000 0.00000 0.03279 0.0000000
six_amounts L10h16 3870100 75166 536 285 2605.00 9.14 0.019422 0.000138 0.00713 0.0000736 0.00379 0.53172 0.0006731
information links - - 45 0 0.00 0.00 - 0.000000 0.00000 0.0000000 0.00000 0.00000 0.0000000



Test 6: text and highlight in drop down

[edit]

Monday, 23.12.2013, 4:00 pm - Tuesday, 01.01.2014, 00:00 Uhr

Text and layout of the donation appeal inside the drop down were tested this time. We compared two versions of the facts text against each other. The test version featured an additional text highlighting. That's why the test is strictly speaking a little bit inaccurate, but given the few tests we could realize our main goal was to find improvements of the banner. Our test version really improved donation/click by 8% compared to the control banner. Unfortunately, the result is not statistically significant but the p-value is close to the threshold value.

Banner Landing-Page Impressions Banner
Clicks
Visitors
LP
Donations Sum Average Click
Drop/
Imp
LP-
Visitor/
Imp
Visitor/
Banner-Click
Donation/
Imp
Donation/
Click-Drop
Donation/
LP-Visitor
Amount/
Imp
ctrl L10h16 10363700 199258 1291 735 7000.00 9.52 0.019227 0.000125 0.00648 0.0000709 0.00369 0.56933 0.0006754
information links - - 114 0 0.00 0.00 - 0.000000 0.00000 0.0000000 0.00000 0.00000 0.0000000
var L10h16 10344900 199802 1251 791 8322.09 10.52 0.019314 0.000121 0.00626 0.0000765 0.00396 0.63229 0.0008045
information links - - 118 5 190.00 38.00 - 0.000000 0.00000 0.0000000 0.00000 0.04237 0.0000000
  • p-value donation/click = 0,13141519 (chi-squared-test)


--Tobias Schumann (WMDE) (talk) 11:53, 5 May 2014 (UTC)[reply]