Fundraising 2011/Local testing/GB

Add links
From Meta, a Wikimedia project coordination wiki

Pre-FR tests[edit]

Test 1: System test[edit]

Results

Campaign: C_WMUK_systemtest_GB

  • Banner Impressions: 759,494 (B_WMUK_systemtest_GB)
  • Landing Page Views: 3590 (L11_WMUK_systemtest)
  • Click rate: .00473 (.473%)
  • Donation amount: £557.00
  • # of donations:33
  • Conversion rate: .000919 (0.0919%)

Comments: The test was successful: The payment processing worked flawlessly, the chapter was able to follow as donations came in and banners were displayed without any problems. Follow up: The 'PayPal reference code' was not passed on to the Chapter. WMF will follow up on this matter. Problem solved as of 26/07

Question - is it possible to get the number of people who clicked the "Donate" button on the landing page, so we can work out how many people dropped out when they got to the PayPal page? Thanks, The Land 23:03, 26 July 2011 (UTC)[reply]

Test 2: Stephen Creative[edit]

Campaign

http://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&method=listNoticeDetail&notice=C_WMUK_ENcreativetest_UK

Banner

http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B_WMUK_SWcreative_GB

LP 1 (Other Websites)

http://wikimediafoundation.org/wiki/L11_WMUK_SWcreative1/GB

LP 2 (Grandiose Plan's)

http://wikimediafoundation.org/wiki/L11_WMUK_SWcreative2/GB

Results

Ok so we had a few problems with the test so it was pulled

The "Other" amount was not being sent to paypal meaning people had to manually input their amount. Im not sure why this was never seen before as I was almost positive that this was working previously. This problem has been fixed with javacript code within the html to update the amount input rather.

Some tracking data (about one in five) wasn't being passed on to paypal. Thinking it might have been caused by the fact that the email option wasn't being passed along at all. Tracking data is being passed on now but still need to fix the email option here. Jseddon (WMF) 22:18, 5 August 2011 (UTC)[reply]

There's confusion in these texts between the WMF and WMUK, which needs clarifying (i.e. these letters need rewriting) before they are tested again. E.g. "and the bare bones staff who run the infrastructure and keep Wikipedia from getting sued off the Internet." - where does the WMUK's chief exec position fit in there? Ditto the 'People' section in the white box (which, incidentally, has a typo - "invesment" should have another t in it...). Mike Peel 06:32, 6 August 2011 (UTC)[reply]
Mike - I was happy to have these banners run as is as a test - though I quite agree they would need more work before they were used consistently in the campaign. The Land 10:36, 8 August 2011 (UTC)[reply]

Test 3: DD Landing Page systems text[edit]

  • LP Text: whatever the Foundation control is, but [1] works fine
  • Changes to LP: We need to integrate DD functionality onto the landing page. We need a simple button that allows people to select direct debit frequency (monthly or quarterly) and the amount they want to give, and then passes this information on to the forms built into our DD system. You shoul be able to find what we're doing in the source code here. Any questions on the technical aspects of this please ask Mike Peel.
  • Campaign name: UKDDTest
  • Banner: whatever the Foundation control is
  • Timing: 7pm GMT Weds 28th
Results

Campaign: UKDDTest

  • Banner Impressions: 808229
  • Landing Page Views: 13437
  • Click rate: 0.01662
  • Donation amount: Paypal £2,496.50; Monthly DDs (annualised) £60 Quarterly DDs (annualised) £120
  • # of donations: Paypal 191; Monthly DDs 2; Quarterly DDs 4
  • Average donations: Paypal £13.07; Direct Debit (both sorts) £30
  • Conversion rate: 0.0002437 (0.024%)

Test 4: Optimising Direct Debit vs Cash landing pages[edit]

  • Concept: Test what impact further DD optimisations have
  • Banner: Brandon
  • Campaign name: UKDDTest2
  • LP Text: Two landing pages, one DD optimised, one cash optimised, both based on this one.
    1. DD-optimised version. Text of appeal asks for a donation of £3, £5, £10 a month or whatever you can afford. In the donate bar there is a "Make a regular donation" with prompt amounts of £2, £3, £5, £10, £Other and a Frequency radio buttons of monthly or quarterly. Below this there is a radio button or check box saying "I'd prefer to make a one off donation". If this is checked then the £35, £10, £20, £50, £100 prompt amounts and the "donate via paypal/credit card" button appear on the landing page.
    2. Cash-optimised version. Text of appeal remains the same as here. Donate bar has the converse behaviour to above - so the donor is presented with the cash ask string and needs to check a box to get the Direct Debit options to appear.
  • Timing: 7pm GMT Weds 5th October
Results

This is really interesting stuff. The DD-heavy landing page got many fewer donations and a bit less cash. The value of a DD is 4x higher than that of a Paypal donation so if the DD optimisation were a bit better, we might see the DD-heavy page pulling ahead, even on a 1-year time horizon (Ideally we'd look at transactions costs and potential DD attrition as well, but I doubt they will massively change the picture)

  • Landing page A (Direct Debit optimised version): 59 Paypal (£878; avg £14.88) Direct Debits: 49 views -> 13 Direct Debits (10 Monthly, 3 Quarterly) annual value £540 (avg £41.50): 26.5% of DD page views finished DD process. Total income 1 year £1418
  • Landing page B (Paypal optimised version): 127 Paypal (£1334; avg £10.50) Direct Debits: 21 views -> 4 Direct Debits (3 Monthyl, 1 quarterly) annual value £208 (avg £52.00): 19% of DD pageviews finished DD process. Total income 1 year hence £1542
  • Overall average gift Paypal: £11.89 Average gift DD: £44.00

Test 5: GBP Ask Amount Tests[edit]

  • Objective: Establish optimal ask string for cash donations
  • Banner: Brandon
  • Campaign name: UKGBPCashAsk
  • Landing page: 3 landing pages. As per http://wikimediafoundation.org/wiki/L11_0510_UKDDTest2_B/GB/en (i.e. the non-Direct Debit optimised version from the previous tests) but with variant ask strings:
    1. £5, £10, £20, £50, £100, £200 - "lo" version (A)
    2. £10, £20, £30, £50, £100, £200 - "mid" version (B)
    3. £20, £35, £50, £75, £100, £250 - "hi" version (C)
  • Timings: Wednesday 12th 1830pm GMT for 90 minutes (90 minutes suggested as we are running 3 landing pages)
Results
  • Asking for larger amounts produced a statistically significant drop in the number of donations and an *apparent* drop in the total revenue. However, the income from each landing page is in fact not significantly different. We'd need a much longer test to establish that rigorously.
  • Landing Page A: 123 donations, total 1570, avg £12.76 net total 1499.59 (nb one £200 outlier: excluding, total 1420, avg £11.54)
  • Landing Page B: 98 donations, total 1386, avg £14.14 net total 1326.15 (nb one £100 outlier: excluding, total £1336, avg £13.63)
  • Landing Page C: 68 donations, total 1262.10, avg £18.56 net total £1213.04 (no gifts >£50)

Test 6: UK DD Test 3[edit]

  • Objective: Does promoting regular giving on the banner impact on Direct Debit sign-up rates?
  • Campaign name: UKDDTest3
  • Banner: 2 banners. Banner A: Standard Brandon (as we have been using). Banner B: As A, with addition of small text under banner text, "Just £3 a month can keep Wikipedia free".
  • Landing page: 2 landing pages. Landing page A http://wikimediafoundation.org/wiki/L11_0510_UKDDTest2_A/GB/en (i.e. the Direct Debit optimised version from the previous tests). Landing page B: As A, but replacing text "Make a regular donation" with text " Just £3 a month will keep Wikipedia free".
  • Banner A -> Landing page A. BAnner B -> Landing page B
  • Timings: Sunday 16th 1730pm GMT (open to negotiation if Sunday is too much hassle ;-)
    • Could the banner text be modified to include "help" - i.e. "Just £3 a month can help keep Wikipedia free", since the overall costs per year for Wikipedia are somewhat in excess of £36. ;-) Mike Peel 13:44, 9 October 2011 (UTC)[reply]
    • Also: running this on a different day from the EGM might be a good idea, to ensure that the appropriate people can be online at the time... (presumably not a problem for CK; might be an issue for RS.) Mike Peel 13:46, 9 October 2011 (UTC)[reply]
Sunday after the EGM is the least worst option in my diary. Strictly, only one of us needs to be around. The Land 19:28, 12 October 2011 (UTC)[reply]
Results
  • Ended up occurring at 1850 BST on Saturday 15th.
  • It appears that asking more explicitly for £3 a month Direct Debits gives you a) more Direct Debits, b) more monthly Direct Debits, c) bigger DirectT Debits on the whole - but stil waiting for statistical confirmation of some of those.
  • Banner A -> Landing Page A: PayPal 57 donations total £633 average £11.01. Direct Debit 48 impressions -> 13 DDs (8 monthly, 5 quarterly); annual value £448. TOTAL £1081
  • Banner B -> Landing Page B: Paypal 18 donations total £289 average £16.06 Direct Debit 59 impressions -> 17 DDs (13 monthly, 4 quarterly); annual value £944 TOTAL £1233

Possible next steps:

  1. Aim to improve the DD page -> complete DD setup rate - what more can we do (API rather than Smartdebit form; provide more reassurance text on page)
  2. Test ask amounts (our DD amounts were much better than those in the last test, but this would bear further testing; is a £2 a month prompt a good idea?)
  3. Test removing/de-emphasising the Quarterly DD option
  4. Test a banner with a Direct Debit ask -> landing page with further DD language and motivation vs regular banner with "cash" landing page

Test 7: UK DD Test 4[edit]

Objectives

We want to do a further test to observe differences between cash-optimised appeals/landing pages and direct debit-optimised appeals/landing pages. We also want to explore whether the person and the time of day matter.

Test details
Once the test is finished we will compare BA+JA with BB+JB to check (again) the effect of cash versus Direct Debit optimised pages. We will also compare BA+BB to JA+JB to see if people prefer Brandon or Jimmy on the whole.
  • Timing 1200 to 1330 GMT and 1830 to 2000 GMT on Wednesday 19 Oct 2011. The time is split so that we get both "work" and "evening" traffic to the site. I want a long test as the tradeoff between Direct Debits and cash donations is a key piece of information for us. We also need to see if there is any community or donor resistance to us asing heavily for Direct Debits.
Results

Results are here In summary, the DD versions received a noticeably lower click-through from the banner, but a higher Direct Debit income.

Test 8: UK DD Test 5[edit]

Objective

Test whether the text on the banner about setting up a Direct Debit is important, or whether the setup of the landing page is the important feature. Secondary objective; see whether alternative text can work without producing negative feedback from "Give X to keep Wikipedia free" wors.

Setup
  • Campaign setup: 2 banners to 1 landing page.
  • Banner A. Standard Brandon "personal appeal" banner
  • Banner B. Brandon "personal appeal" banner with small text below banner "Can you donate £3 a month to keep Wikipedia free?"
  • Landing Page A: As per DD version from the last test.
  • Scheduling: Wednesday 26 October, 1830 to 2000 GMT
Results

Results are here. The banner with DD text resulted in fewer landing page views, and fewer Paypal donations, while producing no more Direct Debit donations. However, it did result in a higher proportion of those starting the DD process completing it.

Comments
  • Again, "help keep" please... Also, would it be worth adding "for everyone" on the end, or something along those lines, to further steer away from making it seem like readers need to donate to continue reading? ;-) Mike Peel 14:26, 23 October 2011 (UTC)[reply]

Test 9: UK people[edit]

Objective

Establish which of the people work in the UK. We won't necessarily get firm info from this test on the £ per donation per person, but we will get information on banner clickthroughs and donation rates.

Timing

Tuesday 8th November 0830 to 2030 GMT (tbc)

Structure

Banner 1 -> Landing Page 1, Banner 2 -> Landing Page 2 ... so the banner from each person is associated with that landing page.

Banners

(NB not with "£3 a month can help keep Wikipedia free")


Landing Pages

These should all be in the basic format used in Test 8, so LPs 3-6 will need updating with the standard UK changes, e.g. with UK donation options, prompt amounts in text -> "£3 a month"

Results

Full results here. Winners were Brandon, Susan and Jimmy - Jimmy gets a bit more volume, though not necessarily more income. Interesting that neither Alan nor Maryana did as well as they appeared to do when tested in the US? The Roger appeal needs more work to be a contender.

Tests during Fundraiser[edit]

Paypal vs DD Optimised Landing Pages re-test[edit]

When
1200 Nov 21st to 1200 Nov 22nd
What
We wanted to re-test the idea that asking to people to give a monthly Direct Debit is a helpful thing to do and doesn't reduce the income you get. We had already got a fair picture of this from some previous testing but since it was a key factor for us, we went ahead.
Results
Results are here
Conclusion
The DD variant of the form worked just as well as the Paypal-heavy version of the form. The DD version attracted fewer donations but more Direct Debits, and the higher average value of the Direct Debits compensated for the lower number of donations. Since a donor paying b Direct Debit has a much higher long-term value, we continued to mainly ask for Direct Debits.

3 vs 4 Direct Debit amounts test[edit]

When
1200 Nov 23rd to 1200 Nov 24th
What
We wanted to test whether changing the Direct Debit prompt string from "£2, £3, £5, £10" to "3, £5, £10" had any impact, as £2 a month direct debits have higher transaction costs than we really wanted.
Results
Results are here
Conclusion
The 3 asks version worked just as well as the control. Oddly enough the change in Direct Debit asks appeared to have a higher impact on on the Paypal donations - the 3 prompt level version, with no £2 a month prompt, produced fewer Paypal donations, though of a higher value. However, the difference in paypal donation rate was not statistically significant.

Jimmy vs Brandon vs Susan[edit]

When
2 December, all day
What
The Foundation had produced a revised Susan appeal. We'd found that the Susan appeal we used at launch had tailed off quite a bit and we had given that appeal a rest. But we were keen to try the new Susan appeal.
Results
are here
Conclusion
While 2 December was quite a disappointing day in terms of income on the whole, Susan's appeal didn't seem to be performing any worse.

Pictures on Landing Pages[edit]

When
3 December, all day
What
Wikimedia DE's testing appeared to show that putting pictures on landing pages increased income. We wanted to try to replicate this result.
Results
are here.

Conclusion: For Brandon and Susan appeals, adding a picture to the landing page appeared to suppress results. But not for Jimmy. However, it might be that the issue is that we have a photo as well as the "where your donation goes" text box, so we will re-test this without the text box.

When people visit the landing pages, they will have already seen the picture of the person making the appeal since it's on the banner. So the picture/person will already be in their minds when they are reading the text - which reduces the benefit of having the picture on the landing page as well. I wonder: does this situation change when the landing page picture is different from the banner appeal, or reveals more of the background/context? Mike Peel 00:47, 5 December 2011 (UTC)[reply]
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Fundraising_2011/Local_testing/GB&oldid=5435318"