Fundraising 2010/Report/Draft Test 1

From Meta, a Wikimedia project coordination wiki

Landing Page Test: "Where Your Donation Goes Info" Info Box[edit]

For additional documentation on the testing methodology please see the following pages:


Landing Pages[edit]

"Where your money goes (Info Box)".



"Donate button (no box)".

Result[edit]

Test Time: 2010-12-30 20:56:00 UTC - 2010-12-30 21:54:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 8 minutes
Total Number of Samples per Class = 29

Donations / View*: "Info box" landing page won by 13.71%.
Amount50 / View**: "Info box" landing page won by 40.12%.

WINNER: "Info box" Landing Page


(*) The rate of donations per banner view over a fixed time interval.

(**) The rate of amount50 per banner view over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.

Data Analysis[edit]

This section analyzes and interprets the results of the tests. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.

Data Consistency and Cleaning[edit]

Views broken out in two minute intervals over the test period.


Donations broken out in two minute intervals over the test period.


Donations/View broken out in two minute intervals over the test period.


Amount50/View broken out in two minute intervals over the test period.

Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-30 20:56:00 UTC - 2010-12-30 21:54:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.

Modelling and Hypothesis Testing[edit]

The plots below depict the mean and standard deviation of each banner over four ten minute testing intervals. Samples were taken over two minute periods. The hypothesis testing was performed over the average mean and variance of the test intervals. The landing page featuring a "Info box" won in each case for donations/view and amount50/view with increases of 13.71% and 40.12% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/view and amount50/view is at least 97.5% in each case. It is notable that the increase in amount50/view is more than donations/view. There are two potential causes here, (1) the variability of amounts tends to be much greater than donations (since donations are binary and amounts are values subject to a relatively wide range), (2) the winner actually had an affect on engaging higher amount donors.

Mean and standard deviation over test intervals over Donations / View.


Mean and standard deviation over test intervals over Amount50 / View

TOTAL DONATIONS "Where your money goes (Info box)": 349
TOTAL DONATIONS "Donate button (No box)": 320

TOTAL AMOUNT50* RAISED "Where your money goes (Info box)": $6491.59
TOTAL AMOUNT50* RAISED "Donate button (No box)": $4749.00

* AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.


DONATIONS PER VIEW:

Between 97.5% and 99.0% confident about the winner.

Pie Box VS No pie box -- 2010-12-30 20:56:00 - 2010-12-30 21:54:00

item 1  = No box
item 2  = Pie box

The winner Pie box had a 13.71% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.02306		0.05517		0.01493		0.01316
1		0.05379		0.07330		0.04721		0.01289
2		0.07057		0.06291		0.00842		0.02517
3		0.04835		0.06040		0.03297		0.02013
4		0.04772		0.05847		0.00494		0.03185
5		0.07303		0.07110		0.03519		0.02404
6		0.07154		0.05990		0.01473		0.03591


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.05544		0.06303		0.02696		0.02467


AMOUNT50 PER VIEW:

Between 97.5% and 99.0% confident about the winner.

Pie Box VS No pie box -- 2010-12-30 20:56:00 - 2010-12-30 21:54:00

item 1  = No box
item 2  = Pie box

The winner Pie box had a 40.12% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.28990		0.86432		0.24925		0.23407
1		0.98852		1.56268		0.95563		0.69020
2		0.96921		1.06585		0.03701		0.68849
3		0.64897		1.31208		0.49512		0.26510
4		0.78157		1.08586		0.04731		0.43027
5		1.11967		1.23437		0.66021		0.29319
6		1.03137		1.04250		0.17909		1.25092


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.83274		1.16681		0.49165		0.64491

Endnotes[edit]

  1. Campaign = "20101230JA096_US"
  2. "Info box" landing page = "WMFJA073"
  3. "No box" landing page = "WMFJA074"