Fundraising 2010/Report/Draft Test 9
Banner Test - Bold Border Compare[edit]
For additional documentation on the testing methodology please see the following pages:
Banners[edit]
Result[edit]
Test Time: 2010-12-16 23:30:00 UTC - 2010-12-17 00:10:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 40 minutes
Total Number of Samples per Class = 20
TEST RESULT INCONCLUSIVE
(*) The rate of donations per banner impression over a fixed time interval.
(**) The rate of amount50 per banner impression over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.
Data Analysis[edit]
This section analyzes and interprets the results of the tests.
Data Consistency and Cleaning[edit]
The plots below display the counts of the data sources over the testing period as verification of the consistency of the donation pipeline data used in testing. It is of note that there are a couple of intervals where the impressions dipped down to zero, however these were sparse enough not to effect the quality of the data significantly. It should also be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.
It seems that several impressions were being seen for the test campaign on the "Heavy Border" banner that are in excess of what should be there. This likely has something to do with impressions coming from a concurrently running campaign on which this same banner was also featured. The bottom-most plot shows the impressions from the other campaign which may be compared to the impressions from the test campaign. The test period was therefore chosen to cut out the strange impression results although there appears to exist at least a small amount of noise through the entire period.
Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-16 23:30:00 UTC - 2010-12-17 00:10:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.
Modelling and Hypothesis Testing[edit]
"Light Border" won in each case for donations/impression and amount50/impression with increases of 2.60% and 14.59% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/impression and amount50/impression is at least 60.0% and 75.0% respectively. This result is not significant and the banners perform effectively the same.
TOTAL DONATIONS "Light Border": 79 TOTAL DONATIONS "Heavy Border": 78 TOTAL AMOUNT50* RAISED "Light Border": $1755.00 TOTAL AMOUNT50* RAISED "Heavy Border": $1506.23 * AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.
DONATIONS PER IMPRESSION: Between 60.0% and 75.0% confident about the winner. Bold Border Compare -- 2010-12-16 23:30:00 - 2010-12-17 00:10:00 item 1 = Heavy Bold Border item 2 = Light Bold Border The winner "Light Bold Border" had a 2.60% increase. interval mean1 mean2 stddev1 stddev2 0 0.00011 0.00012 0.00003 0.00006 1 0.00012 0.00012 0.00003 0.00005 Overall Parameters: mean1 mean2 stddev1 stddev2 0.00012 0.00012 0.00003 0.00005
AMOUNT50 PER IMPRESSION: Between 75.0% and 90.0% confident about the winner. Bold Border Compare -- 2010-12-16 23:30:00 - 2010-12-17 00:10:00 item 1 = Heavy Bold Border item 2 = Light Bold Border The winner "Light Bold Border" had a 14.59% increase. interval mean1 mean2 stddev1 stddev2 0 0.00202 0.00238 0.00089 0.00156 1 0.00247 0.00276 0.00086 0.00037 Overall Parameters: mean1 mean2 stddev1 stddev2 0.00225 0.00257 0.00088 0.00113
Endnotes[edit]
- Campaign = "20101216JA029"
- "Animated Progress Meter" utm_source = "20101216_JA013B_US"
- "Static Progress Meter" utm_source = "20101216_JA013C_US"