Learning and Evaluation/Evaluation reports/2013/WLM

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
This page contains changes which are not marked for translation.


Wiki Loves Monuments
This page is part of the Evaluation Report (beta). Important background information can be found on the Overview page.

This is the Program Evaluation page for Wiki Loves Monuments. It currently contains information based on data been collected in late 2013 and will be updated on a regular basis. For additional information about this first round of evaluation, please see the overview page.

This page reports data for three program leaders, about a total of 27 Wiki Loves Monuments implementations, which includes 24 that were mined for additional data. Data requests for this survey conflicted with the end of Wiki Loves Monuments 2013, therefore we had to supplement data ourselves, collecting data for 24 implementations using publicly accessible data and tools. We also had program leaders representing those 24 implementations confirm data per our request.

Key lessons include:

  • The more participants, the more photos uploaded; however, the more money spent to implement a Wiki Loves Monuments event doesn't equal higher participant counts or more photo uploads. Just because an event invests more money in implementing their Wiki Loves Monuments, doesn't mean they'll attract more participants, or result in a higher upload count than those events that have lower budgets.
  • A very small percentage of images (0.08%) from Wiki Loves Monuments 2012 are rated as Quality, Valued and/or Featured pictures, and at time of data collection, it was too soon to know quality ratings for Wiki Loves Monuments 2013. The small percentage of quality ratings makes it impossible at this point in time to determine what the overall quality of the produced content is.
  • About 17% of the images uploaded through Wiki Loves Monuments 2012 are in use on our projects. These are the photos that are getting the most views and therefore have the most impact on our readers. We don't know at this point how many of the pictures are getting re-used on other websites, so the actual usage might be higher.
  • The majority of Wiki Loves Monuments participants are new users; however, the survival rate of new users is low (1.7% of the 2012 participants made at least one edit and 1.4% uploaded at least one new file to Commons six months after the event). Not surprisingly, existing editors who participate in Wiki Loves Monuments are retained at higher numbers.
  • Half of the existing editors who participated in Wiki Loves Monuments 2012 also participated in Wiki Loves Monuments 2013, with new users making up for the other half. While new user retention is low for 2012, new users help offset the existing editors who didn't participate in Wiki Loves Monuments 2013.
  • The global Wiki Loves Monuments organizing team helps support Wiki Loves Monuments organizers around the world, providing replication opportunities via direct support. This makes it easier for someone to organize their own Wiki Loves Monuments contest.
  • It's obvious that Wiki Loves Monuments is successful at increasing the number of freely licensed images of historic monuments, but is it successful at educating participants about open knowledge and free licensing? We suggest better initiatives to educate participants, and qualitative surveying to explore if Wiki Loves Monuments is indeed successful at encouraging known support and education about these two areas, which was a selected priority goal.


Planning a Wiki Loves Monuments or other photo upload event? Check out some process, tracking, and reporting tools in our portal and find some helpful tips and links on this resource page

Program basics and history[edit]

Wiki Loves Monuments logo

Wiki Loves Monuments is the world's largest photo competition. The goal of Wiki Loves Monuments is to document the entirety of historic buildings and places around the world through photographs, which are then uploaded to Wikimedia Commons. The project, which was first held in the Netherlands in 2010, is held every September, lasting the entire month. Wiki Loves Monuments uses government generated lists of historic monuments and places to create on-wiki lists that guide participants to monuments and places needing photographs. Participants from more than 50 countries (as of 2013) photograph the listed buildings and places and then upload the photos under free licenses to Wikimedia Commons. On wiki, volunteers categorize and might distribute the images to appropriate Wikipedia articles.

Each country that participants in Wiki Loves Monuments has a jury that reviews each photograph submitted for their specific country. Juries usually comprise of professional photographers, cultural heritage academics, and Wikimedians. The images are reviewed base on quality and value of the image. Generally, the jury selects the 10 best photographs submitted within their country. Prizes are often awarded to the top three, and sometimes, exhibitions are held for the winning photos. Each country submits their top ten images to the international Wiki Loves Monuments jury, who then reviews the images and awards prizes for the winners. There are also special awards, which are sponsored by organizations. Prizes vary, from a grand prize trip to Wikimania 2014 in London, to photography related gear and money.

Data report[edit]

Response rates and data quality/limitations[edit]

Three program leaders submitted data, and we mined data for 24 additional WLM implementations.

Collecting Wiki Loves Monuments data for 2013 was particularly challenging, since our request for data collection responses fell during the time period when Wiki Loves Monuments 2013 was being completed. Only three program leaders reported a Wiki Loves Monuments implementation directly to us. To supplement the need for data, we collected data from an additional 24 Wiki Loves Monuments implementations from 2012 and 2013 which had received grants through the Wikimedia Foundation, including FDC funding. These implementations included specific budgets for each Wiki Loves Monuments contest. In total, we evaluated 27 Wiki Loves Monuments implementations: 12 from 2012 and 15 from 2013.

Year of Wiki Loves Monuments Number of implementations used for this report
2012 12
2013 15
Total 27

For the 24 mined Wiki Loves Monuments implementations, we used publicly available information that was found on organizer websites and on wiki. We were able to collect information about: number of participants, number of photos added, number of photos used in the projects, and ratings of photo quality (Featured, Quality, Valued). We were unable to determine priority goals or input hours through this process of mining the data, but we did contact program leaders for the 24 mined Wiki Loves Monuments implementations to review and confirm the numbers gathered, and to share whether they had used additional funds or any donated resources. However, since we did not gather the complete set of reporting questions from those program leaders whose implementations were mined, we also do not know there program status in terms of shared learning. Still, this report does not represent findings from all Wiki Loves Monuments events, and does not reflect the value of the many hours of volunteer time that are input into planning and implementing these events.

Report on the submitted and mined data[edit]

Priority goals[edit]

Program leaders for both Wiki Loves Monuments and other photo upload initiatives reported twelve priority goals, with the two most popular focusing on bringing awareness to more people about Wikimedia projects and having fun.

Since Wiki Loves Monuments has similar goals to other photo-based competitions, the goals of these were analyzed together. Five Program Leaders reported their "priority" goals for their photo upload events (100% of those providing direct report). Participants selected between 4 and 14 priority goals for their upload events. [1] Looking at the aggregate selection of priority goals for content donations, only two goals stood out as priority across implementations, however, many were selected by the majority of the five reporting program leaders. The top two goals, selected by four of the five program leaders, were to (1) Increase awareness of Wikimedia and its projects and (2) Make contributing fun, although, clearly, increasing contributions, in terms of photo content, is very much a core target of all these events (see Table below).

Photo Upload Competitions Goals.png

Inputs[edit]

In order to learn more about the inputs that went into planning workshops, we asked program leaders to report on
  • The budget that went into planning and implementing the workshop
  • The hours that went into planning and implementing the workshop
  • Any donations that they might have received for the event: a venue, equipment, food, drink, giveaways, etc.
Budget[edit]
Reported budgets were available for the majority of Wiki Loves Monuments contests. The average national contest costs just over $3,600 US to implement.

Reported budgets were provided for 26 Wiki Loves Monuments implementations (96%). One event, directly reported, indicated only volunteer hours. Budgeted costs for Wiki Loves Monuments ranged from $392.05 to $29,101.59 with an average[2] of $3,614.50. [3]

Hours[edit]
Too few program leaders and implementations reported staff and volunteer hours for us to provide an evaluation for this pilot report.

Due to a low response rate by program leaders on this pilot survey, we are unable to report a solid evaluation on input hours for Wiki Loves Monuments at this time. We obtained data on staff input hours for only one, and volunteer hours for just two, directly reported WLM implementations.

Donated resources[edit]
The majority of Wiki Loves Monuments implementations used donated resources, with meeting space and prizes/give-aways being the most common.

The majority of Wiki Loves Monuments implementations (78%) reported whether they had used any donated resources. More than half of the implementations reported using donated meeting space (62%) and donated prizes/give-aways (52%). In addition, 19 percent reported using materials/equipment that were donated and 14 percent used food donations (see Graph 1).

Graph 1: Use of donated resources.
The donated resources most often reported for Wiki Loves Monuments were meeting space (62%) and prizes or give-aways (52%). Wiki Loves Monuments events appear to benefit often from the use of donated resources to carry out the events.

Outputs[edit]

We also asked about two outputs in this section
  • How many participants contributed to Wiki Loves Monuments?
  • How many of those participants were new users?

We used this output and budget inputs to learn:

  • What was the dollar cost for each participant
Participation[edit]
The average Wiki Loves Monuments implementation has 190 participants. For 2012, the majority of participants were new users, making up 87% of the participant base.

We were able to collect participant numbers for all 27 Wiki Loves Monuments implementations. The number of participants in the national contests ranged from 67 to 2,005 people. The average national contest has 190 participants.[4] The majority (87%) of users who participated in Wiki Loves Monuments 2012 were new users. We were unable to gather this information for Wiki Loves Monuments 2013, since the contest was just ending as we were analyzing the data.

On average, Wiki Loves Monuments implementations cost about $14.50 USD per participant.

Budget and participant count was available for 26 of the 27 implementations used in this evaluation. Looking at programming costs in terms of the number of participants, we learned that programming costs ranged from $1.20 USD to $326.98 USD and averaged $14.58 USD per participant.[5]

Outcomes[edit]

Content Production and Quality Improvement[6][edit]
The average national Wiki Loves Monuments contest leads to just over 5,600 photos uploaded to Commons during the month of September. For Wiki Loves Monuments 2012, the average number of photos placed in Wikimedia projects was 885.

We collected photo upload counts for all 27 WLM implementations. Upload counts ranged from 1,572 to 32,987 (Graphs 2 and 3). The average event uploaded 5,635[7] photos during the month of September 2012 or 2013.

Regarding image use:

  • Wiki Loves Monuments 2012 data for image use ranged from 387 to 6,103 unique images used. The average number of images used from a 2012 Wiki Loves Monuments event was 885.[8] 2012 also shows us that the more images uploaded, the more images are used (see Graph 4).
  • Wiki Loves Monuments 2013 data for image use ranged from 10 to 1,687 unique imaged used. The average number of unique images used from a 2013 Wiki Loves Monuments event was 262[9][10] (see Graph 5).

Do note that the numbers reported for 2013 are not final. Since the event ended around the time that we were collecting data, we expect image use in Wikimedia projects to rise.

The average Wiki Loves Monuments photo upload costs almost 50 cents USD. However, more money spent on a national Wiki Loves Monuments event does not necessarily relate to more participants or more image uploads.

Based on reported budgets for Wiki Loves Monuments implementations and the actual photo uploads, the amount of dollars spent for each photo uploaded ranged from $0.09 USD to $9.17 USD each. The average implementation cost $0.48 USD [11] per photo. We also looked at how much it cost in terms of Wiki Loves Monuments photos which were used on Wikimedia projects. For 2012 implementations, the cost per used photo ranged from $0.51 USD to $26.51 USD, averaging $2.43 USD[12] per image. There does not seem to be a direct relationship between budget, participant count, or images uploaded (see Graphs 2-5).

Graph 6: Cost per photos added.
This boxplot shows the distribution of 2012 and 2013 Wiki Loves Monuments competitions. In comparing between the two years, we see that the dollar cost per photo range was greater in 2012 than in 2013; however, the median cost was lower for 2012 at $0.37 per photo added. This suggests that spending per photo between both years may not be very different. More data would be needed to confirm.
Photo quality ratings[edit]
Only a very small percentage (0.8%) of Wiki Loves Monuments 2012 images are rated as Quality, Valued and/or Featured Images on Wikimedia Commons.

This section focuses only on images uploaded for Wiki Loves Monuments 2012, since the 2013 competition had was just ending as we finished of our data collection; not enough time had passed for ratings to have been awarded at that time.

By November 2013, about 13 months after the 2012 contests ended, out of the 12 contests examined, five (42%) had Quality images, three (25%) had Valued images, and most (7 contests, 58%) had Featured Pictures associated with them. Out of the 126,424 photo uploads we examined for Wiki Loves Monuments 2012, 994 (0.8%) were rated as a Quality Image, Valued Image, or Featured Picture (see Graph 8). Out of those 994 rated photos[14]:

  • 932 (94%) were rated as Quality Images. Reports of images rated as Quality ranged from 0 to 484 with the majority (91%) of Quality images generated by only two contests, one which contributed 484 (52%), and another which contributed 364 (39%).
  • 30 (3%) were rated as Valued Images. Reports of images rated as Valued ranged from 0 to 27 with the majority (90%) of Valued images generated by one contest.
  • 32 (3%) were rated as Featured Pictures. Reports of Featured Pictures ranged from 0 to 17 and were associated with several of the contests.

Importantly, that the majority of images rated Quality and Valued generating from only one or two contests, indicating that those program leaders are likely doing something that others are not in order to achieve these ratings.

Recruitment and Retention of New Editors[edit]

As of reporting time, we were only able to evaluate recruitment and retention of active users for 2012. A few things to remember with this section:

  • An active retained user is considered one that makes 5 or more edits a month. [15] We examined both the "pages created" (a new page gets created when someone uploads a file) and "edit count" using Wikimetrics to measure user activity on Wikimedia Commons.
  • A "survived" editor is defined as one that made at least one edit in a given window of time after the event ended.
  • User retention was measured for both survival and active user retention. Retention was assessed for the first three months after the event (Quarter 1) and the second set of three months (Quarter 2: months four, five, and six) following the event. Here we considered users as surviving if they had made a single contribution, and "active" if they had averaged [16]5 or more edits a month, during the follow-up window.
  • We couldn't isolate only Wiki Loves Monuments uploads so user contributions and retention numbers are based on any contributions made by participants during September 2012, and the three and six month retention dates.
  • Three month follow-up represents the first quarter following the event, from October to December 2012.
  • Six month follow-up represents the second quarter following the event, from January to March 2013.
New users[edit]
Six months after the end of Wiki Loves Monuments 2012, 0.4% of new users were retained as active contributors to Wikimedia Commons.

Due to Wiki Loves Monuments 2013 ending recently, recruitment and retention data for only 2012 competitions is reported here. Our retention period of 3 months examines the number of uploads over the 3-month period from October to December 2012 while the 6-month retention period lasts through March 2013. The count of uploads per user is pooled; that is, the number of uploads from October through December of 2012 are counted together. Thus, in order to measure 5+ edits per month, we actually use a limit of 15 edits over three months for the 3- and 6-month follow-up point.

For these data, we used Wikimetrics to calculate the number of pages created in namespace 6 for files of Wikimedia Commons. The pages created option in Wikimetrics includes not only photos uploaded, but may also count videos and audio files added. We are unable to isolate just photos uploaded from these other possible media types, so user activity may include these as well. We also could not isolate photos uploaded for Wiki Loves Monuments, versus for other reasons, during the month of September so that the counts associated with the month September are not necessarily photos for the contest. For instance, in 2012, this made for the observation of 399,123 pages created by the full cohort of new and existing users while the actual number of photos uploaded specifically for Wiki Loves Monuments 2012 was 365,322, 8.4% of the pages created by the cohort of contributors were not photos for Wiki Loves Monuments. In addition to pages created, we also examined Wiki Loves Monuments participant behavior in terms of their edit count on Commons. Edit counts may include creation of a new file page, edits to category or other information on an existing file, or new versions of the image that may be uploaded.

13,179 new users[17] participated in Wiki Loves Monuments 2012. 5,399 (41%) of these uploaded five or more files and 5,552 (42%) made five or more edits in September 2012. In regard to retention (see Graph 9):

  • Three-month follow-up (First quarter: October–December 2012)
Survival (Any contributions)
  • 797 (6%) of new users from Wiki Loves Monuments 2012 had made at least one edit and 627 (4.8%) had uploaded at least one new file to Wikimedia Commons.
Active (Average 5+ contributions a month)
  • 152 (1.2%) of new users from Wiki Loves Monuments 2012 uploaded 15+ files during the first quarter of retention follow-up.
  • 221 (1.7%) of new users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the first quarter of retention follow-up.
  • Six-month follow-up (Second quarter: January–March 2013)
Survival (Any contributions)
  • 220 (1.7%) of new users from Wiki Loves Monuments 2012 had made at least one edit and 187 (1.4%) had uploaded at least one new file to Wikimedia Commons.
Active (Average 5+ contributions a month)
  • 47 (0.4%) of new users from Wiki Loves Monuments 2012 uploaded an average of 15+ files during the second quarter of retention follow-up.
  • 59 (0.4%) of new users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the second quarter of retention follow-up.

Although new user activity declined over the three and six month follow-up periods, during the month of Wiki Love Monuments in 2013, there was an increase in activity for the new users from Wiki Loves Monuments 2012. In September 2013, 451 of the previous new user cohort (3.4%) contributed photos/media to Commons. Of those previously new users which were active that month, 306 (68%) added five or more files during the 2013 contest time.

Existing users[edit]
Not surprisingly, existing editor retention is higher than new user retention for Wiki Loves Monuments, with 29% actively uploading files and 57% actively editing on Wikimedia Commons six months after the event ended. Half of the existing users who participated in Wiki Loves Monuments 2012 also participated in Wiki Loves Monuments 2013. The number of returning existing contributors seen for Wiki Loves Monuments 2013 was, for the most part, replenished by those Wiki Loves Monuments 2012 contributors who were retained as near the same number of existing users who participated in 2012 did not return for 2013's competition.

1,632 existing users participated in Wiki Loves Monuments 2012. 1,236 (76%) of these uploaded five or more files and 1,307 (80%) made five or more edits during the contest in September. In regards to retention (see Graph 10):

  • Three-month follow-up (First quarter: October–December 2012)
Survival (Any contributions)
  • 1,073 (66%) of existing users from Wiki Loves Monuments 2012 had made at least one edit and 975 (60%) had uploaded at least one new file to Wikimedia Commons.
Active (Average 5+ contributions a month)
  • 566 (35%) of existing users from Wiki Loves Monuments 2012 uploaded 15+ files during the first quarter of retention follow-up.
  • 759 (47%) of existing users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the first quarter of retention follow-up.
  • Six-month follow-up (Second quarter: January–March 2013
Survival (Any contributions)
  • 934 (57%) of existing users from Wiki Loves Monuments 2012 had made at least one edit and 845 (52%) had uploaded at least one new file to Wikimedia Commons.
Active (Average 5+ contributions a month)
  • 477 (29%) of existing users from Wiki Loves Monuments 2012 uploaded an average of 15+ files during the second quarter of retention follow-up.
  • 643 (39%) of existing users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the second quarter of retention follow-up.

Half of the cohort of existing users (811 users) contributing to Wiki Loves Monuments 2012 also uploaded media during Wiki Loves Monuments 2013. Of those 652 users (80%) uploaded five or more new files to Wikimedia Commons during that time. While new editor retention appears low, compared to existing editor retention, the number of new users that returned for Wiki Loves Monuments 2013 (457 previously new users) helps to replenish the pool of returnees for 2013.

Summary, suggestions, and open questions[edit]

How does the program deliver against its goals?[edit]

One of the goals for Wiki Loves Monuments is to increase the number of freely available photos of historical monuments. The program's success in this area is obvious. Over the years, Wiki Loves Monuments has produced an impressive number of images under free license that can be used to illustrate Wikipedia articles, as seen in this table:

Year Total number of images
uploaded to Wikimedia Commons
through Wiki Loves Monuments
2010 12,626
2011 169,150
2012 365,335
2013 372,156

The sheer amount of new images that got freely available is amazing and it's also very commendable that the global Wiki Loves Monuments organizers were continuously able to broaden the list of countries that participate in the contest.

When it comes to the other goals for Wiki Loves Monuments, like e.g. "making contributing fun", things get more complicated. The number of returning existing contributors seen for Wiki Loves Monuments 2013 indicates that a good amount of long-term Commons photographers enjoyed the 2012 event enough to come back a year later. But a new user survival rate of 1.7% six months after the end of Wiki Loves Monuments 2012 is troubling. Event organizers have crafted a popular event that attracts new contributors to Wikimedia Commons, a major hurdle for the Wikimedia movement throughout all the projects, and they should be commended for doing so. But just like textual contribution campaigns, this image contribution campaign suffers from issues prevalent throughout the Wikimedia projects. Without a significant investment in technical resources to make the contributing and interacting interface on Wikimedia Commons better and community consensus to make Commons more welcoming to newcomers, it's hard to imagine image upload campaigns like Wiki Loves Monuments will achieve a higher retention rate of new contributors. Such challenges within the Wikimedia Commons project are not the fault of event organizers, however, and certainly the significant amount of photos uploaded through Wiki Loves Monuments demonstrates the competitive elements of the photo contest and the act of finding locations and taking and contributing photos indicates they do well at making participating in the Wiki Loves Monuments competition fun, even if the challenges of Wikimedia Commons' user interface and workflows removes enough of the fun that contributors don't stick around.

One of the Wiki Loves Monuments goals that 60% of our survey respondents reported to us is to "Increase the support for open knowledge and the free content movement". Given the number of people who participated in the contest and who had never before released their images under a free license is an indicator of general support for free knowledge, it is unclear what other program mechanics have been put in place in order to achieve a change in attitude among the participants. However, this area offers a lot of potential. For example, it might be possible to include information about the benefits of free licenses in the upload process (maybe at the end of the upload, in order to encourage participants to upload even more images under a free license) or to highlight the benefits of free licenses in press releases and on the local contests' websites.[18]

Overall, Wiki Loves Monuments is one of the biggest success stories of the last couple of years when it comes to programmatic activities that fill content gaps. Its success has been supported by the general shift of the web to become more visual and the widespread availability of photographic devices. And it is one of the very few programs that shows impact at scale.

How does the cost of the program compare to its outcomes?[edit]

Given the fact that a comparably large number of chapters received grant funding, we were able to better establish a basic cost-benefit baseline for Wiki Loves Monuments than for any other program covered in this ongoing evaluation. Keep in mind that given we used data mining from grant reports, this section only addresses Wiki Loves Monuments implementations that were funded by grants, not unfunded implementations.

Our cost-benefit-analysis comprises of data from Wiki Love Monuments implementations from 20 different countries, which received WMF or FDC grant funds, representing 12 out of 35 (34%) countries which participated in 2012 and 15 out of 51 (29%) countries which participated in 2013 (7 countries are represented in both 2012 and 2013 data). The average median budget of these events (2012 and 2013 combined) was $3,615 with a high of $29,102 and a low of $392. The average median number of photos being uploaded to Wikimedia Commons per country was 5,635 with a high of 32,987 and a low of 1,572.

Looking at the cost per uploaded image, we get to the following results:

Year Number of countries Average budget (median) Average number of uploaded images (median) Average cost per uploaded image (median)
2012 12 $3,100 6,266 $0.37
2013 15 $4,129 4,102 $1.02

As photos that are being used in Wikipedia articles reach a much larger audience than those that are not included in articles, we also looked into the number of images that are currently (as of November 4, 2013) being used across Wikimedia projects. As the usage of images increases over time, we only included the numbers for 2012:

Year Total number of photos uploaded Photos in use on Wikimedia sites[19] Percentage of images in use [19] Cost per used image
2012 126,424 21,639  17% $3.03[20]

Knowing the cost for an image that got uploaded through Wiki Loves Monuments and that's also in use on Wikipedia and elsewhere is important as it allows us to compare that number to its equivalent from other photo upload initiatives.

Now, is $3.03 a high or a low price for a photo of a monument that's in use on Wikimedia projects? Ideally, we would like to know how the cost per used photo uploaded through Wiki Loves Monuments compares to the costs in a scenario where we would have relied on business photographers to shoot those monuments. In order to make that comparison, we would need to know the following:

Average time that a Wiki Loves Monuments photographer spends on

  • researching the location
  • traveling to and from the location
  • shooting at the location
  • selecting and postprocessing the photos
  • uploading the photos

Once we have those numbers, we can calculate the costs that would incur if a professional photographer would offer the same services. Our report on other photo upload events, which will be released next week, contains an example of how this can be done.

How easily can the program be replicated?[edit]

Over the years, the global event team was very effective in continuously improving and promoting the concept, which led to a growing number of countries joining the competition. The global event team also offers support for local organizers about legal advice and questions around promotion, prizes, and potential partners. Furthermore, the global event team has become continuously more sophisticated in documenting the event and providing the community with tools to measure the competition's results. All of this makes Wiki Loves Monuments the one program within our programmatic landscape that has the most cohesive and clear set of goals, measures of success and documentation for replication of success.

Next steps[edit]

Next steps in brief
  • Increased tracking of detailed budgets and donated resource inputs by program leaders.
  • Improved ability for program leaders to collect and track staff and volunteer input hours.
  • Valuation of volunteer hours input and estimation of comparative costs.
  • Use of better tracking systems for participant cohorts to examine recruitment and retention associated with specific contests.
  • Increased ability to collect, and/or link to existing, experiential survey data to contests and participant activity.
Next steps in detail

As with all of the programs reviewed in this report, it is key that efforts are made toward properly tracking and valuing programming inputs in terms of budgets, donated resources, and hours invested in activities related to planning and implementing the contests as well as comparative costs. In addition, it will be important to track particular participant cohorts so we can investigate recruitment and retention. In 2013, the Wiki Loves Monuments dashboards were designed to link to uploader lists by country, so this should be easier in the future. Lastly, further investigation of expectations and efforts directed toward the many other goal priorities is needed, including for the top two goals of: increasing awareness of Wikimedia projects and making contributing fun. To some extent these may be answered by the survey administered by the WLM International team this year; to whatever extent there are relevant survey data there or not, it will be important to more clearly articulate programming activities specifically aligned to those goals and to develop strategies for program leaders to measure those potential outcomes.

Appendix[edit]

Summative Data Table:Wiki Loves Monuments (Raw Data)[edit]

Year Percent Reporting Low High Mean Median Mode SD
Non-Zero Budgets 2012 96% $392.05 $29,101.50 $6,340.68 $3,614.50 n/a $6,516.49
Staff Hours 2012 4% 100.00 100.00 Report count of 1 0.00 0.00 0.00
Volunteer Hours 2012 7% 200.00 300.00 250.00 250.00 n/a 70.71
Total Hours 2012 7% 200.00 400.00 300.00 300.00 n/a 141.42
Donated Meeting Space 2012 62%[21] Not Applicable - Frequency of selection only
Donated Materials/ Equipment 2012 19%[22] Not Applicable - Frequency of selection only
Donated Food 2012 14%[23] Not Applicable - Frequency of selection only
Donated Prizes/Give-aways 2012 52%[24] Not Applicable - Frequency of selection only
Participants 2012 100% 67 2005 394 190 n/a 456
Dollars to Participants 2012 96% $1.20 $326.98 $40.20 $14.58 n/a $66.77
Input Hours to Participants 2012 7% Report Count of 2
Bytes Added 2012 Not Applicable (Photo counts instead)
Dollars to Text Pages 2012 Not Applicable (Photo counts instead)
Input Hours to Text Pages 2012 Not Applicable (Photo counts instead)
Photos Added 2012 100% 1572.00 32987.00 8863.78 5635.00 n/a 8845.83
Dollars to Photos 2012 96% $0.09 $9.17 $1.41 $0.48 n/a $2.00
Input Hours to Photos 2012 Report Count of One
Pages Created or Improved 2012 Not Applicable (Photo counts instead)
Dollars to Pages Created/Improved 2012 Not Applicable (Photo counts instead)
Input Hours to Pages Created/Improved 2012 Not Applicable (Photo counts instead)
UNIQUE Photos Used 2012 92% 378 6103 1929 885 n/a 1948
UNIQUE Photos Used 2013 100% 10 1687 413 262 n/a 478
Dollars to Photos USED (Non-duplicated count) 2012 92% $0.51 $26.51 $6.12 $2.43 n/a $8.22
Dollars to Photos USED (Non-duplicated count) (Wiki Loves Monuments 2013) 2013 100% $1.50 $241.40 $64.30 $38.01 n/a $77.11
Input Hours to Photos USED (non-duplicated count) 2012 Report count of 2
Input Hours to Photos USED (non-duplicated count)(WLM 2013) 2013 Report count of 0
Good Article Count 2012 Not applicable
Featured Artice Count 2012 Not applicable
Quality Image Count 2012 100% 0 484 77.67 9 0 164.04
Quality Image Count 2013 100% 0 74 11.50 1 0 22.98
Valued Image Count 2012 100% 0 27 2.50 0 0 7.74
Valued Image Count 2013 100% 0 4 0.57 0 0 1.28
Featured Picture Count 2012 100% 0 17 2.67 1 0 5.08
Featured Picture Count 2013 100% 0 1 .07 0 0 0.27
3 Month Retention (Existing Users Wiki Loves Monuments 2012)[25] 2012 Retention Rate = 43.5% (22% 5+ Photos in December 2012)[26]
3 Month Retention (New Users Wiki Loves Monuments 2012) 2012 Retention Rate = 2.5% (0.32% 5+ Photos in December 2012)[27]
6 Month Retention (Existing Users Wiki Loves Monuments 2012) 2012 Retention Rate = 40.4% (21% 5+ Photos in March 2013)
6 Month Retention (New Users Wiki Loves Monuments 2012) 2012 Retention Rate = 2.4% (0.26% 5+ Photos in March 2013)
Percent Experienced Program Leader 2012 Report count of 2
Percent Developed Brochures and Printed Materials 2012
Percent Blogs or Online Sharing 2012
Percent with Program Guide or Instructions 2012
Percent Experienced Program Leader 2013 Report count of 0
Percent Developed Brochures and Printed Materials 2013
Percent Blogs or Online Sharing 2013
Percent with Program Guide or Instructions 2013

Bubble Graph Data[edit]

Data for Graph 2 & 3. Budget, Participation, and Photos Added.

Report ID Year Budget Number of Participants Bubble Size: Number Photos Added
48 2012 $10,000 941 27,054
76 2012 $1,920 392 5,635
81 2012 $944 75 2,573
89 2012 $7,795 299 2,028
95 2012 $2,896 651 32,987
78 2012 $1,880 330 4,087
79 2012 $3,100 2,005 22,037
92 2012 $4,735 153 3,774
97 2012 $16,995 128 1,853
86 2012 $917 767 10,186
4 2012 $13,068 242 7,314
70 2013 $6,652 151 4,102
72 2013 $12,272 190 11,098
83 2013 $11,182 177 7,000
77 2013 $9,342 154 4,047
75 2013 $1,270 226 2,554
82 2013 $2,157 1,485 11,397
88 2013 $1,150 84 2,621
73 2013 $10,890 121 3,187
90 2013 $3,041 99 2,689
74 2013 $2,414 156 2,373
71 2013 $4,129 176 23,920
80 2013 $392 67 1,572
85 2013 $2,125 629 6,793
93 2013 $4,489 400 23,124
96 2013 $29,102 89 6,421

Data for Graph 4 & 5. Budget, Photos Added, and Unique Photos Used

Report ID Year Number Photos Added Budget Bubble Size: Number Unique Photos Used
47 2012 6,896 416
48 2012 27,054 $10,000 4,812
76 2012 5,635 $1,920 1,824
81 2012 2,573 $944 885
89 2012 2,028 $7,795 459
95 2012 32,987 $2,896 1,934
78 2012 4,087 $1,880 474
79 2012 22,037 $3,100 6,103
92 2012 3,774 $4,735 686
97 2012 1,853 $16,995 641
86 2012 10,186 $917 378
4 2012 7,314 $13,068 3,027
70 2013 4,102 $6,652 55
72 2013 11,098 $12,272 1,125
83 2013 7,000 $11,182 52
77 2013 4,047 $9,342 160
75 2013 2,554 $1,270 595
82 2013 11,397 $2,157 374
88 2013 2,621 $1,150 11
73 2013 3,187 $10,890 532
90 2013 2,689 $3,041 80
74 2013 2,373 $2,414 10
71 2013 23,920 $4,129 702
80 2013 1,572 $392 262
85 2013 6,793 $2,125 26
93 2013 23,124 $4,489 1,687
96 2013 6,421 $29,102 527

More Data[edit]

Program Inputsː Budgets, Hours, Donations

Report ID Program Length( Month) Budget Staff Hours Volunteer Hours Total Hours Donated Space Donated Equipment Donated Food Donated Prizes
47 1 month 200 200 Yes Yes Yes Yes
48 1 month $10,000.00 100 300 400 Yes Yes
76 1 month $1,920.00
81 1 month $944.05 Yes Yes
89 1 month $7,795.00
95 1 month $2,896.03
78 1 month $1,880.00 Yes Yes
79 1 month $3,100.00 Yes Yes
92 1 month $4,735.24 Yes Yes Yes
97 1 month $16,994.74 Yes
86 1 month $917.40 Yes Yes Yes
4 1 month $13,068.31 Yes Yes
70 1 month $6,652.45
72 1 month $12,272.00 Yes
83 1 month $11,182.00
77 1 month $9,342.00 Yes
75 1 month $1,270.00
82 1 month $2,157.31 Yes
88 1 month $1,150.00 Yes
73 1 month $10,890.00 Yes Yes
90 1 month $3,040.54
74 1 month $2,414.00
71 1 month $4,129.00 Yes Yes
80 1 month $392.05
85 1 month $2,124.76 Yes
93 1 month $4,489.17
96 1 month $29,101.50 Yes

Program Outputsː New User Accounts, Number of Photos/Media Added

Report ID Number of Participants Number NEW User Account Created Number Photos/Media Added
47 451 6896
48 941 941.00 27054
76 392 5635
81 75 2573
89 299 2028
95 651 32987
78 330 4087
79 2005 22037
92 153 3774
97 128 1853
86 767 10186
4 242 7314
70 151 4102
72 190 11098
83 177 7000
77 154 4047
75 226 2554
82 1485 11397
88 84 2621
73 121 3187
90 99 2689
74 156 2373
71 176 23920
80 67 1572
85 629 6793
93 400 23124
96 89 6421

Program Outcomesː Quality Improvement

Report ID Number UNIQUE Photos Used Number Quality Images Number Valued Images Number Featured Pictures
47 416 0 0 0
48 4812 364 27 8
76 1824 2 0 0
78 474 0 0 0
79 6103 28 2 2
81 885 484 0 17
86 378 11 0 1
4 3027 24 0 2
89 459 1 0 0
92 686 9 1 1
95 1934 9 0 1
97 641 0 0 0
70 55 0 0 0
71 702 0 0 0
72 1125 74 0 0
73 532 0 0 0
74 10 0 0 0
75 595 1 0 0
77 160 0 0 0
80 262 6 0 0
82 374 3 0 0
83 52 3 3 0
85 26 0 0 0
88 11 0 0 0
90 80 22 1 0
93 1687 52 4 1
96 527 0 0 0


Notes[edit]

  1. (Mean= 9, SD= 4)
  2. Averages reported refer to the median response.
  3. Mean= 6,340.68, SD= $6,516.49
  4. Mean= 394, SD= 456
  5. Mean=$40.20, SD=$66.77
  6. Note: Although "content production" is a direct product of the program event itself and technically a program output rather than outcome most of the program leaders who participated in the logic modeling session felt this direct product was the target outcome for their programming. To honor this community perspective, we include it as an outcome along with quality improvement and retention of "active" editors.
  7. (Mean = 8,864, SD=8,846)
  8. (Mean=1929, SD=1948)
  9. As of November 4, 2013.
  10. (Mean=413, SD=478)
  11. (Mean=$1.1, SD=$2.00)
  12. (Mean=$6.12, SD=$8.22)
  13. Note: This implementer also had the largest budget in 2012 and did not use the full amount so they applied the remaining budget to their 2013 event budget. It is unknown whether the full budget will be allocated to event expenses.
  14. Some photos may have been rated in multiple categories.
  15. https://www.mediawiki.org/wiki/Analytics/Metric_definitions#Active_editor
  16. by counting those who had made 15 or more edits for the three month quarter. However, users could make 15 or more edits during any period of time, not five or more each month necessarily. This solution allows us to even much of the month-to-month variation of users to get at those users who are more or less staying active on the project. For example someone who made five edits in the first month, three edits in the second month, and six edits in the third month following the event would be counted as active for the full three month quarter. At the same time, someone who made all their contributions in the first month following the contest, and nothing further, would also be counted as surviving or active for the quarter.
  17. New users were defined as WLM 2012 uploader usernames which had not contributed to commons in the previous year.
  18. Encouraging contributors to familiarize themselves with the Illustrating Wikipedia brochure may help address confusion about free content.
  19. a b As of November 4, 2013. Image usage numbers will increase over time.
  20. Calculated for the 21,223 images used from the 11 WLM 2012 contests for which budget was reported
  21. Percentages out of 21 who provided direct report.
  22. Percentages out of 21 who provided direct report.
  23. Percentages out of 21 who provided direct report.
  24. Percentages out of 21 who provided direct report.
  25. Wiki Loves Monuments' retention listed refers to both new and existing users; see program-specific reporting for details on retention of new vs existing contributors.
  26. Used Wiki Loves Monuments 2012 Overall, not just programs reported; Retention rate for Wiki Loves Monuments 2013 (adding 5+ photos) existing users = 34%.
  27. Retention rate for Wiki Loves Monuments 2013 (adding 5+ photos) new users = 2.3%.