Grants:Project/WM HU/Editor retention program/Midpoint

Add links
From Meta, a Wikimedia project coordination wiki


Report accepted
This midpoint report for a Project Grant approved in FY 2018-19 has been reviewed and accepted by the Wikimedia Foundation.
  • To read the approved grant submission describing the plan for this project, please visit Grants:Project/WM HU/Editor retention program.
  • You may still review or add to the discussion about this report on its talk page.
  • You are welcome to email projectgrants(_AT_)wikimedia.org at any time if you have questions or concerns about this report.



Welcome to this project's midpoint report! This report shares progress and learning from the grantee's first 6 months.

Summary[edit]

In a few short sentences or bullet points, give the main highlights of what happened with your project so far.

  • Until now, the main focus of the project was creating a statistical analysis about the community trends, and running surveys in order to understand better what is going on and what can be the background of it. The project also provided a set of editor statistics to the community members, to serve as incentive and recognition for their own activity, and a monitoring tool of others' (especially newcomers') activity.
  • Beside that, the project focused on improving community health through discussions about the importance of friendly communication and safe physical places, which resulted in familiarizing ourselves with and translating some global policies (friendly space policy and event policy), a guideline about civilized discussions, and a global ban. Additionally, large number of user and user talk pages of anonymous contributors were deleted in order to prevent bad experience from receiving outdated messages posted to someone else earlier. Moreover, regular in-person events (meetups) were organized in order to forge personal relationships, build empathy between contributors, and to learn from each others, and a three-day event (WikiCamp) was organized in order to allow contributors to socialize and spend more time together, get to know each other better and discuss specific topics deeper.
  • The third focus was preparing future activities with discussions, community consultations, software development and translations, like joining the Growth prototype, introducing the Wikimedia Space and the community service award, or improving the accuracy of ORES / WikiLabels on the Hungarian Wikipedia.

Methods and activities[edit]

How have you setup your project, and what work has been completed so far?

Describe how you've setup your experiment or pilot, sharing your key focuses so far and including links to any background research or past learning that has guided your decisions. List and describe the activities you've undertaken as part of your project to this point.

Project start[edit]

The first month of the project was spent on setting it up and starting it:

  • consultation with a lawyer, accountant;
  • finalizing and signing the agreement with the WMF;
  • composing and publishing a job posting;
  • evaluating the applicants, making a decision;
  • preparing and signing the contract with the project manager;
  • creating the project page on Meta;
  • making announcements about the project;
  • preparing a task list, defining first tasks and priorities.

Research, surveys and statistics[edit]

The first part of the project focused on gaining a better understanding of what the actual editor retention trends are and what do the community members find good or bad, what would they change.

  • The Flagged Revisions extension was enabled on the Hungarian Wikipedia in 2008. The effect of that change had been a perennial discussion topic. After a lot of debates, many theories and hypotheses, in 2018, the community decided to change the FR setting for an experimental time period in such a way that all visitor can see the latest versions of the articles (which in a sense amounts to partially disabling Flagged Revisions). Half a year later, we started discussing whether the experiment was successful or not, what changed and how should we measure it. Part of the community proposed to change back to the pre-experiment setup, others wanted to keep the experimental settings. The consensus was that we wait a bit longer until we see some objective data which can help in the decision. A few editors tried to make some manual measurements (for example they took samples from both periods and evaluated the ratio of vandal edits), but it was not enough for a well-founded decision, and over time later more and more editors became angry about the situation. Therefore we decided to help with a statistical evaluation. The evaluation started in June based on the user activity data available on stats.wikimedia.org/v2, but it had to be restarted in July because of a methodological change in the data set. The evaluation is complimented with an analysis of the vandalism ratio based on ORES data (for which the model had to be retrained first, and a dataset created). The results are being presented to the Hungarian community, and a report in English will be available with the main conclusions and learnings before mid October.
  • Activity statistics and toplists of users are very popular among editors; a set of such pages (like monthly, yearly or all-time activity counters, or editors ordered by their first edit) was generated using software developed by one of our community members, and these pages are updated regularly. Some of these pages are useful for monitoring as well, for example list of new editors or list of inactive editors.
  • We started more consultations where we asked the community members about their experience, for example about the usage of talk pages or about friendly communication. More details about these activities can be found below.
  • We started to collect feedback from our events. The responses help us improve our future events and understand what the participants felt useful or interesting and what not, why they came and what they would like to see or experience next time. Beside collecting feedback after the meetups, we used the opportunity to make some personal interviews during the meetups with a few volunteering participants. We received responses from active editors, currently inactive contributors, and newcomers (who have never edited before) as well.
  • To understand community needs and problems better, we prepared community surveys, separately for active, inactive and new contributors. A bot was developed to send out large number of messages (through the on-wiki e-mail form if it is available, otherwise via the talk page). After the list of users were generated, the messages with the survey were sent out to a randomly selected subset of the users. Now, we are receiving large numbers of responses.

Guidelines, community health[edit]

  • One of the most important factors in retaining active contributors is a friendly community. Therefore, we would like to put a great emphasis on improving community health, and reaching a more welcoming, helpful and friendly community. As a first step, a community consultation took place, and it resulted in a set of behavioral recommendations and best practices in online communication.
  • In-person events have an important role in an online community: they help to bring the members closer, into a more personal relationship with real faces attached. They are also usually attended by a few people with no former wiki involvement. Therefore, it is very important to provide a calm, friendly, welcoming and harassment-free space for everybody. In order to help this goal, the Friendly space policies and the Event Ban policy were translated and presented to the Hungarian community, and then enforced during the in-person events. As a result of feedback from event participants, an event ban was applied on one of the community members.

Welcoming new editors[edit]

  • One common piece of community feedback was that it's a bad experience for anonymous editors that they often receive unintended messages (like Do not delete!, Stop doing that!, You are blocked! etc.) after their first edit, because they see the messages of an earlier user of the same IP. Even when the messages are not offensive, they see conversations on 'their' talk page from the previous user, which is very confusing. Therefore, all anonymous pages with dynamic IP address were completely emptied using a bot. This is only a temporary solution (fix) until the short term solution or the long term solution will be developed.
  • A common wish was to have a list of newcomers, so we can see which editors need more attention. The list of newcomers has been generated and is regularly updated.
  • We realized that many of the tools and ideas we planned in order to welcome, orient and onboard newcomers better on the Hungarian Wikipedia, are covered by work done by the WMF Growth Team, so we requested to join the Growth prototype. After the team approved, we started a community discussion to reach consensus about this move, started translating the project pages, software interface and documentation, and looking for volunteers for the mentor program.

Motivating editors[edit]

  • Activity statistics, which provide metrics and overview to the contributors about their activity in comparison with other contributors, can help the motivation of active contributors and wikiaddicts. In order to make use of this, regularly updated activity statistics (such as a list of editors based on their monthly activity, a list of editors based on their yearly activity, a list of editors based on their all-time edit count, and a list of editors based on their first edit) were created and maintained.
  • Participating on regional or international conferences or trainings can energize and inspire participants as they get to meet and talk to many similar enthusiasts, and allow them to learn from each other and exchange ideas and best practices. Therefore community travel scholarships were offered on behalf of Wikimedia Hungary for Wikimania 2019 (two scholarships) and for the Wikimedia CEE Meeting 2019 (two scholarships).
  • Community service awards have been discussed for a long time (more than ten years) on the Hungarian Wikipedia, but despite the popularity of and support for community discussions, they have not yet been realized. As part of the editor retention project, we would like to provide software support to make these lists easier to maintain. Therefore a contractor was hired to develop a tool for this task. The development is ongoing, and the service awards will be presented to the community around end of September.

Events[edit]

  • Organizing regular programs and meetups offers the possibility for meeting in person, discussing questions about online activities, building relations, and for learning from each other and from invited speakers. These events offer a personal learning space for people who are interested in Wikipedia but not familiar with it, or just started to get involved in the community, and also provide meeting space for experienced contributors to strengthen their sense of community. In order to reach these goals, meetups were organized in May, in June and in July. These events were more organized than the usual spontaneous wikimeetup, with socializing programs, trainings and presentations.
  • Wikicamp 2019 had been organized in Kecskemét. The camp offered a three-day occasion for contributors to get to know each other. The event provided cultural, natural and educational programs, and time for the community members to spend time together. Beside community building, it was an ideal occasion for deep and thorough discussion of important community issues or project ideas, for developing new proposals, and also for effective educational programs.

Communication[edit]

  • The online social life of the community moved more and more outside of Wikipedia in the recent years, mainly to Facebook groups and closed mailing lists. The editor retention project tries to serve these needs, therefore we updated our Facebook surfaces and catalyzed some activities there. We started a community discussion in order to see what would be the reception if links to the the Facebook pages and groups would be displayed more prominently on the Hungarian Wikipedia. In parallel, we started to prepare for introducing Wikimedia Space to the (Hungarian) Wikimedia community.
  • In connection to preparing and executing our project in the last couple of months, we met many enthusiasts, who work in the same field improving the capability for retaining volunteer contributors. In order to join our efforts, work together and exchange ideas, we started to build up an international group focused on editor retention. As a first step, we organized a meetup at Wikimania 2019. At the meetup we decided that we will set up and use a communication channel for future collaboration where we can easily keep in touch and share our ideas and results.
  • The blog of the Hungarian Wikipedia has not been very active for a while, but we posted three short stories about WikiCamp 2019, Wikimania 2019 and the malicious attack on Wikipedia in September.

Technical environment, software development[edit]

  • One of the most important technical improvements we planned to offer for the community was introducing a more modern, user friendly interface for editing talk pages and community discussion pages. This is mainly important for newcomers, who often have problems with asking or answering questions, or commenting about mistakes. The existing user experience is confusing because of the different appearance and behavior of Visual Editor (which can only be used in articles, not talk pages) and the wikitext editor. And even if they can solve the technical challenges of editing using wiki code, they often face unfriendly reactions because of missing signatures, wrong formatting etc. That said, a modern interface would provide more convenient user experience for experienced contributors, too. Therefore (already before the start of this project), a community consultation was performed about enabling the Flow extension (since then renamed to Structured Discussions) on the Hungarian Wikipedia. Some motivated community members invested a lot of energy into translating all interface messages of the extension to Hungarian, and other blocking technical problems were also solved. Meanwhile, the global talk page consultation started, and we provided a summary of the feedback given by the Hungarian community in the local consultation. During the global talk page consultation, we were told we have to wait for enabling the extension until the end of the consultation. And after the consultation ended, the responsible team at WMF decided to change the conceptual direction of talk page improvements, and declined the request to enable Structured Discussions due to its discontinued status. That was a big disappointment for many involved participants and community members, but we expressed our intent to help and participate in the new Talk pages project.
  • The technical tasks related to editor retention on Hungarian Wikipedia and to the grant were collected and organized on Phabricator in order to have an overview and to make managing them easier.
  • Our plans involve creating a wide range of activity statistics for the different user groups inside the Hungarian Wikipedia. For this aim, we made plans for a statistical portal and identified needed features. We proposed it as a Google Summer of Code (GSoC) 2019 project, evaluated four applicants for it, but in the end had to withdraw the project due to changes in our of mentoring capacity. At the Wikimania Hackathon two volunteer developers expressed their interest, and we will continue the work with them in the following months.
  • ORES is a machine learning service for Wikimedia projects, and it helps (beside other features) with identifying bad faith and damaging edits, which helps the work of the recent changes patrollers. The Hungarian Wikipedia is using this service since its experimental, first phase, but the used model had some bias. Therefore a new set of labels was created, and with that the model was retrained and updated in order to make better predictions.
  • As already mentioned above, we expressed our interest to join the Growth prototype and its tools, which will help newcomers in their first, critical period through an improved software interface. The features will be enabled as soon as the Hungarian Wikipedia is prepared for it, expectedly within a month.
  • We plan to present Wikimedia Space for the Hungarian community as a new communication platform. To prepare for that, we started trying out the software and its services, and joined the testing and developing efforts in order to fix the most critical problems before it can move from the prototype to the production phase.
  • We asked a community member for software development work which will support the planned community service awards.

Midpoint outcomes[edit]

What are the results of your project or any experiments you’ve worked on so far?

Please discuss anything you have created or changed (organized, built, grown, etc) as a result of your project to date.

Outcome details[edit]

Statistical analysis
Wikicamp group photo
Wikicamp discussion
Wikicamp praxinoscope
Wikicamp museum
Meetup with discussion
Meetup with training
Cake on the anniversary meetup
Wikimania presentation
Activity statistics about the contributors

We created regularly updated activity statistics for the Hungarian editors:

Eventually we intend to incorporate this functionality in the planned statistical portal, but these pages are a useful stopgap solution until then.

Community dynamics and trends
Event surveys

We started making surveys and collect feedback after the organized events (meetups, trainings, Wikicamp) to improve the experience of the participants.

The answers received from the first survey helped with coming to consensus about an event ban. The other two surveys are not yet evaluated and published (coming soon).

Community surveys

The talk page consultation was integrated into the project.

We prepared and started community surveys for three important target groups:

We started these surveys a few weeks ago, and the results will be available later.

We made some personal interviews about the user experience of Wikipedia during meetups. They are only on paper yet, but will be integrated into the report about the surveys above.

Further collaborations

We are aiming at building an international network of people and organizations which are active in the field of volunteer retention and research about volunteer communities, including

This is more of an idea than an existing collaboration at this point, but we had

Community health, guidelines
Welcoming new editors
  • Based on a community discussion, deleting the content of the discussion pages of anonymous contributors (over 32 thousand pages): temporary solution until the short term solution or the long term solution will be developed.
  • Creating a List of newcomers page to monitor the new contributors, their activity, and if they were welcomed.
  • Initiating the discussion and doing preparations to join the WMF's Growth prototype by
    • translating the main project pages, newsletters;
    • translating the software interface;
    • translating the software documentation;
    • finding volunteers to be mentors in the program.
Motivating editors
Events
  • Community meetups:
    • On 4th May with presentations (20 participants),
    • On 15th June with a training and presentations (18 participants),
    • Anniversary meetup to celebrate the 16th birthday of the Hungarian Wikipedia on 8th July (with 27 participants and a cake; more than third of the participants were newcomers or non-editors).
  • WikiCamp 2019 with educational and cultural programs (with 14 participants, two of them were newcomers, one of them never edited Wikipedia before, but both of them have been active contributors since then).
Communication
Technical environment, development

Outcome summary[edit]

The project proposal promised some minimum outcomes for each field of activity. From these plans, the

  • Surveys, statistics goals are already fulfilled;
  • Guidelines goals are not yet fulfilled;
  • Welcoming new editors goals are partly fulfilled, and they are on track to be fulfilled;
  • Motivate the contributors goals are in a preparatory phase;
  • Events for in-person meetings goals are partly fulfilled;
  • Communication, online community building outside of Wikipedia goals are in a preparatory phase, with some measurable outcome;
  • Technological environment, tools, gadgets goals have partly failed because of external reasons and partly in a preparatory phase.

Looking at the metrics mentioned in the proposal:

  • Survey about the experience of the new editors (with at least 10 editors): the survey is still running, but we have already received 60+ responses.
  • Survey about the experience of the old editors (with at least 30 editors): the survey is still running, but we have already received 60+ responses.
  • Survey about the experience of the inactive editors (with at least 10 editors): the survey is still running, but we have already received 30+ responses.
  • Organizing regular meetups with presentations and trainings (at least 5 meetups with at least 80 participants in sum): 3 meetups with 65 participants in total have been organized until now.
  • Organizing a Wikicamp with educational programs (with at least 20 participants): the Wikicamp was organized with 14 participants.[1]
  • Publishing blog posts to spread news and events about Wikipedia and Wikimedia (at least 10 blog posts): 3 blog posts were published until now.

Looking at the goals around participation:

  • At least 50 participants will be engaged in various surveys (personal and online interviews, online surveys): over 150 participants have been engaged until now.
  • At least 100 participants will join the meetups and events organized in the frame of the project: 79 participants have joined until now.

Finances[edit]

Please take some time to update the table in your project finances page. Check that you’ve listed all approved and actual expenditures as instructed. If there are differences between the planned and actual use of funds, please use the column provided there to explain them.

Expense Approved amount Actual funds spent Difference
Community and project manager 2,804,298 HUF 0 HUF
(2,804,298 HUF) 		2,804,298 HUF
(0 HUF)
Occasional contractors (developer, translator, tutorials) 618,182 HUF 0 HUF
(40,000 HUF) 		618,182 HUF
(578,182 HUF)
State taxes 2,309,752 HUF 0 HUF
(1,907,281 HUF) 		2,309,752 HUF
(402,471 HUF)
Event space 730,578 HUF 120,500 HUF 610,078 HUF
Wikicamp 337,190 HUF 211,409 HUF 125,781 HUF
Total 6,800,000 HUF 331,909 HUF
(5,083,488 HUF) 		6,468,091 HUF
(1,716,512 HUF)

Then, answer the following question here: Have you spent your funds according to plan so far? Please briefly describe any major changes to budget or expenditures that you anticipate for the second half of your project.

  • Basically, we spent the funds according to the plan.
  • Explanatory notes for the table:
    • Numbers in the Actual funds spent column in brackets mean that these amounts have been committed via signed contracts, but not yet spent (paid).
    • Numbers in the Difference column in brackets mean that these amounts are still freely available (and not committed via a signed contract).
    • Numbers without brackets show the actual funds spent (paid out).
  • Additional note: we spent less money (63%) on Wikicamp compared to the plan, because the participation was around half of what we expected.[1] We might ask to reallocate the remaining funds in this line.

Learning[edit]

The best thing about trying something new is that you learn from it. We want to follow in your footsteps and learn along with you, and we want to know that you are taking enough risks to learn something really interesting! Please use the below sections to describe what is working and what you plan to change for the second half of your project.

What are the challenges[edit]

What challenges or obstacles have you encountered? What will you do differently going forward? Please list these as short bullet points.

  • Work on the project started more than a month later than scheduled because of administrative holdups, see
  • Limited possibilities to improve the communication interfaces inside the Hungarian Wikipedia since the Flow extension became discontinued in the meanwhile (it would have been especially important for newcomers).
  • Teaching and training community members about the importance of friendly and civilized behavior, and achieving real change and results is a hard and very slow process.

What is working well[edit]

What have you found works best so far? To help spread successful strategies so that they can be of use to others in the movement, rather than writing lots of text here, we'd like you to share your finding in the form of a link to a learning pattern.

  • We received unexpectedly large numbers of responses from the community surveys.
  • The statistical analysis about the effect of Flagged Revisions on the Hungarian Wikipedia provided some interesting results about the trends and processes, and
  • This project fits very well with some of the ongoing projects at the WMF (for example Growth features, Wikimedia Space), and we are very hopeful about the outcome of their application on the Hungarian Wikipedia.

Next steps and opportunities[edit]

What are the next steps and opportunities you’ll be focusing on for the second half of your project? Please list these as short bullet points.

On the short term[edit]

In the longer term[edit]

  • Statistical portal
  • New tools for patrollers on the Hungarian Wikipedia
  • Meetups and trainings
  • Updated and new tutorials and guidelines
  • Joint research with partners
  • “Many more”

Grantee reflection[edit]

We’d love to hear any thoughts you have on how the experience of being an grantee has been so far. What is one thing that surprised you, or that you particularly enjoyed from the past 6 months?

  • We were a little bit surprised by the results of the statistical analysis of the effect of the Flagged Revision configuration change (which showed a large increase in anonymous edits, but no change in new users).
  • We loved to receive so many helpful and thoughtful responses from the community in the surveys. Now, we need some time to summarize and evaluate them :D



  1. ↑ a b For this metric we reached only 70 % of the target value. We tried to identify the reasons for the low level of interest, but haven't succeeded. Both the city and date of the Wikicamp were chosen by a community survey, and over 30 people signalled their strong interest. There could be an effect from the negative experiences during earlier in-person meetups (see at the event ban). We asked the participants about their experience after the event, and they were satisfied or very satisfied. We tried to ask other, non-participating community members about the reasons on wiki and on mailing lists as well, but only a few responses were received with (very clear and acceptable) personal excuses. We hope that next year we will have much higher participation.


Retrieved from "https://meta.wikimedia.org/w/index.php?title=Grants:Project/WM_HU/Editor_retention_program/Midpoint&oldid=19406813"