Grants talk:Project/Eurecat/Community Health Metrics: Understanding Editor Drop-off/Midpoint

From Meta, a Wikimedia project coordination wiki

Midpoint Report approved[edit]

Dear CristianCantoro,

Thank you for submitting this midpoint report. I am accepting it now with the following comments:

  • Congratulations on the progress you've made. I'm really looking forward to experimenting with the dashboard.
  • Thank you for working with advisors from the Wikimedia Foundation's Research Team and Product Design Strategy Team. I know that having an advisor from the Research Team was a request made from the Project Grants Committee, so thank you for making sure that happened. Since one of the committee concerns was managing risk for the dashboard, it would be great if you could provide a summary of how you thought about and mitigated risks when you submit the Final Report. This is probably going to be especially important for editor-level metrics.
  • I want to make sure you have good support from me around the aspect of your proposal that focuses on making sure prospective users of the dashboard are able to imagine and understand what potential benefits they might get from using the tool, and how to get those benefits - including how to use the dashboard in a basic sense, and what kind of analysis might be applied to the results they can find in the dashboard. I know that your original proposal was going to include in-depth analysis of 4 language editions, which I think will be helpful for modeling the kind of understanding that can be drawn from the data, as well as a video for explaining how to use the data. I think these are good starts and I have a few questions for you:
    • How will you share the in-depth analysis so that Wikimedians who might want to participate are aware of the work? Do you have plans for a blog post or anything like that?
    • For the video tutorial, is it possible to make the slides and video scripts available on Commons in a way so that it would be possible for them to be translated into additional languages, should that be desirable by other language communities?
  • In your outcomes section, you mentioned that one of your desired goals would be for your findings to inspire new initiatives to prevent editor drop-off and improve community health. I love this vision! I wonder if you have ideas of how my team (the grants team at the Wikimedia Foundation) could help other grantees (affiliates, individual organizers, etc) understand the potential value they could get from this tool, in order to make it more likely that they will incorporate it into their own strategic planning.

Thank you again for all your work on this, and my apologies for the late review (I've been on medical leave for some months). Our grants administrator will follow up about next steps.

Warm regards,

15:02, 10 February 2022 (UTC) Marti (WMF) (talk) 15:02, 10 February 2022 (UTC)[reply]

P.S. CristianCantoro, your report points to your finances table, but the table is still empty. When you have a chance, it would be great if you could fill it in. Thanks.

Response to the Midpoint Report[edit]

(Creating a new section for clarity) Thank you for your review, Marti.

I will quote parts of your message and put our response below. 

 * Congratulations on the progress you've made. I'm really looking forward to experimenting with the dashboard.

We are currently working on the dashboards and website, and we have a few screenshots. The website is hosted on Wikimedia Cloud, here's the link, with a caveat, it is still a work in progress at the moment, and it may not work: https://vitalsigns.wmcloud.org/ 

 * Thank you for working with advisors from the Wikimedia Foundation's Research Team and Product Design Strategy Team. I know that having an advisor from the Research Team was a request made from the Project Grants Committee, so thank you for making sure that happened. Since one of the committee concerns was managing risk for the dashboard, it would be great if you could provide a summary of how you thought about and mitigated risks when you submit the Final Report. This is probably going to be especially important for editor-level metrics.

All the statistics presented in the dashboards present aggregated data of groups of user (e.g.: active editors in a month in the Wikipedia namespace), so we are not presenting statistics about single users. We will discuss this issue in the Final Report. 

 * How will you share the in-depth analysis so that Wikimedians who might want to participate are aware of the work? Do you have plans for a blog post or anything like that?
Community Events

We have already engaged in several dissemination activities with the community, in fact results from this project have been presented to:

We also hedl a joint session with the Wikimedia Research teams at Wikimania:

and we will present them to the Italian Wikipedia community:

  • 24/02/2021 | Public session for the Italian Wikipedia community, organized by WM-IT event page on itwiki


Affiliates

We have also presented these results to:


Scientific dissemination
 * For the video tutorial, is it possible to make the slides and video scripts available on Commons in a way so that it would be possible for them to be translated into additional languages, should that be desirable by other language communities?

Yes 

 * In your outcomes section, you mentioned that one of your desired goals would be for your findings to inspire new initiatives to prevent editor drop-off and improve community health. I love this vision! I wonder if you have ideas of how my team (the grants team at the Wikimedia Foundation) could help other grantees (affiliates, individual organizers, etc) understand the potential value they could get from this tool, in order to make it more likely that they will incorporate it into their own strategic planning.

The tools and dashboards will be available online and accessible to anybody, with statistics and indicators for all the Wikipedia language editions. We believe these metrics can be useful to measure the state of a community, and could also be used to measure the impact of actions taken in a community. We have discussed this topic with community members and affiliates whenever we have had the occasion to present our work. On the one hand, we think the Vital Signs metrics we developed can be used to formulate targets for projects, with all the caution about Goodhart's law of metrics becoming goals in themselves.

More pragmatically, we have given some advice when presenting our results, I report here a couple of them since we have seen that these apply to most communities:

  • Retention is critical for all communities. We have seen the retention rate go down over the last 10 years across all communities we have looked into. So, affiliates should work to establish mentoring strategies such that more new editors keep being engaged with the projects in the longer term. The work from the WMF Growth Team is also very significant on this front.
  • Any initiative to increase the number of technical contributors (i.e., people who can write templates and modules, operate bots, etc.) has a significant potential since the number of very active technical contributors is small relative to the general editor population across all communities. Even in big language editions (it, de, es, etc.), the number of very active technical editors is measured in a few dozens (< 100); for medium-sized communities (ca, ar) these numbers are lower (20-50); and for smaller communities, you can have the extreme case of just one or even no very active technical editors. Our suggestion for the affiliates is to organize initiatives and events to train the editor community on technical aspects, e.g., workshops on creating and using templates and modules, writing a wiki-bot, etc., host hackathons, etc. With these small groups, even a gain of just one contributor will make an impact.

For each metric we have presented, we have formulated targets. We have submitted them for review by the scientific community, as they are part of our paper mentioned above, currently under review. It will be Open Access, if and when it gets accepted. 

 P.S. CristianCantoro, your report points to your finances table, but the table is still empty. When you have a chance, it would be great if you could fill it in. Thanks.

I have updated the table in the Finances section.

-- Cristian, David, and Marc (on behalf of the team CristianCantoro (talk) 16:54, 4 March 2022 (UTC))[reply]

Retrieved from "https://meta.wikimedia.org/w/index.php?title=Grants_talk:Project/Eurecat/Community_Health_Metrics:_Understanding_Editor_Drop-off/Midpoint&oldid=23078246"