User talk:JMatazzoni (WMF)/drafts

Sept. 6, 2018: Comment on proposed metrics features and reports

Please use this space to tell us your thoughts about the Sept. 6 posts, Proposed metrics features—what do you think? and  New data and reports in detail.  I’ve offered the sections and questions below to help structure the discussion.

Comments about ‘Lots of new data and ways to get it’

• Are the proposed reports the right ones?
• Is the assumption that wikitext tables should provide only key figures instead of the whole dataset correct?

Your comments:

Comments about the ‘New filters for customizing reports’

• Which of these filters is the most important for you (i.e., what should we develop first)?
• What is missing?
• Are you interested in namespaces beyond Main, or is that just an extra complication?’

Your comments:

Comments about ‘Assumptions, limitations, ideas for future release’

• How important would auto-updating (on demand) of on-wiki metrics tables be to you?
• How important is mobile for you? If we optimize pages for mobile, is reporting or setup more important? Or both?

Your comments:

Comments about ‘New data and reports in detail’

• Are these the right metrics? What could we drop? What missing?
• Do the Wikitext tables’ show the right short list of figures?

Your comments:

Event Tool metrics—what do you want?

What type of data do event organizers most want—we need your input! To give you some ideas, I’ve posted a breakdown of the breakdown of the reports available now from both Grant Metrics  and the Program and Events Dashboard. Our operating assumption is that the Event Tool will use Grant Metrics for reporting (in fact, it may ultimately be just a series of enhancements to Grant Metrics).  So as you look at the lists linked to above, pay particular attention to Grant Metrics, especially to any key features you think it lacks.

What is the main purpose of the reports you want to make?

• For whom are you producing metrics  (e.g., grantors, your managers…) and what are the main things you need to understand or demonstrate (e.g., type of content created, content impact, editor retention…)?
  • Answers:

We use metrics for grant reports, our managers (board) and in some cases, for reports sent to institutional partners. We are interested in content creation and new user engagement. Ariel Cetrone (WMDC) (talk) 14:00, 23 August 2018 (UTC)

Well if WMF is going to want the event reported on (e.g. reporting on an existing grant or for applying for a new grant), then whatever the WMF wants has to be included. I find the participants (and in organisational contexts, the senior management) like to have both input metrics (# participants, # edits, # articles) but also want (and really love) the impact metrics like the # article views since the edit which Dashboard provides *and* how it increases over time (you get a snapshot each time you check but you can't get a graph over time). What is missing on the Dashboard is that it gives you metrics on image uploads but not on the number of articles that incorporate the image and the # article views, which is a problem if the program/event is about photos rather than edits. If it's about photos being uploaded and added to articles, then the edit to add to the article puts that article on the impact metric so the situation is OK. There are separate tools like Baglama etc which do report those metrics for images but I don't think I can set one of those up myself (they mostly seem to be used for large image content donation organisations). I'd really prefer if Dashboard could be "one-stop shop" to capture all impact metrics for both articles and images. Having seen Dashboard, I know a number of WikiProjects and individual contributors have said "I'd like a dashboard that tracks all the article views arising from all of our articles (meaning ones tagged by the project) or all of the articles I have ever contributed to". So do be aware that the desire for impact metrics is bigger than just programs/events (which tend to target new users) as there are likely to be some real performance issues if Dashboard starts being used by active editors or WikiProjects. Personally I dislike the way that Dashboard wants to display "recent edits", especially when I have no control over what recent means. Indeed, I am rarely that interested in recent edits, I want total edits. In my ideal universe, I can send a link to the Dashboard to anyone (WMF grant manager, head librarian, etc) and know that if they look at it today or next year, they will see the total edits. Of course, I also want the full spreadsheet of information too, so I can do whatever analysis I want , but rarely do senior staff want to be given a spreadsheet for them to analyze. They want the single Dashboard URL to do it all for them (everything they want on one screen). Kerry Raymond (talk) 22:59, 23 August 2018 (UTC)

These are all really great points Kerry Raymond. Thanks! Just a note, in case you missed it: we'll most likely be using Grant Metrics for reporting from Event Tool. (I've posted a breakdown of the reports available now from Grant Metrics and Dashboard, in case you'd like to compare or look for gaps.) JMatazzoni (WMF) (talk) 21:26, 24 August 2018 (UTC)

Editor retention metrics should be opt-in. I say this because editor retention statistics are usually very bad and you probably don't want anyone to see them. They are bad for two reasons. Reason 1. Many participants in programs/events do not see it as anything different from a Clean Up Australia weekend or a fun run for charity or attending an interesting seminar. It's something they will give a day to because they think it would be interesting to know how to contribute to Wikipedia or that we need more women biographies of artists on Wikipedia, but rarely do they come along with the intention of becoming an active editor and devoting every spare minute to it (like I do!). While many of them do have a secret desire to write an article about a usually non-notable topic, all our rules are a big turn-off to them. You really don't see a lot of people continue to contribute after an event of their own free will (obviously if done in an educational setting where there is course credit or please-the-teacher opportunities, things might be different but that is not free will). I am inclined to believe that *active* Wikipedians are born not made. Reason 2. Even when a person does continue contributing, they often do it under a different account name (because they forget their username or password and making a new account seems easier) or they decide they didn't make a good account name choice in the first place (too close to their real name, or too silly) and they don't know they can ask to change their account name. Or they just do it as an IP. I see loads of edits on my watchlist from "new" accounts/IPs that are not typical of a true newbie (e.g using templates, detailed policy knowledge in discussions, etc) -- as someone who trains genuine newbies, I know what their edits look like. So a lot of active editing does takes place that way. So, even if you have retained people from a program/event, you probably can't track it. But I know users sometimes tell me that I trained them or they met me at an event but their account history doesn't support that story, so I am well aware that retained contributors are likely to be using different user names over time and hence not able to be tracked. Kerry Raymond (talk) 22:59, 23 August 2018 (UTC)

(copied from below) ...to allow us to report on which events were most useful / which trainers were most effective in editor retention Lirazelf

Metrics reports to judge effectiveness makes sense Lirazelf. The name of the trainer is not currently part of the event data in Grant Metrics. Would you need to have them listed, to make this type of comparison, or would you just know who did what? (I'm thinking that listing the trainer's names might make some people uneasy.) And is there usually one lead trainer? Or would a report need to be able to handle multiple names? JMatazzoni (WMF) (talk) 17:33, 23 August 2018 (UTC)

• On the topic of retention, I'm sharing a brief summary of the main points of feedback I've gotten from grantees when I've asked about a retention metric over the last few years. This feedback has come from multiple projects that have include 60+ grantees. I'm adding it here since it's not aggregated anywhere. In general, a retention metric has been a constant wish from many many grantees over almost 5 years, but it's metric that has to be calculated (mostly) by hand; the significant time investment to calculate has meant its only viable for really small programs, but that doesn't diminish the usefulness to larger programs.
  • Benchmarks: As said above, retention numbers are usually low and that can be demoralizing to some organizers. One thing that has helped fight that demoralizing effect is looking at the retention for the language project overall. So say only 10% of a program's participants keep editing, if the overall retention of the language project is 5%, that 10% looks much better than before.
  • Multiple time intervals: Since retention isn't calculated by most metric tools today, there is no good baseline on what the "right" time interval should be for programmatic work. Almost every grantee I've spoken with has asked for multiple time intervals - 30 days, 60 days, 6 months, 1 year - so that they can see this change over time. Having a short and long time interval was especially necessary since the length of a program can vary widely. Here's a mock of what it could be.
  • Multiple namespaces: Right now, almost all metrics are calculated off contributions to the main namespace of a specific project. But this doesn't account for a reality where work in Draft, User, or File namespaces are required or recommended (e.g. after ACTRIAL). Also, it doesn't account for a reality where people will edit whatever they want, and the organizer can't know what they've done and where (i.e. on what project or in what language). So having a retention definition that covers multiple namespaces and multiple language projects is a practical necessity.

Tracking edits to all namespaces makes sense Shouston (WMF). I can most easily imagine including all edits in a downloaded report, where we can just include a column labeling the namespace where the edits were made (so users can do a spreadsheet sort and isolate the namespaces they want). But I'm wondering about what I call the "summary" statistics we put on the main reporting screens—for things like total "Pages created" or "Pages improved". Do people want those figures to include talk, user, draft and other namespace edits? Or should those totals remain focused on mainspace only? Anyone? JMatazzoni (WMF) (talk) 18:04, 29 August 2018 (UTC)

• • New editors: When WMF changed from Global Metrics to grant metrics, we asked about whether or not to have a retention metrics, and importantly, who was retained. Overwhelming, we heard that it should be new editor retention, not existing editor. Mostly this was because existing editors were editing before and will likely edit after the event/program; counting their "retention" as due to the program seemed untrue. This wasn't true for all programs, but was the majority of answers.
  • Editing retention vs. program retention: There are two different kinds of retention: Does someone edit Wikimedia projects again? vs. Does someone come back to program again? The first is the one we typically talk about. The second has become very useful for programs that run multiple events, or like someone noted above, only happen once a year. For example, if you only run an editathon on International Women's Day, but you've done that editathon for 6 years, you'd like to know how many "repeat" or "retained" people you have in your program (regardless of what editing they do the rest of the time). -- Shouston (WMF) (talk) 23:44, 27 August 2018 (UTC)

We use metrics for grant reports but also for our partners, they are usually interested in knowing the results of the event. About the data, type of content and edition retention. Also, I agree with Kerry Raymond, impact metrics would be nice. --Rodelar (talk) 14:54, 28 August 2018 (UTC)

Just the same, we use metrics both for internal purposes (to refer to board and members) and for external one (for partners and also for communication, if they are good enough). I would like to have retention/medium period metrics too, in addition to the ones strictly related to the event. --Marta Arosio (WMIT) (talk) 08:31, 3 September 2018 (UTC)

• What types of output formats are most important to you (e.g., spreadsheet download, automatic wikitext output, nicely designed onscreen displays in the tool…).
  • Answers:

A spreadsheet would be useful for internal purposes. It would also be nice to have an onscreen display of outcomes as well as a template of outcomes to share on the event page.Ariel Cetrone (WMDC) (talk) 14:18, 23 August 2018 (UTC)

Spreadsheet is ok but would nice if we can share the outcomes on the event page. --Rodelar (talk) 14:54, 28 August 2018 (UTC)

I agree, it is useful in order to communicate and commit to have a clear onscreen display. --Marta Arosio (WMIT) (talk) 08:31, 3 September 2018 (UTC)

Metrics about event participants

• In addition to basics like # of edits made, what data do you need about participants (e.g., editor retention metrics, gender or other demographics…)? Understand that some user data may be sensitive from a privacy viewpoint.
  • Answers:

I am particularly interested in motivating factors for new participants and participant demographics. -Ariel Cetrone (WMDC) (talk) 14:49, 23 August 2018 (UTC)

Although I find the demographic data problematic, it would be interesting to know the number of women participating and new users. Apart from that, edition retention. --Rodelar (talk) 14:54, 28 August 2018 (UTC)