Grants talk:IdeaLab/Longitudinal study of new user experiences

Comment - I like the idea. The more data there is about editing patterns and trends, the better. I would suggest that the stated goal should be to identify what percentage of editors on a given day remain active six months later, with a side benefit of listing the most common reasons why editors appear to lose interest. There will be easily identifiable buckets - for example: persistent editors (at least one edit in the past day), editors (one edit on the past week), casual editors (one edit in the past month), vandals (everything was reverted), an overlapping category of single purpose accounts (SPA), any I missed? Also, it would be interesting to see of the editors who've stuck it out, where each spends his/her time. Do they spend time doing maintenance such as copy editing/cleanup, article creation, with a single topic area of interest, etc. One other suggestion - you'll have to take into account whether that first day six months back is a weekend of weekday. It's possible that the editing motivation is different for those days, which might skew the results. Timtempleton (talk) 18:56, 30 July 2018 (UTC)

Thanks for your comments Timtempleton:

I would suggest that the stated goal should be to identify what percentage of editors on a given day remain active six months later - I think this would be the "top-line" metric. I'd also be interested in what factors result in someone becoming a more useful editor - i.e. making more edits and transitioning from a single-article focus to making contributions across the wiki. This "editor engagement" is the other, brighter, side of the coin to editor loss.

There will be easily identifiable buckets … any I missed - I think its worth considering a distinction between helpful and unhelpful SPA accounts. Some editors come to Wikipedia to work on a particular article (perhaps to create it) and are able to work within the rules and guides to make a small, but useful, addition. Others are a net negative, either because their intended goals are incompatible with Wikipedia's rules, or because—despite having viable goals—they fail to engage well (technically, procedural or socially).

…you'll have to take into account whether that first day six months back is a weekend of weekday. True. Strictly speaking there'd also be a potential bias from time of year, but, practically, that's unavoidable. There's likely to be some technical work in the preliminary stage which will scale linearly with the time span chosen, regardless of the size of the randomly selected cohort. Expanding the time range to a week would be beneficial, but might not be practical.

Cheers, --LukeSurl (talk) 15:17, 31 July 2018 (UTC)

Questions from WMF

Hello LukeSurl,

Thank you for submitting your idea to this Inspire Campaign. I really enjoyed reading it and have a few follow up thoughts and questions for you to consider:

1. What role will you have in contributing to the research related tasks of this project?
2. How will you recruit and select appropriate volunteers? (We are happy to offer support for recruitment). Further, what do you envision the volunteers will do?
3. How many new users will be in each cohort and how do you plan to interact with them? Will you be conducting interviews?
4. Lastly, does this idea require funding and do you have an estimation for these costs?

Thank you for time and additional thoughts. I look forward to learning more about your idea! All the best, LMiranda (WMF) (talk) 21:29, 23 August 2018 (UTC)

Hello LMiranda. Sorry for the delay in getting back to you. Considering that there were no endorsements or volunteers to this idea during the course of the campaign, I'm not sure how viable taking it forward would be, but if the WMF wanted to work with this that would be interesting. In answer to your questions:

1. Overall, I would undertake an initial "scoping" exercise to better ascertain how much work the project would involve, and the necessary work to determine the "cohort" and divide into sub-cohorts. Then I would design a reporting system (i.e. a standardised form) for the researchers to use when collecting data, as well as written guidance for them. Before the recruited researchers begin their projects, I would run a small cohort myself to test the process. During the "data gathering" stage I would be "on call" to answer questions from the researchers. Once the researchers had done their work and submitted their reports, I would combine the overall statistics and overall report.
2. Recruitment on the English Wikipedia through notices placed at Wikiprojects, Signpost etc.. Volunteers would be assigned a list of usernames, and be given written guidance as to how to derive qualitative/quantitative data for these.
3. Total cohort would be, ideally, every person who created a username on one day. Very rough estimate, this is about 5000 persons. However it seems that the majority of accounts never edit, so the actual number of accounts that would need inspection would be a fraction of this. The project can be done "restrospectively" from public logs, so there is no need to interact for the core of the project. However, to add extra context, every editor in this group who has edited in the last month (i.e. the "survivors") would be sent a talk page message asking for their comments on their experiences as a new editor. This might be a simple, single, open-ended question, or a series of questions could be devised.
4. No funding need is envisaged. However it would be good to get a legal "OK" from WMF regarding this, checking that the processing of these data does not violate any privacy laws.

Personally, my personal circumstances (gap between two jobs) mean I would be able to do a lot of work on this late October-early December but not much afterwards or before. Cheers, Luke. --LukeSurl (talk) 09:37, 27 August 2018 (UTC)

@LukeSurl: Hi Luke! Just wanted to check in and see what sort of planning or thinking you might have done around this project so far, or if you've encountered any needs or issues. Did you have a timeline in mind for conducting this work? Regarding the ethical/legal questions you raised around this research, observational work -- that is aspects that involve reviewing editor contributions or logged actions -- don't require any special permission or approval. Contacting surviving editors for follow-up may bring up some privacy issues or the need for informed consent depending the kinds of questions they are asked and how you would be gathering this feedback, but I think more information about how this would be done is needed to provide a clear answer. I JethroBT (WMF) (talk) 20:46, 13 November 2018 (UTC)

@I JethroBT (WMF): - Honestly, I haven't thought about this for the best part of a month, and, realistically, I do not have the time to work on this in the near future. You - or anyone else - are welcome to adopt this idea

if you wish. --LukeSurl (talk) 10:03, 14 November 2018 (UTC)