Wikimedia monthly activities meetings/Quarterly reviews/Growth/June 2014

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Growth team, June 11, 2014, 1PM–4PM PDT

Present: Kaity Hammerstein, Moiz Syed, Steven Walling, Howie Fung, Lila Tretikov, Erik Möller, Tilman Bayer (taking minutes), Danny Horn, Terry Chay, Dario Taraborelli, Toby Negrin, Danny Horn

Participating remotely: Aaron Halfaker, Matthew Flaschen, Rob Moen

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Agenda

1. Review the team's mandate and framework
2. Discuss 2013-14 and 2014-15 targets
3. What we've shipped up to the present
4. Roadmap for next quarter and through 2014-15

Slides

Introduction / team members

Steven: Introduction (entire team present, except Sam Smith because of timezones)
This time, a bit more about the entire FY (instead of just the past quarter)

The team today:

• Engineering: Matt, Rob, Sam, AndyRussG (contractor – Campaigns)
• UX: Kaity, Moiz
• Research & Data: Aaron
• Product: Steven

Recent team history:

Previously Pau and Moiz on UX
Around February, Matthew was only engineer
Lila: Kaity and Moiz, do you guys code too?
Kaity: no
Moiz: a bit
Steven: Moiz will often jump in on e.g. CSS, but not do code review etc.
Aaron and I often do code review for functional requirements
Lila: UX can be performance issue in frontend, who reviews that?
Steven: all three of us
Matthew: the three engineers review each other's code
Steven: we hire for JavaScript expertise with cross-team code review for more specific issues, e.g. for i18n issues, bring in language engineering
for performance: ...

The Growth team's mandate and framework

"Attract the right kind of user in order to sustainably grow our community of editors"
"sustainably" part is important
e.g. WLM: huge spikes, but few stick around
Erik: Andrew Russell Green's involvement?
Steven: yes, contractor for Terry, but for campaigns, not main work

VE, Flow, Multimedia – these features are kind of agnostic of what kind of users are being attracted. Growth's job is to get more users in the front door, get them to use these new products, and get them to stick around.

Draft targets for 2013-14

2013-14 AP goal was: 2400 per month increase, adjusted
was extrapolated from A/B testing:

• Assumed registration/mo: 120k
• Additional % of registrations make it to 1+: 10%
• (constant) 1+ to 5+ survival: 20%
• additional # of 1+ editors: 12k
• additional # of 5+ editors: 2400

Lila: assume that the survival rate stays the same? Steven: Yes.

The team's operating framework

The User lifecycle: Readers -> Unregistered editors -> 0 -> 5 -> 10 -> 100+

• Acquisition
• Activation
• Retention
• Reactivation

1. Areas of focus (one of the above)
2. UX problems in each lifecycle stage
3. Experimental solutions
4. Products

Tested everything that was rolled out this year, with one exception

Product areas planned this year:

1. Acquisition: Get more anonymous editors to create accounts
2. Activation: Deliver onboarding across wikis and devices
3. Activation: Improve article creation process from the perspective of new users
4. Retention & Reactivation: Provide task suggestion & notifications

Lila: how was our work time distributed among these 3 areas?
Steven: around February, most on #2 (onboarding productization and rollout)
only in the last quarter got to the anon acquisition and task suggestions, due to lack of engineering resources (Matt was sole engineer until Jan/Feb.)

Lila: retention the most expensive part to fix, but need to fix right side of the funnel to prevent dropoff
Steven: yes, we actually started with account creation process and onboarding for similar reason, so that new acquisitions don't have a horrible first time experience with no direction

Discuss 2013-14 and 2014-15 targets

What we shipped this year so far

• DONE A/B testing of onboarding, build recommendations API
• DONE Launched onboarding on 30 wikis including top 10
  

Lila: why not more beyond top ten?
Steven: need to wait for translations and diminishing returns (80% of new users on these 30 wikis)
On long tail of much smaller projects, it's kind of accepted to roll out in English and wait for local users to translate, but not on (say) top 50
Erik: also, need local maintenance categories for task suggestions, not all wikis have them
Steven: yes but can roll out anyway
Erik: So translation is the only blocker (Steven:)

• DONE Refactor GuidedTours
• DONE Completed first A/B test for anon conversion
• MVP (minimum viable product) done Launched draft namespace and tested prototypes
  

have functioning draft namespace on enwiki, used to be 100 new articles or so per day, 20-30 published, now jumped to about 1000
Lila: workflow for this?
Steven: article creation wizard sends people to draft creation
from redlinks though, users are still sent to direct article creation
Erik: background: before that, had AfC (article for creation) process set up by enwiki community
very complex and bitey
low success rate
community asked us last year to implement this in software
implementing whole AFC mechanism would probably take the whole team 1-2 months of work
Steven: yes, and much of it switching to drafts namespace now anyway
Howie: redlink creations are several orders of magnitude higher
Steven: Aaron did a comprehensive research report on article creation on several wikis
this will be a significant help once we do further work in that area
Matt: also, at the moment can't create article as anon user (on enwiki)
Howie: also, deletion rate is much higher for articles created by new users who signed up for that
Lila: probably also because there is more advocacy etc.?
Howie: yes, garage bands etc.
Aaron: 30% of new user creations go to AfC
Lila: can we look at process?
(Dario on chat: Research:Article_Creation_Workflow/Clicktracking)
Erik: (on background) in the old days, all redlinks lead directly to edit page
in 2005, after Seigenthaler affair, Jimmy asked to restrict article creation on enwiki to logged-in users, as an experiment (not on other Wikipedias)
two years ago, community debated further restrictions (to autoconfirmed users), we vetoed that
(Steven:) demos Article wizard
Lila: will I always end up in that workflow?
Steven: No, not when going via redlink
research by Aaron
usual workflow was: published immediately, if then nominated for deletion, this normally happens very quickly
AfC kind of reverts that (publication in the end, rejection often comes late)
Erik: keep in mind that other languages are different
community still has high focus on quality
AfC does not have a very collaborative feel, rather "submit" to verdict of other users
Steven:
around 20k article creators per month, but the number of people taking on the three GS tasks is much higher, so focus there first

• Aimed for next 2 sprints: test task suggestions

Our overall goal

Editor population decline: ~4.96% year over year
Our goal: Halt that
Our question: What would we need to do that?

Lila: so analyze population that is leaving? Steven: Yes
Aaron: retention rate of new editors has dropped (since 2006). Total active editors breakdown (is a combination of new active editors, surviving new editors and old/returning active editors):

TAE = NAE (new) + SNAE (surviving new) + OAE (old) + RAE (returning)

• NAE targeted by acquisition x activation rate
• SNAE targeted by retention rate
  

(see also Research talk:Modeling monthly active editors)

Lila: these are disjoint sets? Aaron: Yes

Breakdown of draft 2014-15 targets: how much change would we need to achieve in each of these levers to halt the decline? (by itself)

• Acquisition: +32,361 (+15.35%)
• Activations: +.8% (from 5.3% to 6.1%, i.e. +15.35% proportionally)
• Retention: +10% (from 11.78% to 21.98%, i.e. +86.59% proportionally)

e.g. +33k new users
or: increase activation rate
that's how we arrived at our new Annual Plan goals

Toby: we know that retention rate for old editors is quite constant
Howie: baseline YOY around 80%
Erik: i.e. not enough to achieve enough 5+ editors
Aaron: if we are happy to only halt decline by means of more new users, that also means higher turnover
Lila: also, what might unintended consequences be
Aaron: also, when talking about retention, these are already pretty "good" users
Lila: measuring success only in terms of active editor numbers
need to qualify type of work they are doing – e.g. only copyedits?
Howie: e.g. used revert rate as a signal
Steven: also, did some analysis on type of editor
Lila: or, contiguous bytes of input (how much text added)
Dario: yes, started with these metrics (edit as atom) as they are easy to measure
next stage from Analytics would be to algorithmically extract types of editors
handcoding only good for small samples
Steven: also, interest graphs (for topical suggestions)
Lila: yes, don't want to stop the work you are doing, just move further into more detailed analysis
in the end, we care about quality content
so we need to understand the editors and the edits they are making
how do we get newbies to a level where they are making lots of quality contributions
Erik: yes, will need segmentation by content, geography
that said, need to focus on testable hypotheses
e.g. bringing in lots of copyeditors is good, can find negative side effects (if any) easily
and these are good onboarding tasks
a lot of really experienced editors started like that
Wikihow has systematic onboarding via spelling mistakes
not because it's a priority to improve spelling, but because it's a good way to draw people in
Lila: we really need to understand this data, it just should not paralyze further work
Steven: also, qualitative research by Abby
Aaron: we always measure block rates, revert rates, etc.
Steven: focus on all three areas for goals (acq, activation, retention)
It was kind of not very motivating for team to monitor TAE over this year (a lot of delay until new monthly number comes out)

2014-15 targets

June 2014:

• Acquisition: increase registrations 15% vs. control
• Activation: increase rate registrations become active by 15% vs. control
• Retention: increase survival 86% (one month survival)

Toby: other products contribute to retention rate as well...
Steven: yes, that was another problem with TAE (as goal) last year – no control over how much due to Growth team's work
Toby: may be possible to find some causality in metrics
Lila: should avoid "siloing out" – other areas should take this view too
Steven: there might be combined effect (e.g. copyediting made easier by VE ;)
Dario: in the past, very careful not to cross streams, isolate effects
in the future, can control better

Moving these numbers

Howie:
acquisition goal is reasonable
retention is very ambitious, almost doubling

How much have we improved conversion to date?

• Account creation tests: +4% acquisitions
• Anonymous signup test invite: +200% acquisition
• Onboarding tests: +2% activation rate to 1+ edtiors

Lila: what is the gut feel on these three – easy, medium, hard?
Steven: acquisition probably easiest
Howie: also, get Mobile on that
Steven: about retention, no idea
Howie: as an upper bound, can compare retention rate in 2006
Aaron: it was about twice as high
data suggests retention is more of a social problem than a feature problem
Lila: personally, I find editing just scary – if I wouldn't have Erik next to me I wouldn't do anything ;)
Erik: Teahouse type efforts might be a good use case for Flow
Dario: can move acq numbers – Mobile showed that
Lila: but, they drop off
Dario: not necessarily more than on desktop
what's hard is retention
Steven: can focus more
Dario: did early "necromancy" test on reactivation, was problematic for many reasons
but now, have Notifications system in place
Lila: sounds great
We need to make sure numbers are achievable

break

Erik: back to 2014-15 targets
so if we hit all three targets, that would more than halt the decline? yes
so these are very ambitious
also, note that 5% is the average monthly y-o-y drop, not the month-to-month drop
Lila: it's also fine to have low and high watermark goal (something that we know we are likely to achieve, vs. stretch goals)

Key users

• Anonymous editors: "I've edited before. Why should I create an account?"
  

(really focusing on those who already have an interest in editing)

• New editors: "I've made my first edits. What's my next step?"
• New article creators: "I want to start a new article. Where do I begin?"

Toby: about the personas (photos), do we have actual demographics?
Steven: it's halfway between user story and persona
we could do more demographic persona stuff
Kaity: builds more empathy
Lila: want to understand demographics
Steven: problem for new users are the same for different demographics (13yo boy in India with Windows machine vs 45yo woman retiree with Mac in the US)
Lila: but are they really?
Steven: editor motivation study found common motivation
Howie: of course this has survivor bias
Erik: we could go into demographic complexities for years but...
Lila: fair enough

Steven: on draft roadmap (see mw:Wikimedia_Engineering/2014-15 Goals#Editor Engagement - Growth)

Roadmap for next quarter and through 2014-15

• Q1. Anon editor acquisition products tested and released on top Wikipedias
• Q2: Personalized task suggestions and notifications of new editors released
• Q3: Improvement to article creation entry points A/B tested and released. A/B testing completed of Draft publish/unpublished workflow (in en.wiki)
• Q4: Enhanced new article/draft creation tools (e.g. notifications, Flow integration, and automated quality heuristics) released.

At one point need to stop and look of numbers for task suggestion – then refine that or move to harder tasks like article creation work

Erik: mentoring new users not on the agenda of team?
Steven: no, social stuff like Moodbar 2.0 not on roadmap
Erik: ok, but should talk about mentorship, revisit findings from Moodbar experiment
Lila: should Flow team own the social stuff?
Erik: ideally, but they are right now focused on getting their product mature, before use cases. let's see if they can work on mentorship workflow in first quarters, or make it a more cross-team effort
Steven: right now, our products are pretty much separated from other efforts, might connect more
on mobile, different tasks suitable
Lila: want us to get into the habit of testing on Mobile too
Steven: say for article creation, won't have the same success rate
Lila: sure, but monitor it
Erik: e.g. right now have to be logged in to edit on mobile, so not the same situation. When we activate anon editing, yes
Steven: in the past, in two cases parallel efforts with Mobile: 1. anon signup (we compared notes, similar findings)
2. tasks in apps...(?)
Moiz: shipping apps with tight deadline: just signups CTA, no tasks suggestions...
Matt: maybe we should embed mobile web team member, gain experience with MobileFrontend
Lila: we won't solve this right now, but keep it on our minds
Steven: we are all on board with making mobile part and parcel
learning lesson from task API, integrate service-oriented development earlier

Lila: question to keep in mind: "who is going to benefit from what you do?"

Targeted acquisition

Steven: on acquisition:

1. Focus on users with high motivation to edit
2. Make login/signup easier
   

Lila: UX guys, you need to look at the two buttons in that screenshot ;)
Moiz: this is *not* the actual version ;)

Task suggestions & notifications

(Steven:)

• Suggest interesting things to edit and showing users how

2 options; either make suggestions immediately, or walk user around a bit first before giving specific tasks
Lila: good idea for a test
Steven: that's a general question for us at WMF – how much handholding?
Lila: let the data show!
Lila: can we show edit count? a bit of gamification?
Dario, Erik: talked quite a bit about this ;)
Lila: sync vs. async communication? (e.g. send email instead of interrupting workflow with notification)
(Steven:)

• Tooltip: see your work in action in view history (a tour) + a let's keep going

Moiz: shows experiment:
point people to edit history page, highlight their entry there. state achievement: "you now know how to ... see all your contributions"
Lila: if the history page UX was better...
Erik: do we have instrumentation for clicks in that kind of workflow?
my intuition is that the own contributions page (and that of other users) could become an important single place for that kind of information, already established for experienced users (I assume from my own experience – need actual data on that)
have existing pathways that users take
Steven: sometimes new users already see the contributions link and assume it contains suggestions
Lila: so that page becomes relevant after like 10 edits, think about augmenting it...
if this was a game, what would be the first achievement/"level" be?
Steven: thought about
Lila: education experience
Steven: careful with game framing – what is the core product value for editors? working theory: it's the sense of achievement, positive accomplishment
Lila: I completely agree, "game" has some wrong connotations for us. I mean users exploring and finding things on their own (as achievement) as opposed to getting lectured
But this is all good stuff.

Key dependencies

Steven:
1. Mobile web and apps
2. Analytics: Aaron's presence is a huge force multiplier for us. But: talked with Dario and Toby about bringing in Morten Wang (author of SuggestBot)
If this works well, storing some kind of interest graph for users – that's really more of an Analytics task.
Toby: actually, Platform would need to be brought in too
3. Community Engagement: during this year, not a lot of help – but because not much was needed. For task suggestions and anon acq, can work with same level. But talking to Rachel about more for e.g. article creation.
Erik: that's Q2 material
(Steven:)
4. Team Practices Group: We are not heavily Scrum-ized (just doing standups, planning retrospectives. I do a lot of that scut work). Could use review, maybe some embedding
Erik: let's first do a needs analysis. They are focused on Platform now
Steven: Yes, OK, it's more of an optimization thing for us, don't have capacity for that ourselves
Overall, feel quite good about resourcing now, dependent on projects of course
re UX team, feels they are maxed out on number of projects they can take on, need ability to focus
Erik: stepping that up now
Moiz: 5-6 projects simultaneously are a lot, especially of several of them need to ship
Steven: also, had a lot of churn, need consistency to avoid onboarding costs
Moiz: yes, Growth may be even more difficult in that regard – need to understand numbers, etc.

Discussion

Lila: I would be happy to maintain a six month + vision roadmap (instead of full year)
Erik: right now, assume that later quarters have less precision – that's kind of accepted
Steven: also, our team depends a lot on test results for further planning
Lila: happy with this meeting, and the way you do this work. Where I disagreed I brought that up, but I'm fine with you going ahead as long as you consider these questions.
Steven: I appreciate the direct feedback. :)
Dario: regarding the external collaboration with Morten, be mindful of his availability in the timeline
Toby: would be good to see which activity is intended to drive what metric
Lila: ideally would like to be able to report directly what was achieved with $x budget
Howie: Haven't talked about velocity, and what the impediments there might be
Steven: With addition of Rob Moen, we should be able to ship a bit faster
Previous sprints, were cleaning up some tech debt, ...
Engineering capacity has now caught up, I and the designers need to keep enough ahead of them
Lila: do we have junior/senior PMs?
Howie: somehow, but it's not structured as one PM reporting to another
Steven: Arthur brought that up to me: could have a dedicated tech lead (owning e.g. code review more)
Toby: so you are scrum master at the same time? that sucks
Steven: yes
Lila: but for 3 person team, scrumming shouldn't take near 50% of one person
Erik: this is what I mean by discovery process for team practices
Kaity: plan for more engineers on the team? Steven: not right now
Lila: so are you guys satisfied with what you get from each other?
Erik: from a distance, it does not seem we are under-resourced on UX right now. Bigger concern: slicing, assignment
Moiz: I would prefer more focus, work on 2-3 projects instead of 5-6 (Growth + other things like mobile)
Lila: engineers, how are you feeling?
Matt: feel OK about capacity – Sam ramped up well, Rob ramping up now
can still get more efficient, improve on Scrum
Howie: what about MediaWiki infrastructure needs?
Matt: Frontend is more modern than backend in MediaWiki  ;)
Toby: had some discussions about research being more involved from the beginning, feel we are evolving in that direction
Aaron: my work is stretched a bit thin, did a lot of the stats for you guys on weekends
would like to focus less on bean-counting, business analyst role might help
Steven: I agree it would be better if Aaron would spend less time on data analyst stuff and more on research stuff. it's mostly not happening because our A/B testing is still manual after 2 years. Early in team's history, we spent a lot of time building infrastructure (Ori's work on EventLogging). I am also aware that Analytics' roadmap is very overloaded
Lila: hearing that a lot of problems could be solved by more junior personnel. At Sugar, we had a program with interns from really good CS schools – sort of what Geoff does in Legal
Dario: one part is better infrastructure for everything – Vital Signs will help with that
regarding internships, we are already discussing that in Research
Erik: ... in partnership with Jared's team