Wikimedia Foundation metrics and activities meetings/Quarterly reviews/Editing, Collaboration and Language Engineering, April 2015
Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material
Present (in the office): Erik Moeller, Danny Horn, Roan Kattouw, Trevor Parscal, Neil Quinn, Elena Tonkovidova, Erik Bernhardson, Lila Tretikov, Pau Giner, Terry Gilbey, Garfield Byrd, James Forrester, Tilman Bayer (taking minutes), Wes Moran, Toby Negrin, Damon (from 10:05); participating remotely: Damon Sicore (later, in person), Niklas Laxstrom, Santhosh Thottingal, Runa Bhattacharjee, Kartik Mistry, Amir Aharoni, Sherry Snyder, Ed Sanders, Geoff Brigham, Matt Flaschen
re stretch goal: felt it wasn't possible to run a test of sufficient quality yet
Trevor: i.e. data quality
achieved burndown goal
had weekly triage meetings, public, not a lot of community participation, but helpful for identifying priority bugs
don't have overall number for speed improvements, but it's far below benchmark
VE (load) is now faster than wikitext editing for a lot of users
Lila: this rocks
Trevor: Ori was pretty helpful too
James: this turned our attention to other perf issues, e.g. [load times for users in low bandwith regions like] India
Lila: still, the fact that we are here three months later and are able to say that is great
Trevor: this was important
Lila: how can we go about sandbox testing?
Roan: did synthetic benchmark one time (simulating typical North American connection only)
--> Lila: can we do that [sandbox testing / synthetic perf benchmarks] as part of continuous integration?
Trevor: it would be awesome if Jenkins -1ed your code because of such perf regressions
Terry: why aren't we doing it?
would resourcing help?
Lila: it's resourcing and who should be responsible for it
James: also, some tradeoffs
e.g. could make VE faster at expense of readers, but don't want to do that
James: positive, and constructive about those bits that don't work yet
Lila: major roadblocks?
Trevor: this is more for enwiki, other languages are more challenging
James: (lists other languages where it's live)
thanks to Ori, Mariel and Alex too
Damon: this is exactly the kind of success I want to see, very happy about outcome
misses: A/B test
--> Lila: next time have KPIs at the top level
you guys were the guinea pig for running this kind of integrated team
Trevor: bug nomination process was different this q
not a lot of live community participation in triage, but a lot of contributed bugs beforehand
--> Lila: give the numbers for that [community-contributed bugs] to Luis' team
Trevor: they come from many channels, e.g. talk pages
don't yet feed result back a lot to those channels
--> Lila: should automate that more [reporting the closure of bugs that had been raised outside of Phabricator, e.g. on talk pages, back in that original channel where they had been reported]
strategy: reach out to different communities, identify what's important for them
didn't reach any of the objectives, but 4 of these are on track for end of April
Trevor: LQT to Flow conversion...?
Danny: not related to these...
VE team did great job enabling multiple instances of VE on one page
now use VE as preview for wikitext editing
built custom mention feature, with autocomplete, worked really well
made contact with some new people (communities), e.g. Hebrew
Telugu and other small languages: friendly community, eager to try new features
Erik: team became good at listening to users and addressing concerns (e.g. about editing other editors' posts, threading model)
system is changing in response to user feedback and that is great
--> Lila: identify lessons learned [from Flow team's changed approach in interacting with users/communities], so they can be applied elsewhere
do a before/after comparison overview
present that [to community] e.g. in metrics meeting, my talk page
Trevor: excited about this
Lila: emphasis not on deployment [at a set time], but on getting it right
how decoupled is this UX technology from backend? is it going through APIs?
ErikB: everything in UI is going through [JS?] API, so bots can do that too
ErikB: partly yes, partly no
Lila: should have strategy for making this kind of tool work in [mobile] apps
Matt: big win; some initial experimentation in mobile web
expect people will at one point build fully native experiences
UX won't be identical
goals were ambitious
ErikM: Co-op is mentorship program supported by WMF grant, may want to present that at some point
Danny: search is really needed
ErikB: all search code review done now, just several weeks later
Terry: too optimistic because of lack of resources, or because of dependencies?
Danny: down on resources, lost team member at end of Q2, only hired recently
Trevor: seems there were a lot of externalitites too. more than in previous qs?
Danny: goals were too ambitious
--> Lila: separate objectives into core goals and stretch goals [for Q4]
Joel left team
goal was to expand presence of CX, and add some features
now 23 Wikipedias, 20 by end of Q3
most new languages came through community requests
see appendix for # of translators, articles created
last goal: have data, see needed work for ULS
have developed a comfortable pace for (new language deployment and) bug prioritization
Lila: are we able to handle all the inbound requests from community?
Runa: as of now we are; have a checklist, count number of days from request to deployment
apart from cases where there is a very specific blocker, have been able to complete all requests within 10-15 days
Lila: ready to push the tool out more?
Runa: currently waitlist of 25 Wikipedias that we are processing now, and some specific events
took a break for a bit before quarterly review
Lila: think about topic-specific campaigns (e.g. translate math / physics / medicine articles)
program, not just a tool
connect with others on that
turn on the spigot
Trevor: connect with Luis on this
Lila: and Katherine
--> Lila: connect with Luis and Katherine to discuss topic-specific translation campaigns
Pau: Catalan community organised a translation campaign and suggested the use of Content Translation
Runa: introduction of new entry points worked really well
analytics needs were a diversion for team, outside our area of expertise
can't anticipate full scope of this, need more guidance from Analytics team
Lila: why do such a huge percentage of users not go on to translating a second article?
Runa: we don't quite understand it yet
Lila: have we reached out to these users?
Runa: some of them
some small Wikipedias hard to get hold of with outreach on village pumps or Facebook groups
Lila: how many of these translators do we have email addresses for?
Runa: don't know
Lila: critical to understand such roadblocks
figure out how many of these users can be reached directly
why are they not continuing to use the tool?
Trevor: have you worked with Abbey's group?
Pau: have been doing a lot of user research
first on translation tool
might just be a matter of time, e.g. released the new entry points only recently
but it's indeed a signal that is good to focus on
Lila: btw, I love the infographics
--> can we have a funnel overview next time? how many people start/ complete...
Damon: slide 16 is useful for this
Pau: (explains contributions page, contributions menu)
Damon: do we have an analysis of the contributions page, and where people are going
Pau: for people who enable beta, 60% are going to CX
Damon: should work more on this, also for other things
thanks Runa, great stuff
Runa: (shows contributions menu)
Lila: love the appendix slide
Flow unique users per week (from http://flow-reportcard.wmflabs.org/ --> "unique-users")
Lila: these spikes correspond to launch points? yes
dropoff after launch suggests people try it out, then leave
Danny: yes, there are test pages where people try it out
Damon: numbers are still small
ErikB: intentionally restrained deployment
Lila: general comment: graph slides such as this [unique Flow users per week] are really helpful, but expect such questions [about reasons for spikes] and provide interpretations in advance
Feedback on meeting format
Danny: worked well, no complaints
Roan: seems that what we did in 1h before can be done in 20min
Trevor: this meeting had a more positive vibe than previous ones
Lila: well, you guys did great this q ;)
Trevor: tried to tell story in text instead of charts
maybe should have less text on tables
Neil: [slide deck] both for presentation and reports, isn't an ideal marriage
Lila: should go through some kind of massage before it's published
Elena: I liked how team was really focused on achieving a very high quality product at end of quarter, responsiveness was great
ErikB: not sure what to improve
Pau: opportunity for team to think in new ways about how to improve products for users
James: very happy with format, better than last time
Runa (on hangout chat): I really liked the 3 slide format. It was easier to focus on the important things without going too deep into details that may have lacked context for people outside the team.