Wikimedia Foundation metrics and activities meetings/Quarterly reviews/Editing, Collaboration and Language Engineering, April 2015

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search

Notes from the Quarterly Review meeting with the Wikimedia Foundation's Editing, Collaboration and Language Engineering teams, April 16, 2015, 9:30 - 11:00 PDT.

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Present (in the office): Erik Moeller, Danny Horn, Roan Kattouw, Trevor Parscal, Neil Quinn, Elena Tonkovidova, Erik Bernhardson, Lila Tretikov, Pau Giner, Terry Gilbey, Garfield Byrd, James Forrester, Tilman Bayer (taking minutes), Wes Moran, Toby Negrin, Damon (from 10:05); participating remotely: Damon Sicore (later, in person), Niklas Laxstrom, Santhosh Thottingal, Runa Bhattacharjee, Kartik Mistry, Amir Aharoni, Sherry Snyder, Ed Sanders, Geoff Brigham, Matt Flaschen

Presentation slides from the meeting

Danny: Welcome


slide 2

[slide 2]

re stretch goal: felt it wasn't possible to run a test of sufficient quality yet
Trevor: i.e. data quality

slide 3

[slide 3]

achieved burndown goal
had weekly triage meetings, public, not a lot of community participation, but helpful for identifying priority bugs
don't have overall number for speed improvements, but it's far below benchmark
VE (load) is now faster than wikitext editing for a lot of users
Lila: this rocks
Trevor: Ori was pretty helpful too
James: this turned our attention to other perf issues, e.g. [load times for users in low bandwith regions like] India
Lila: still, the fact that we are here three months later and are able to say that is great
Trevor: this was important
Lila: how can we go about sandbox testing?
Roan: did synthetic benchmark one time (simulating typical North American connection only)
--> Lila: can we do that [sandbox testing / synthetic perf benchmarks] as part of continuous integration?
Trevor: it would be awesome if Jenkins -1ed your code because of such perf regressions
Terry: why aren't we doing it?
would resourcing help?
Lila: it's resourcing and who should be responsible for it
James: also, some tradeoffs
e.g. could make VE faster at expense of readers, but don't want to do that
autofilled citations
Lila: feedback?
James: positive, and constructive about those bits that don't work yet
Lila: major roadblocks?
James: no
Trevor: this is more for enwiki, other languages are more challenging
James: (lists other languages where it's live)
thanks to Ori, Mariel and Alex too
Damon: this is exactly the kind of success I want to see, very happy about outcome

slide 4

[slide 4]

misses: A/B test
--> Lila: next time have KPIs at the top level
great job
you guys were the guinea pig for running this kind of integrated team
Trevor: bug nomination process was different this q
not a lot of live community participation in triage, but a lot of contributed bugs beforehand
--> Lila: give the numbers for that [community-contributed bugs] to Luis' team
Trevor: they come from many channels, e.g. talk pages
don't yet feed result back a lot to those channels
--> Lila: should automate that more [reporting the closure of bugs that had been raised outside of Phabricator, e.g. on talk pages, back in that original channel where they had been reported]


slide 5

[slide 5]

strategy: reach out to different communities, identify what's important for them
didn't reach any of the objectives, but 4 of these are on track for end of April
Trevor: LQT to Flow conversion...?
Danny: not related to these...

slide 6

[slide 6]

VE team did great job enabling multiple instances of VE on one page
now use VE as preview for wikitext editing
built custom mention feature, with autocomplete, worked really well
made contact with some new people (communities), e.g. Hebrew
Telugu and other small languages: friendly community, eager to try new features
Erik: team became good at listening to users and addressing concerns (e.g. about editing other editors' posts, threading model)
system is changing in response to user feedback and that is great
Lila: agree
--> Lila: identify lessons learned [from Flow team's changed approach in interacting with users/communities], so they can be applied elsewhere
do a before/after comparison overview
present that [to community] e.g. in metrics meeting, my talk page
Trevor: excited about this
Lila: emphasis not on deployment [at a set time], but on getting it right
how decoupled is this UX technology from backend? is it going through APIs?
Danny: yes
ErikB: everything in UI is going through [JS?] API, so bots can do that too
Lila: compartmentalized?
ErikB: partly yes, partly no
Lila: should have strategy for making this kind of tool work in [mobile] apps
Matt: big win; some initial experimentation in mobile web
expect people will at one point build fully native experiences
UX won't be identical

slide 7

[slide 7]

goals were ambitious
ErikM: Co-op is mentorship program supported by WMF grant, may want to present that at some point
Danny: search is really needed
ErikB: all search code review done now, just several weeks later
Terry: too optimistic because of lack of resources, or because of dependencies?
Danny: down on resources, lost team member at end of Q2, only hired recently
Trevor: seems there were a lot of externalitites too. more than in previous qs?
Danny: goals were too ambitious
--> Lila: separate objectives into core goals and stretch goals [for Q4]


slide 8

[slide 8]

Joel left team
goal was to expand presence of CX, and add some features
now 23 Wikipedias, 20 by end of Q3
most new languages came through community requests
see appendix for # of translators, articles created
last goal: have data, see needed work for ULS

slide 9

[slide 9]

have developed a comfortable pace for (new language deployment and) bug prioritization
Lila: are we able to handle all the inbound requests from community?
Runa: as of now we are; have a checklist, count number of days from request to deployment
apart from cases where there is a very specific blocker, have been able to complete all requests within 10-15 days
Lila: ready to push the tool out more?
Runa: currently waitlist of 25 Wikipedias that we are processing now, and some specific events
took a break for a bit before quarterly review
Lila: think about topic-specific campaigns (e.g. translate math / physics / medicine articles)
program, not just a tool
connect with others on that
turn on the spigot
Trevor: connect with Luis on this
Lila: and Katherine
--> Lila: connect with Luis and Katherine to discuss topic-specific translation campaigns
Pau: Catalan community organised a translation campaign and suggested the use of Content Translation
Runa: introduction of new entry points worked really well

slide 10

[slide 10]

analytics needs were a diversion for team, outside our area of expertise
can't anticipate full scope of this, need more guidance from Analytics team
Lila: why do such a huge percentage of users not go on to translating a second article?
Runa: we don't quite understand it yet
Lila: have we reached out to these users?
Runa: some of them
some small Wikipedias hard to get hold of with outreach on village pumps or Facebook groups
Lila: how many of these translators do we have email addresses for?
Runa: don't know
Lila: critical to understand such roadblocks
figure out how many of these users can be reached directly
why are they not continuing to use the tool?
Trevor: have you worked with Abbey's group?
Pau: have been doing a lot of user research
first on translation tool
might just be a matter of time, e.g. released the new entry points only recently
but it's indeed a signal that is good to focus on
Lila: btw, I love the infographics
--> can we have a funnel overview next time? how many people start/ complete...
Damon: slide 16 is useful for this

slide 16

[slide 16]

Pau: (explains contributions page, contributions menu)
Damon: do we have an analysis of the contributions page, and where people are going
Pau: for people who enable beta, 60% are going to CX
Damon: should work more on this, also for other things
thanks Runa, great stuff
Runa: (shows contributions menu)
Lila: love the appendix slide

slide 12

[slide 12]

Flow unique users per week (from --> "unique-users")
Lila: these spikes correspond to launch points? yes
dropoff after launch suggests people try it out, then leave
Danny: yes, there are test pages where people try it out
Damon: numbers are still small
ErikB: intentionally restrained deployment
Lila: general comment: graph slides such as this [unique Flow users per week] are really helpful, but expect such questions [about reasons for spikes] and provide interpretations in advance

Feedback on meeting format[edit]

Terry: feedback?
Danny: worked well, no complaints
Roan: seems that what we did in 1h before can be done in 20min
Trevor: this meeting had a more positive vibe than previous ones
Lila: well, you guys did great this q ;)
Trevor: tried to tell story in text instead of charts
maybe should have less text on tables
Neil: [slide deck] both for presentation and reports, isn't an ideal marriage
Lila: should go through some kind of massage before it's published
Elena: I liked how team was really focused on achieving a very high quality product at end of quarter, responsiveness was great
ErikB: not sure what to improve
Pau: opportunity for team to think in new ways about how to improve products for users
James: very happy with format, better than last time
Runa (on hangout chat): I really liked the 3 slide format. It was easier to focus on the important things without going too deep into details that may have lacked context for people outside the team.