Wikimedia monthly activities meetings/Quarterly reviews/Research, Design Research, Analytics, Performance, January 2016
Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material
Attendees: Dario, Kevin, Chris, Leila, Faidon , Guillaume, Lila, Ori, Greg G., Madhumitha, Abbey, Stephen LaPorte, Abbey, Rob Lanphier, Trevor Parscal
Remote Attendees (BlueJeans attendees): Tilman, Aaron Halfaker, aotto, Dan, dchen, Faidon Liambotis, Gregory Varnum, Joseph Allemandou, Luca Toscano, Mark Bergsma, mforns, Moushira, Nathaniel Schaaf, Nuria Ruiz, QuimLaptop, Samantha,Wes, Chase Pettet
- 1 Research and Data
- 2 Design research
- 3 Analytics
- 4 Performance
Research and Data
Dario: This quarter was really important. Many initiatives that came to fruition and depended on a large group of contributors and fellows.
Dario: Request to clarify who owns what in data and research. Talked to team members to clarify roles and streamline ownership. I've made a list of responsibilities ("Research fingerpost") to see who to go to for what. The map is published on the office wiki and would like to move to a public wiki. This isn't a replacement for individual team processes
Dario: Goal was to bring revscoring to users as Beta Feature. Missed because volunteer time became scarce.
Dario: Achieved actual 90-99% reduction in curation workload. This was a massive gain. Nice blogpost on Wikimedia Germany's blog.
Dario: Communications team announced revision scoring. Received positive community reaction and good media attention.
Lila: Amazing work. How does this connect to previous slide?
Dario: Service is available in many languages via the API.The goal of enabling it to production as a beta feature is to come up with standardized way to integrate it into the user interface (e.g. Recent Changes, etc.)
Lila: Is this an official goal?
Dario: not an official goal but confident we should be able to release in Q3
Lila: This is the first show of what is possible with machine learning. Allows editors to focus on important tasks. Really excited about the capabilities of this.
Dario: Productization refers to underlining service. We started seeing the first third party tools adopting the service.
Dario: Saw up to 16% of all saved translations come from article recs. Good confirmation of early results.
Lila: how many are there?
Leila: About 280??
Dario: This is partly a miss because work that went into other projects. First part was presented at Wednesday's research showcase. I would encourage everyone to watch the presentation.
Dario: Exploring working with other service providers.
Dario: First time we invested into this kind of research . We are trying to understand who our readers are from both a qualitative and quantitative perspective.
Dario: Second part of project is starting now. Had a lot of contributions from the reading team.
Dario: Lila asked if we could come up with template to make services replicable. Came up with first draft of proposal.
Dario: Hired first dedicated software engineer, Nathaniel. Very competitive position.
Dario: 3 papers and 1 peer reviewed poster accepted at major conferences in the field. 3 workshop proposals accepted. Trying to bring results to fruition and engage with academic community to secure new collaborations. Expanded on last year's success of a dedicated Wikimedia workshop with 50 attendees and 21 submissions.
Dario: Worked with Discovery Team to support Big English fundraising
Lila: Look at the impact your team is having on users and so grateful of what the team has been able to accomplish. So grateful
Kevin: Do you still get data analysis requests?
Dario: we do; sometimes we provide some advice, but it's nothing compared to requests in previous quarters
Robla and Quim: It would be great if everyone is as active as the research team is in conferences.
Abbey: Realized as we tried to coordinate with project teams that people did not understand about about process. Decided it would be good to do a little design thinking and try to understand other teams. Conducted workshop where we laid out generic product development process. All of the people in the workshop from various disciplines the outlined all their activities as part of that process.
Abbey: About 40 people attended.
Abbey: There were 4 workshops total. Talked about programatic personas and inspired people to use those as they build products.
Abbey: Question if this should be red or green. Started prototyping lab at the end of December. Going to have more regular sessions of the reading team to understand users. Since we are a small team, want to rely on mentoring and collaboration with other teams to broaden our reach.
Abbey: We need to talk to the people we are building for. It is important for the team to have a dedicated person to find the right people and schedule them. Made a lot of improvements to process this quarter.
Abbey: Worked on integrating tools - database and mass emailing tools. Worked with IT to navigate through that. Abbey: Made efficiencies in how we communicate with participants. For example, May made a gif on how to screenshare on google hangout. We are also using talk pages and querying databases to find the right participants.
Abbey: Working on integrating design research into how we build products. Conducted 11 research products. Need a lot of QA.
Lila: Within the product team?
Abbey: Yes. This may be more of a conversation regarding when something is ready for usability testing. How will be people know when something is ready?
Lila: Quim and Wes - can you make a note? (offline note by RobLa: I have an idea or two on how to resolve the QA problem Abbey brought up...feel free to ping me)
Abbey: Did 5 heuristic reviews and some focus groups to understand users needs?
Abbey: Have a set of programatic personas built from gathered secondary research. Workshopped them with engineers, designers.etc. Iterated them from that workshop and used them to help build products around users. Now we are doing deeper research around users. Conducted about 5 interviews.
Also, working on the recruiting process -important part of the work. We are continually learning about users to understand their needs and how they navigate around software we and the community are building. When we go to Mexico in couple weeks, learning about users in person and how technology is part of their social fabric.
Abbey: Going to do a survey of higher education to understand how they learn and tools the use. Working with them on a weekly basis and developing relationship with this organization. This is another way of expanding reach.
Abbey: All participants have a point person to reach out to if they have questions.
Abbey: The design team is also working with the L&E on implementing their surveys and mentoring on how to do design research. Step back only once team is confident that they have the right tools.
Abbey: Delivered several presentations this quarter. We don't go to conferences at rate that the research team does but would like to in the future
Wes: Really great work this quarter.
Lila: The way you built esteem and input into team has been great - so impressed. Need to make sure every engineer understands design thinking. How do we keep that going? There are some things that have enormous impact on product. Pick one flagship product and use that to understand how users really engage with learning. Part of changing internal engineering thinking is how it is perceived externally. Let's talk about it.
Guillaume: Regarding the work with U of W, students are conducting surveys that will inform possible decisions in editing.
Lila: Those are the things that are important.
Dario: Really want to see if you can take role in consulting/mentorship and to carve out time to figure out what products need support.
Lila: Would like to train product teams.
Abbey: We're luck to collaborate . It's a unique situation we should be able to take advantage of.
Abbey: Bringing two engineers to Mexico. Want to collaborate toward process that we all think will work.
Robla: There's a cultural norm of people not doing code/"real work". We need to remove that barrier.
Wes: There is a nice momentum between design and research team.
Madhu: Pageview API was a big ask and a carry on goal from the last quarter. The team met the goal - APIl was launched and blog post released. Also had a number of requests come through phabrictor and received some great community feedback.
Madhu: Scale is logarithmic. Graph show that the per-article endpoint gets several orders of magnitude traffic compared to the top endpoints,
Lila: External or internal?
Madhu: mostly external
Lila: Wes - Is this API usage presented part of those numbers?
Madhu:Erik Z has been maintaining. Monthly pageviews with breakdown reports have been updated and are available.
Lila: is the URL still the same?
Madhu:As of now this is done, but missed the goal as of the end of the quarter.
Lila: It would really help..
Kevin: 800 hours saved using wikimetrics. This will greatly simplify.
Madhu: Stretch goal was done by services team abut the deploy freeze kept new code being deployed before the end of this quarter.
Madhu: Much better docs for users doing self-service analytics. Now Close to over 400 million devices?? [Madhu - can you confirm]
Lila: Just English?
LIla: Would like to schedule meeting to discuss more. (Lila asked Wes for this)
Madhu: Purple on top is the production hdfs user. Ellery runs a close second - runs a lot of jobs for fundraising. 30 uses running jobs on the cluster in December compared to 15 last June. A lot more users.
Lila: We need to think about capacity.
Kevin: Will need even more servers.
Lila: Great progress.
Dario: Would like to stress the level and quality of documentation. Pretty remarkable and want to thank you for that.
Lila: Want to thank you for all of the work you have been doing - you are the instruments the product team is using to navigate. Great work!
Ori: Unable to calculate the percent difference - there were no reliable metrics for several weeks.
Ori: We did not meet goal because we kept finding low hanging fruit - hard to anticipate how much we will be able to accomplish in a quarter. Should have set a more modest goal in hindsight.
Ori: There's been a lot of work by the ops team on the caching layer. The blue line is backend response time and green line is the user-perceived response time.
Lila: we use our own CDN ,right?
Ori: Use varnish but a lot of our own custom code.
Ori: Ashburn, Amsterdam, SF, Dallas,
Lila: Think about distribution next year?
[Unknown]: Going to include in annual plan which we are working on.
Lila: Need to think about splitting the measurement.
Ori: Its fairly stable over time.
Lila: Think we need to introduce first to read? metric for performance.
Ori: This is the time anything appears on the browser.
Ori: Availability project. Some systems are born distributed; others become distributed; others have distribution thrust upon them,
Ori: All the high level things we set out to do are done but can't say we can flip a switch. There be a lot of bugs resulting in unknown additional work. Will work with the dev ops team.
Ori: The team is still responsible for lion's share in Mediawiki - Over 50% of all commits - largely due to Aaron and Timo
Ori: Image manipulation set of requirements is complicated . Proposed solution is to move out of MediaWiki so people can run locally. Want to get it formally vetted to make sure we are good to beign rollout next quarter.
Ori: A little tricky because there is a dependence on varnish 4. Tech ops has reviewed. There is block but they hope to rollout soon.
Ori: Put a lot of work into job queue.
Lila: Can you explain the bottom of the graphs?
Ori: We currently have access to actual Iphones and androids that are submitting metrics to us. Mobile performance still has a way to go.
Ori: For the next quarter, both Petr and Timo will be collaborating with reading team on base load of images.
Dario: Many practices that editors follow. Many changes that are invisible to community that we could advise upstream editors on
Ori: Onwiki performance inspector is a major goal for next quarter. Planning to have a dashboard that will appear and show a lot of the performance data. This will give editors a sense of how article will load in different parts of the world.
Lila: Should make our editor tools smarter. Lets start thinking about editor's experience. Conversation with James will definitely be helpful.
Lila: Great session. Very informative and great results. Thanks everyone.
Wes:Split into two groups so we can go through review in regular pace seems to have worked out much better and we should continue this format.