Wikimedia Foundation Annual Plan/Quarterly check-ins/Audiences2 Notes May 2018
Present (in the office): Katherine, Abbey, Nirzar, Marshall, Megan N, Joady, Josh W, Lani, Jaime, Heather, Victoria, James F, Tilman, Toby, Jon K , Danny H, Grace, Josh M
participating remotely: Michael H. (scribe), Adam B, Alex, Carolyn, Charlotte, Cooltey Feng, Corey, Danny Horn, deb, Dmitry, Eileen Hershenov, Jazmin, Joady, Lisa, Lydia, Maggie Dennis, Margeigh Novotny, MarkTraceur, Michael, Olga, Quim, Ramsey, rho, Stephen N, Toby, Volker E.
- 1 Slide 1
- 2 Slide 2
- 3 Slide 3
- 4 Slide 4
- 5 Slide 5
- 6 Slide 6
- 7 Slide 7
- 8 Slide 8
- 9 Slide 9
- 10 Slide 10
- 11 Slide 11
- 12 Slide 12
- 13 Slide 13
- 14 Slide 14
- 15 Slide 15
- 16 Slide 16
- 17 Slide 17
- 18 Slide 18
- 19 Slide 19
- 20 Slide 20
- 21 Slide 21
- 22 Slide 22
- 23 Slide 23
- 24 Slide 24
- 25 Slide 25
- 26 Slide 26
- 27 Slide 27
- 28 Slide 28
- 29 Slide 29
- 30 Slide 30
- 31 Slide 31
- 32 Slide 32
- 33 Slide 33
- 34 Slide 34
- 35 Slide 35
- 36 Slide 36
- 37 Slide 37
- 38 Slide 38
- 39 Slide 39
- 40 Slide 40
- 41 Slide 41
- 42 Slide 42
- 43 Slide 43
- 44 Slide 44
- 45 Slide 45
- 46 Slide 46
- 47 Slide 47
- 48 Slide 48
- 49 Slide 49
- 50 Slide 50
- 51 Slide 51
- 52 Slide 52
- 53 Slide 53
- 54 Slide 54
- 55 Slide 55
- 56 Slide 56
- 57 Slide 57
- 58 Slide 58
- 59 Slide 59
- 60 Slide 60
- 61 Slide 61
- 62 Slide 62
- 63 Slide 63
- 64 Slide 64
- 65 Slide 65
- 66 Slide 66
- 67 Slide 67
- 68 Slide 68
- 69 Slide 69
- 70 Slide 70
- 71 Slide 71
- 72 Slide 72
- 73 Slide 73
- 74 Slide 74
- 75 Slide 75
- 76 Slide 76
- 77 Slide 77
- 78 Slide 78
- 79 Slide 79
- 80 Slide 80
- 81 Slide 81
- 82 Slide 82
- 83 Slide 83
- 84 Slide 84
- 85 Slide 85
- 86 Slide 86
- 87 Slide 87
- 88 Slide 88
- 89 Slide 89
- 90 Slide 90
- 91 Slide 91
- 92 Slide 92
- 93 Slide 93
- 94 Slide 94
- 95 Slide 95
- 96 Slide 96
- 97 Slide 97
- JK: Note also there's a readers metrics meeting on 30 May, this is just highlights of the interventions we've taken.
- JK: red line on the left represents where pageviews had been Jan-march
- Overall, metrics (pageviews) are looking a little better than last year
~8b in March, which is around the higher end of last year
- These represent only pageviews; we'll work on other page interactions in the future, probably June.
- JK: Data is still an issue for us (TN: all of audiences, not just reading), as it has been for many quarters; so it'll be one of two big goals for next FY. Over the next few months we'll figure out what this means specifically. Traditionally, each team had its own approach; we have hired Data Analysts to help us standardize.
- We have traditionally focused on Readers, but are finding that mobile editing is an impediment. We will increasingly focus on mobile editing.
- TN: This fits into our platform view of the world; building platforms to create great experiences for readers as well as mobile editors.
- Most of our traffic comes from Google Search; things they do (or don't) impact us greatly. We are working with an SEO consultancy to get a handle on this.
- Marshall: Big goal for coming year. Figuring out how to get a head-start on that work.
- I am talking to as many people/perspectives as I can to get a handle on:
- What should our framework be for thinking about measurement?
- What are our needs for data collection?
- Needs for deliverable creation?
- MM: Two big questions:
- Where do we measure?
- How do we measure?
- MM: We want to have a "garden" of data -- be able to find carrots when we need carrots, etc.
- We have a jungle -- can feel disorganized. (N.B.: Some orgs have a desert—no data!)
- As this chart shows, we currently have the same metrics in multiple sources -- hard to know where to look for what
- Victoria: Shared with Nuria?
- MM: Yes.
- Victoria: Would like to go through it with you?
- Heather: Also Comms.
- Maggie: And CE!
- Josh W: Cross-departmental.
- ACTION: Marshall to present to Victoria, Heather, and Maggie
- TN: That would be an awesome discussion; we have ideas about distribution channels
- KM: Should Maggie be involved
- Maggie: I would like to
- Victoria: My sense was that on the product side, we didn't necessarily have a clear idea of what it was we wanted to measure
- Jon K.: There's areas for improvment throughout
- Victoria: Also complicated is that the customers for this data are not only WMF PMs but the community at large; we have multiple stake holders
- MM: I agree. We have made a lot of good tech decisions, need to contiue to improve
- TN: We need to have a scoping discussion; need to be careful about setting concise goals and figuring out how to get there.
- Page previews:
*Last Q we worked on deploying page previews to english, german Wikipedias. Goal is to reduce the cost of exploration. People can interact w/ content from multiple pages w/o opening them all.
- This is probably the biggest change to the desktop experience in a decade, so we wanted to be very careful.
- Analyzed data and published a/b tests; worked with community, comms.
- (page previews examples)
- Now that we're live we can confirm some of our a/b tests results. page previews are used ~50m times/day. If we consider pageviews and page previews together, we can see that people are interacting with about 25% more pages. But note that this includes duplicates; it's counted twice if the user views a preview then opens a page. But we removed dupes in our a/b tests.
- Feedback on the internet was very positive.
- a/b test results:
- - 21% more page interactions despite 3-5% fewer pageviews
- - v. low disable rates (0.01% on enwiki, dewiki)
- We were worried that lower pageviews would negatively affect donation rates, but that didn't happen; in fact page previews may have had a positive impact.
- Worked with comms on well-received blog posts
- the feature got a lot of press
- (press coverage examples)
- Work with communities:
- enwiki community decided not to deploy this 2 years ago. It's good to see that we were able to address their concerns.
- JK: Along with ACTRIAL this is one of the few times we've done an a/b test with the community and saw a big impact
- Olga; We were able to answer community questions immediately because of having data on hand
- Maggie: From a user perspective, this is one of the "best things since sliced bread" features, thank you!
- KM: Yeah, this is awesome.
- TN: It's cool particularly with areas like In the News -- helps with recall
- Between this and compact language links we shipped 2 new things
- KM : People are liking a change on Wikipedia ;)
- Heather: introducing a new design
- PDF rendering: We had to wait until this Q to deploy it, but hoping to hand off to RI this quarter
- Currently the PDF Download button's only on Chrome on Android, and is used ~100-200k times a day; we hope to add to more browsers later
- Olga: Next big project is page issues for the mobile website; these are currently hidden on mobile, which makes it more difficult for readers to gauge quality/reliability compared to desktop. We hope that changing the way we display this will improve transparency and trust.
- Development is planned for this quarter; planning and investigation happened last quarter.
- TN: Community members were not happy that page issues were not shown on mobile, to the point of creating templates that did not result in a good reading experience.
- Alex H: (slide depicts page issue on desktop web)
- (slide depicts page issue link on mobile web)
- Clicking on the grey text (which is a link) displays a modal dialog; but 0 participants in the user testing group noticed this even when prompted to evaluate the reliability of the page.
- We experimented with a few different designs, trying to raise awareness without being too distracting. The slide depicts one of two prototypes we tested.
- (examples of issues)
- They range from issues mainly of interest to editors, to things like neutrality or lack of sources that are of interest to the general reader. One of the broader implications we were thinkign about was the "fake news" question.
- Our goals:
1. Make the readers more aware of page issues 2. Evaluate how awareness affects readers' judgments about page quality/reliability
- Results: We saw higher awareness of page issues, and readers appreciated them enthusiastically. (Note that the sample size is small so this wasn't statistically significant; but it's much better than zero awareness.) We found that they increased *trust; they may encourage editing but this needs more investigation.
- (Tilman: )
- By exposing these, we are increasing awareness that Wikipedia is created, monitored, etc. by a community.
- See link on this slide to read the full results.
- NP: We were also able to improve our settings greatly last Q.
- Main goals were around transparency of benefits around opting into beta, and allowing changing text size. Our goal is to increase WP Beta participants.
- JK: Huge issue: beta opt-in users not reflective of the general population. Things that look great in beta don't always turn out that way.
- CG: Last Q, we meant to pilot an awareness campaign in Nigeria; this was delayed due to an issue with our partner.
- We are experimenting now with localized store descriptions and app store assets. We don't have statistically significant results yet but hope to share at next QCI. We want to figure out what features are most important to users in various markets and be able to tailor app store descriptions to them.
- We are delighted to announce that synced reading lists are released! (i.e.: adding to a list on one device will make the change available on all other devices). This was a long-standing request and users are quite happy about it.
- Next quarter we want to improve the experirence for multilingual users: explore feed, search
- We'll be doing user research on this at the hackathon and will have results next quarter.
- (depictions of reading list syncing)
- (representative user feedback examples)
- KM: What % of users are using this feature?
- CG: see below
- CG: The feature has only been out 2 weeks, so too early to see if it drives repeat usage; but slightly over half of people who creating reading lists enabled syncing. 108k users have created lists. This number seems smallish so we're going to run an *awareness campaign this quarter. We've also seen a spike in account creation, now tapering off (currently ~1k new accounts/day after a peak of 1.4k/day).
- (Tilman: just a note that A/B testing such design changes is still a bit of a technical challenge in our technical context (considering our privacy constraints and current lack of a general framework for variant testing); the web team is putting some work into this instrumentation currently. )
- (pics of reading lists on iOS)
- iOS and Android released synced reading lists simultaneously.
- This feature is part of our 2-yr arc of the 'better encyclopedia experience' work. One thing we're interested to see is how this affects 'rabbit-holing' across sessions. We want to know how to support this in a mobile-context, where you don't necessarily have the option of opening a bunch of tabs.
- Next Q we'll work on some of the customizability that Android did last Q. This is in keeping with our 'portfolio iteration' strategy wherein the apps alternate developing features so that we can learn from each other.
- Android did a gradual release and so did iOS. Graph shows usage: blue lines are additions to lists and yellow is checks for updates.
- (usage stats since release)
- JM: RI is focused on consolidating code that has traditionally been distributed among multiple sources despite identical functions, duplicating maintenane burden.
- page library
- page content service
- reading lists service
- A benefit of this approach is that it allows us to adopt new tech across multiple platforms quickly. For instance, we were able to use the existing API used in the apps to quickly create a browser extension.
- TN: The upload to commons app developed by the community will begin using our APIs as well.
- JM: Similariy, we were able to use a shared API backend for page previews on the web and link previews in the apps.
- Ramsey: 3D is here! Launched end of Feb. We have a few hundred files uploaded so far; we're working with a project called Scan the World who are uploading more content. Curators and editors are excited, and are using the files on-wiki.
- Last Q our goal was to support 3D which we did; this Q we'll be working to fix bugs and ?? STL format.
- (examples of on-wiki file usage)
- Note: one example is a wikidata page; what's going on there?
- A: '3D model' is now a Wikidata property
- Partnership opportunities: Smithsonian reached out, and wants to tie our APIs together; talking with Oculus (Facebook) and Magic Leap about supporting new formats and educations usage; and Shapeways wants to donate a portion of the price of 3D files to Wikipedia.
- Structured Data on Commons: We're taking a "crawl, walk, run" approach. Last Q prototyped the most basic possible feature; all of our goals were reached. We prototyped changes to uploadwizard, file page, and ??
- Screenshot of the new commons file page combining wikidata with wikitext; this was one of the trickier prototypes. Thanks to Mark for working this out from a starting point of little Wikidata knowledge.
- Prototyped changes to UploadWizard which are still a work in progress. Working with the community. This is a way easier change than the File page.
- JK: Lessons learned from this period—main themes:
- coordination is something we're getting better at but still need to work on
- Grace: Added "technical" to distinguish from other work.
- Grace: We focused on delivering customer value; the product manager defines customer value and program manager makes it happen
- We work behind the scenes to ensure progress
- I hired Lani on 1 March and that's going great.
- Hired Jazmin on 7 May.
- Max works on Multimedia team
- Max and Jazmin are going to Barcelona for offsites/Hackathon
- I am partnering with Dan Garry on TemplateStyles; Dan decided not to target ruwiki after getting pushback, we deployed to Wikivoyage, then dewiki, ruwiki, and others wanted it
- Getting some traction on Scrum of Scrums
- Finance asked me to develop an approach for financial decisionmaking during C-Team meetings
- Bringing SoS accountability model to Audiences
- Improving transparency (e.g., decisions about conference attendance)
- Margeigh: Hi from New York.
- I'll start by talking about my first Q and some of my goals around transparent team practices - how research and strategy fits within audiences.
- How do we gather and capture the questions that are coming up on different platforms as they come up?
- How do we collaborate with other teams?
- How can we standardize how research and design work together?
- (depicts the "design research pipeline")
- We're coming together as a team to decide which questions are highest-impact and most urgent to answer. Figuring out how to cluster Qs and who gets resources.
- Another thing I've been tackling is: what is this function and its purview? What kind of activities and outputs need to get generated to inform product strategy? Can encompass anything from explaining on the one hand to running experiments on the other. Looks like a lot of stuff but we have to be partnering with design, analytics, research, comms.
- VC: I"m sure CE is on this list as well.
- ACTION: MN to add CE to upper left quadrant
- MN: Yes, see upper left quadrant.
- MN: Nirzar and I have been trying to standardize titles and expectations/accountability for each title. It's a work in progress but has been shared with T&C.
- Another thing I'm working on is ensuring that Audiences aligns on a 3-5 year plan. Having a series of work sessions to connect movement strategy and the more tactical 3-5 year strategy. I think we'll be involving other teams but that will be scheduled in coming weeks.
- After having participated in the movement strategy track in WMCON, it's important that product people participate in the working groups.
- TN: I asked MN about how product could interact with the movement strategy and this plan is an outcome of that.
- Bigger picture for our function is to provide better data to support product decisions. I'm in NYC today to work on mobile user personas. The product teams want these in order to decide what audience segments to focus on.
- I'm about to start a project with Neil trying to provide a more nuanced understanding of how the different Wikimedia projects differ from one another. What are their distinctive characteristics and what to they have in common? How old are they? How many users do they have? How many pages, pageviews? How many admins?
- We have ~30 different such parameters. The point is to be able to cluster different projects so that we can start to track more nuanced metrics. Rather than trying to move the needle on all at once, we want to enable a finer-grained focus.
- Abbey: Framework of human-centred design we use to pick out what we're working towards.
- Here's where some of our projects fall on this framework.
- We're starting to work on in-context help w/r/t new editors. Doing mobile personas for iOS and for Android in India.
- On Wikipedia workflows, I've worked very closely with James F to gather all of the possible workflows in order to create a contribution taxonomy. In the end this will be made public for anyone to use.
- (the high-level framework)
- (zoom-in on some of the 88 workflows identified)
- TN: These are the proecesses by which content is created on the Wikipedias
- AR: Yes
- Scoring and weighting
- Red criteria are the abilities an individual needs to be able to complete the workflow
- Blue depicts opportunities for mobile
- Also looking at how much knowledge, social capital, etc. a user needs to complete various workflows
- There's also weighting so that teams can decide what workflows are most important to focus on
- TN: To draw the arc back to Jon's point, this is how we're going to assess to what extent mobile editing workflows can be supported on mobile
- (ex: workflow for copyvio detection)
- These will be publicly available for anyone to use. WMF teams are already using it.
- Lydia (WMDE, product manager for wikidata): Let's talk about Wikidata.
- We had two large annual goals:
- Increasing number of volunteers who benefit from Wikidata
- We are on track in usage; a bit behind in terms of data quality.
- . Making wikidata more useful for other open-data projects. Getting more people to install Wikibase outside Wikimedia. This is happening. We're a bit behind on success stories; this will just take time as people blog, etc.
- Goal: Wikdata use in the Wikimedia projects. Projects including English WP had questions about to what extent recent changes are shown. We've improved this by showing 2/3 less changes, consisting of the most relevant changes.
- Graphs show lowering load on the Job Queue.
- Goal: data quality
- We've been working on this for a long time. Last Q we rolled out constraint checks (rule-based checks for common mistakes).
- (depicts a dashboard providing for quick revision of vandalism)
- (depicts a constraint report) -- showing things that might be wrong/indicate mistakes in how the data should be modeled
- positive feedback.
- Goal: support for lexicographical data. We've had lots of discussions around roll-out and licensing with the community.
- Goal: wikibase adoption outside Wikimedia
- We're working on a Docker container to ease installation
- Goal: data partnerships
- We've had talks with various orgs, seen some good blog posts about their use
- For Wikidata query service, we made some small improvements to make it more responsive and so on. A GSoC student will make further improvements this summer. Usage is picking up.
- People are developing new tools to make it easier to add descriptions, make it easier for third parties to use, lots of other cool stuff. WDCM Journal—highlights interesting things we found while analyzing how Wikimedia projects are using
- We got our own DB servers, yay, thanks SREs!
- A lot of work was done on search integration; thanks to the search team.
- Showcase: Wikidata-powered infoboxes on Commons
- (interesting example of WD data)
- (interesting example of WD data)
- (example of WD data)
- Example of an actionable use of Wikidata: providing statistics about the gender gap in the Wikimedia projects. Can see the global gender gap (in article subjects);
- ... or can see this broken down by project!
- TN: I asked Lydia for example of something actionable and this was a good one. We are working with Research on contributor diversity but hadn't seen data on content diversity; this is a good last demo.
- Tilman on chat: http://wigi.wmflabs.org/ ("Wikidata Human Gender Indicators") has already existed for a while
- Scorecards for ongoing work (showing metrics about usage/coverage and changes therin over time)
- Note: There's only so much we can do in terms of telling people what kind of data they can put into Wikidata.
- TN: That turns around a trend/is an uptick, right?
- Lydia: Yes.