Wikimedia monthly activities meetings/Quarterly reviews/Reading and Community Tech, October 2016

Reading/Community Tech Quarterly review, October 18, 2016
Please review: https://office.wikimedia.org/wiki/Quarterly_reviews

The pdf of the WMF's reading and community tech team's quarterly reviews for the July-September 2016 period (Q1 2016-2017)

Etherpad:https://etherpad.wikimedia.org/p/Reading_CommunityTechQuarterlyReviewOctober2016

Etherpad Template Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Present: In-person: Toby Negrin, Jon Katz, Danny Horn, Ryan Kaldari, Nirzar Pangarkar, Zhou Zhou, Michelle Paulson, Carolyn Li-Madeo, Josh Minor, Tilman Bayer, Katherine Maher, Jon Robson, Joady Lohr Remote: Maggie Dennis, Wes, Adam B, Anne, Baha, Bryan Davis (bd808), Bernd Sitzmann, Corey, Dmitry, Heather Walls, Joe Walsh, Leon, michael holloway, Niharika, Sherah, Stephen Niedzielski, Olga, Piotr, Darian Patrick, Joaquin, Sam S, Rita

Reading

Slide 1

Toby: High level observations: About the team: Every Q we have a Quarterly Review review. PMs, Eng leads, others go through the deck. Work on the slides, v. collegial. I like it a lot. Exciting to see teams' work throughout the quarter. Teams: you're doing great work and it matters. Thanks for another great Q.

JK: New faces on both teams: Sam Wilson, Carolyn, Piotr (developer), Olga (PM). JK: Format: talk a little bit about who we are and what we do. Each team lead gets to talk about what their teams achieved in the past quarter.

NOT getting to most metrics (we never do) -- we'll talk those over in a separate mtg in ~ 2wks.

We're here to serve readers of WM projects. To date we've served mainly Wikipedia, other hosted projects.

There are parts of the world where we're widely known and we want to assist users to use our stuff.

In most of the world we're not well known or widely used; we're working to remove barriers.

Slide 4

The bulk of the effort goes to the first problem: enhanching the online encyclopedia experience.

New readers effort.

They don't happen in a vacuum; it's incredibly important to have them both figured out.

We're the only team that releases products on 4 different platforms: mobile web, desktop web, android, and ios

Crucial to make sure we don't build products 4 times. Focusing on shared services.

Slide 5

We met all of our goals.

Slide 6

Josh: We did 4 releases this Q; 2 major and 2 bugfix.

Feature launch: content notification -- notifying users of content that's potentially interesting to them

we did analysis and prototyping and came to a decision to use a combo of pageview metrics and editor curation

The way it'll work: editors who selected in the news items for main page: If any are in the top 10, the user will be notified.

In beta, but not on yet.

Push notifications for content are a new concept for Wikipedia, so we need to see what people think. Need to get interactions correct.

Data layer migration.

IOS was just saving to disk as files; we've been moving it to use an actual DB backend. Explore feed is loading from the DB now. using API endpoint that Android team developed, big props to them

We also released lockscreen widgets (not a public goal so not listed on the slide).

One shows Top 3 articles in your primary language. Another is a bookmark for an article you were reading before.

Feature was planned to be released with the iOS10 version. Back to school featured in South Korea.

Featured in 'Do More With Your Lock Screen' as well

Resulted in increased installs (155K) over baseline

Slide 7

Impact of features is shown on the slide. Had issues with market awareness of the iOS app. People were not aware of the Wikipedia app. Being featured by Apple is important for awareness.

We went from 100-150k unique impressions per day to millions of unique impressions as a result of being featured. 7000% increase over baseline.

We weren't prominently featured in US but we were heavily featured in the Asia Pacific region.

When featured worldwide, we got a lot of impressions, especially from Latin America and Asia.

Katherine: Specific reason we know of that we're not being featured in US? Is it because we're internationalized in a way?

Josh: Combination of things. Apple appears to regard us as an international brand.

Also, a lot of our commercial competitors only support English; so we have a lot of markets to ourselves in a way we don't in the English app store.

US/Canada users respond differently to users in other areas to being prompted to download. US/Can (dark blue line) conversion rate 15%; Most other global regions were aroud 3-4%. Significantly lower than the US.

We will be working with comms to improve our reach in intl markets. We don't localize our messages.

Toby: So big picture, we started this arc at the outset of reading team via the Finland app banner test. We need to start thinking about how to increase distribution through the varoius channels available.

Josh: There is a lot of branding value in this. Millions of users were seeing the WP logo on their phones, which is free advertising.

Wes (on Blue Jeans chat): Good work iOS - been loving the improvements, checking my app every morning as a habit

Slide 9

Dmitry: Feed was almost finished in Q4; released in first weeks of q1. Explore feed is now available on both iOS and Android apps: trending articles, things on the news, suggestions of things based on the user's reading history.

The response to the feed ahs been very positive from users and the media. Blog post was the most read post of the year. Definitely more coverage than any previous release of the Android app. But, this coverage didn't seem to do a lot for our install numbers.

It's frustratingly difficult to move the needle on installs for the Android app. iOS press releases seem to generate bumps in install numbers, unlike Android ones.

More and more users are engaging with the feed, particularly the Trending portion.

Q1 goal was to overhaul the navigation so as to make features easier for the user to discover. Move away from the hamburger navigation in favor of bottom navigation tab. We did manage to get this done in the Q (released on the last day of the quarter.) We'll need more time to collect data and analyze impact.

Slide 10

Dmitry (cont.): The feed has been very successful as a new way of engaging with our users and presenting content to them. We're still scratching the surface. I feel like we should iterate more and do A/B tests. Trending articles are esp. promising. By far the most engaged-with piece of the feed. Users are very interested in that kind of content. We're going to be doing more thinking of ways to present the most relevant trending results.

Katherine: Community response so far?

Dmitry: No community response.

Toby: The community has responded well to the pageview API the analytics team released 3 quarters ago. It's a good thing. As Dmitry said we haven't had a lot of response from the Community. They are more focused on the Desktop experience.

JK: But Android did put out a page to discuss it.

Katherine: OK, just asked because I had heard various views about automatic vs. editorial curation, so I was wondering...

Dmitry: Total installs are slowly starting to climb again.

Google incident of March 2016 where google was forcing users to install our app. Not great but did result in 1m new users... but they fell back off quickly.

We were featured in the back to school promotion on Google store.

The content service is starting to gain traction in other teams [iOS, Reading Web --ed.]. Content service is something Android team built internally. We're seeing patches from other teams, not from iOS only, but from Reading Web as well. Happy other teams are adopting it.

toby: This speaks to how we work, particularly from efficiency/effectiveness standpoint: iOS team first prototyped, now Android team put it into a service that everyone can use and put our learnings into practice.

Important for efficiency/effectiveness; we're not Google, we don't have thousands of people to build out things like this.

Dmitry: that's all for Android.

Wes (on Blue Jeans chat): Android services and discovery have been great - been using my android as a habit daily too. Nice iterations and like preview links and nearby as much as trending.

Slide 11

Anne: I've been leading the New Readers project. It's a partnership involving five teams: reading, comms, partnerships, community engagement, design research

did 3 ethnographic surveys in 15 languages

Work this Q was about processing and synthesizing. The goal was to make it available org-wide.

Did a presentation on 9/28 w/ all teams involved, all info is on Meta (https://meta.wikimedia.org/wiki/New_Readers/Findings )

Toby: this research is in and of itself really important. a win in itself (not only with a product release resulting, say).

Slide 12

Anne: We (Reading/I) drove a decisionmaking process to get these results into 24 findings. 11K phone surveys

Places we decided to focus:

   -lack of awareness/understanding of WP 
   -supporting readers who want to curate offline content (maybe there are issues with cost, power availability) -- maybe share on P2P networks
   - affordability. Zero has been around for this reason.

We've started working on prototyping this quarter. We did that in Berlin. We'll be iterating this quarter. with communities. Need to make sure people who curate and understand our porjects best are on board.

Katherine: Who was involved(?) in the prototyping session(s)? Anne: Olga, Anne, Joaquin, Piotr participated.

Anne: We've found it's hard to reach the right people in the kind of research we're doing.

Slide 13

Anne (cont.): We've been working hard on buildilng relationships with community members, esp. in target countries. We've done casual interactions and relationship building. Got people to come to research presentations and continue those conversations. Also having discussions on Meta. Backlash has been non-existent.

Another success: Collaboration across teams has been really good.

But multiple-team ownership is also challenging. Talking with Amanda B. about how to structure it and how other nonprofits handle things like this. Formal community consulation has been hard to kick off.

Katherine: This work (and the collaboration involved) is a great model for the rest of the organization: cross-team collaboration. Collaboration with community is the right direction.

Slide 14

Olga: I'm the new PM on the Reading Web team. I started in the middle of August. Main goal: Language switching

A lot of our current readers speak multiple languages and so do a lot of potential new readers. We wanted to work with the language engineering team to better support these users. Moved the language button from the bottom of the page to the top of the page. We made sure it works well with other solutions related to language switching that other teams worked on.

We did research with the community to make sure we're doing it right and actually improving usability.

We were expecting an increase in language switching, which didn't happen. We didn't see a decrease either. Suggests that users that already knew about the language switching button are still using it, but users who didn't still don't -- aren't finding it more readily.

We also worked on two goals from the previous quarter.

Katherine (on BlueJeans chat): Another UX question - is our languge button an industry standard on language switching? as a sadly monolingual human, I woudn't know... I'm curious how recognizeable it - or is it another hamburger of icons? Sherah: this is an icon that is similar to Google Translate Katherine: gotcha, thanks Sherah - so we know it's something used elsewhere, do we have any data around user awareness of what it means? Sherah: yeah, I did some user research around the icon being understood as translation - I think that users understand the use of the icon as meaning "translate this article" rather than getting the model of "switching Wikipedias" but ultimately the goal of being able to read more easily in a preferred language is still achieved Katherine: Ah, thanks @Sherah - interesting

Slide 15

(see picture)

Before: it was necessary to scroll through the entire article to switch languages. Now the button is at the top. It's easy to switch.

Michelle: How does one preset languages, and how are we telling people about it? Olga: We did some testing to make sure people knew where the new icon was. We also kept the old button, which let the users know of the position of the new button. We kept the old button a few months after releasing and removed it last week.

Michelle: Can you set any # of languages to be on the top? JK: Yes, we worked on enhanced language switcher a few? quarters ago.

Michelle: So it's automatically done rather than user-set? Josh: Similar on iOS -- it's auto-detected and the user can change it. JK: This kind of language switching is available on both apps and the mobile web.

Nirzar: We did research showing that language switching at the top was a good user experience.

Slide 16

Olga: Our another goal was Hovercards from Q4.

Hovercards give a preview of the content avaialble via a link while you're hovering over it. Made sure this Q that the experience is thoroughly tested and as smooth as possible, well tested, has good code, doesn't affect fundraising.

Currently, the workflow to get a sense of a page is click to new page, potentially lose your place, have to go back, etc.

This has been a much requested feature. There is also a gadget that does this kind of work. This is kind of similar but has a cleaner design.

Slide 17

Olga: A/B test on huwiki.

Goals: figure out (1) Does this work and is it good? (2) Does it affect fundraising? 50% of huwiki users would see Hovercards, and the other 50% would not.

Hypothesis: It WOULD affect fundraising because users would click fewer links and therefore see fewer banners. It was true. Fewer fundrasing banners were seen and clicked on. But overall didn't have negative effect on fundraising. Banners performed better (earned more $$ per click).

We worked with Design Research to do qualitative reasearch on how the hovercards are affecting their reading behavior and experience. Due to tech problems I won't be able to show videos I wanted to.

We gained insights on how people browse differently with them enabled, as well as what we should call it.

We may change the name "Hovercards" to "Previews" or "Page Previews".

So we'll finish up these a/b tests, gain more quant data, and work closely with the communities to see who wants this feature and how we can help.

Katherine: How, given that it may have an impact on pageviews (which we know isn't the best metric ever), do we have a sense of how we're capturing views on hovercards?

Olga: What we're measuring is the way the users interact with links.

We have a hypothesis for investigation that readers may click fewer links but read further into the page. With the new A/B test: the idea of page actions. Clicking on a link to view another page or hovering over a link to read more about the page.

JK: Pageviews are becoming less and less meaningful, particularly on apps, as we add more ways to engage with content. We're looking at other metrics (such as time on site).

Slide 18

Olga: The next thing we did is lots of work on affordability. Make pages use less data to load. Particularly by lazy loading of images. See Jon Robson's great blog post.

Slide 19

Olga: A big part of getting new readers is affordaiblity. Data is really expensive in many places so using less is very important.

Results: 32% less data in Indonesia, 47% decrease in USA, 51% decrease in Japanese wikipedia.

Toby: To put into industry context, Facebook (instant pages) Google (AMP) and others have been trying to enhance mobile publishing in this way, which involved publishers having to hand over control to them. We did it ourselves. Lots of difficult tech and design work. We dont need to rely on other companies for this.

(on Blue Jeans chat:) Guest 2: we need the Asian cache POP badly Katherine: We have a set of projects we'd like to prioritize using underspent funds, Asian POP is high on the list Guest 2: I think Faidon and Ori can show some ugly graphs from a recent SFO POP outage

JK: in the interest of time let's move on to Wikidata Descriptions.

Slide 20

Olga: Wikidata descriptions; this is an example of a cross-platform feature that we're now moving to mobile web. The feature was available on both iOS and Android.

We talked to the communities while rolling it out. We deployed to all wikis except the top 6. Then people liked it so we began deploying to Japanese and Spanish wiki.

Related pages: another feature with very high click-thru rate. We've been doing lots of bug fixes and deployed to all but the top 6 in the beginning of Q1.

[discussion about wikidata descriptions and how they are handled currently on the mobile platforms]

JK: Android will pilot Wikidata description editing in-app in Q2.

Slide 21

Nirzar: our goal was to work with the Editing team on Design principles.

We developed a color typography, hierarchy, a design language. Now we have coherence cross-platform.

Slide 22

Now we have an identifiable and consistent brand. We've figured out how to work within the design group (in terms of process). Katherine: this makes my heart beat with joy :) appreciate the effort to organize team(s?), work with individuals

Community Tech

Slide 23

Danny: We're wrapping up work resulting from the wishlist survey, heading into 2016 survey.

Quick summary: We're responsible for addressing the top 10 wishes of 2015. Of those 10, 5 were done. 2 wishes are being worked on by other teams. We had to decline 2 wishes. 1 wish is ongoing.

Slide 24

We've also been looking at other ways to do helpful work; foundation departments had tools they wanted to use, e.g., to support admins and checkusers...

In the 2015 survey the WikiSource folks came out in force. We'll do some things with them.

We also did some spot fixes: abuse filter and new page patrol.

Slide 25

Katherine: So top 10 will always be consensus and then you'll set aside time for smaller groups? Danny: Yes

One thing that we're changing about our workflow?: What we want people to do this year is to have not just one person submit proposal and be responsible for it, but to try to discuss and improve them as a community throughout the process. Not about winning or losing.

We found this year that wishlist items that were too vague or too big, ... Also for volunteer developers who looking at (lower parts of) the list for things to work on, it will be useful to have specificity, Developer Relations team looking into this too.

Top 10 will always be our "brand", but also want to allocate some time to categories that aren't on it.

Our mascot is a dog with a santa hat.

Slide 26

Danny: Q1 goals: launch copy patrol, numerical sorting in categories, and cross-wiki watchlist.

Slide 27

Kaldari: Copy Patrol: The community is very concerned with content quality and verifiability. Don't want people to just copypaste stuff from the 'net. It turns out this is hard for humans but this can be detected better through automated means. Interface was jerky. Only one person was using it. No way to search or filter.

So we developed a completely new interface on tool labs that people could use.

We launched this earlier this quarter. Backlog for plagiarism used to be a month or more, Current backlog is no more than a day or two old. 600% increase in user count (now 6). Katherine: That's amazing!

Slide 28

Kaldari: Numerical sorting for categories. This may seem esoteric but Wikipedians care about this deeply. It's analogous to library cataloguing. It used to use alphabetic sorting for everything, even for numbers. Changing this was seemingly simple but wasn't in practice: we had to do DB changes - it was pretty complicated. But now it's deployed on enwiki. Everyone is pretty happy about it.

Slide 29

KaldarI: Next one related to quarterly goals: Cross-wiki watchlists. Anything cross-wiki is difficult to implement technically. Main goal for this quarter was to develop a technical implementation for the backend. We worked with the architecture committee. Talked about various possible implementations. Going to improve CentralAuth, to make it easier to query user data across projects (very challenging right now). Collaboration team told us that when they implemented cross-wiki notifications, they wish they had taken a more holistic approach instead sort of hacky one.

Goal for Q2 is to develop a proof of concept prototype.

We had an outside designer contractor work on this. Designers tend to like to redesign everything from scratch. The community did not like the redesign. So Danny came up with an MVP design that minimized apparent changes. The community liked it way better.

Toby: It shows design is valued by our communities. Not just how it looks, but how it works.

Slide 30

Bryan: I've been working on tool labs support for the last quarter. Project Striker (Tool Labs Admin Console) live at https://toolsadmin.wikimedia.org

Allows contributors to create new repos hosted in Phabricator. We wanted to make it easy for people to host their source code freely (in all meanings).

Before, the user experience for account creation was really bad. I'm working on holding people's hands a little more w/r/t what info is needed, where it will be used. Almost done - in code reivew right now. The initial security review of Striker dragged a little longer than I expected (3 weeks behind).

The other goal this quarter was to kick off a discussion about what to do about abandoned tools. How to allow people interested in abandoned tools to keep working on them. 3 weeks of open discussion, closed about a week ago. Working on amending the proposals to incorporate the community, then will enter the RFC phase.

Toby: This may seem esoteric but actually these tools are critical to the health of the wikis.

Katherine: I love the fact that you identified this as a need. Already heard some great feedback on the fact that those conversations are happening. Thanks all.

Slide 31

Kaldari: One of the other projects we worked on in Q1, PageAssessments, predates the wishlist survey. No longer have to scrape article talk pages / do expensive category link queries. Deployed this extension on enwiki. We're rolling this out to other wikis. Another project we worked on was to build and deploy the Google OCR tool for Indic languages. Most wikis already have their OCR tools, but support for indic language is not good.

5 different Indic languages are actively using this. 5 more are going to be using soon.

This project was completely implemented by our brand new hire, Sam.

Katherine: Love this. When I was in India, the community was talking about improved support for WikiSource, its importance there. This is exciting, thank you.

Danny: It's important for us to do not just big projects for big languages but also more global(?) work. Some like Copy Patrol on enwiki first. Cross wiki watchlist is going to be everywhere.

Slide 34

JK: YOY traffic this Q was flat, though we'd seen a 10% drop in previous quarters Katherine: Flat or stable? JK: Stable maybe, though that implies knowing the future, which we don't. Katherine: Is this just pageviews or uniques? JK: Pageviews. Katherine: Could be more mobile pageviews making up for fewer desktop, etc... JK: it'll take another 6 months until we have year over year comparison for uniques.

JK: Mobile page views has been approaching 50%, but that has stopped recently.

Slide 35

Part of the way we develop which is unique because we're on 4 different platforms is that we develop in a pre-development stage on one platform Amount of work on web is much higher Toby: bar is higher on web, majority of users are there apps high quality requirements too of course, but..

Slide 36

Part of the way we develop which is unique because we're on 4 different platforms is that we develop in a pre-development stage on one platform, do research, see how it does

Once we know it's worth implementing on a different platform, build it into a service that all can use

This has worked very well for the apps

Slide 37

Web is still the challenge here. The amount of work involved is much higher. You have to do more research, consultation, ?.

Toby: The bar is higher on the Web just becase most of our users are there. iOS and Android also have high bars, but with the web things just take longer. Community acceptance is harder.

Slide 38

JK: Issues from last quarter we're making progress on: QA, defining and implementing core metrics, and retention.

Slide 39

...and here are some open issues.

Toby: Want to call out the collaboration. Other collaborations have been happening behind the scenes (PDF rendering, language bar).

Nirzar has been designing for at least 4 verticals and Google (lols from audience).

Thanks everyone, if you have questions direct them towards Toby or JK.

Slide 40

Stoppped here.