Wikimedia Foundation metrics and activities meetings/Quarterly reviews/Analytics/September 2014
Present (in the office): Erik Möller, Anasuya Sengupta, Rachel diCerbo, Ellery Wulcyzn, Leila Zia, Toby Negrin, Dario Taraborelli, Lila Tretikov, Kevin Leduc, Tilman Bayer (taking minutes), Carolynne Schloeder, Howie Fung, Jeff Gage, Abbey Ripstra, Daisy Chen; participating remotely: Aaron Halfaker, Jonathan Morgan, Oliver Keyes, Dan Andreescu, Andrew Otto, Christian Aistleitner, Nuria Ruiz, Arthur Richards, Erik Zachte
Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material
2. Performance against goals
3. Research and Data (includes Q&A)
4. Development (includes Q&A)
This has been a very collaboratively written slide deck
standardize baseline metrics
focus on editors this q, then readers
hit most of those goals
on dev side, focused on dashboard and vital signs
Research and Data
Team started in Q1 on new editor metrics
goal: ship definitions (human readable and machine readable version)
involves identifying bots,....
alignment with editor model (for Growth and Mobile teams)
began work with Multimedia team, but on hold
progress towards traffic definitions (pageviews)
both project and article level metrics (community cares a lot about the latter)
Oliver: including mobile views...
unique client definition (on hold)
Lila: do we have a path to get unblocked?
Dario, Toby: yes
Growth research - led by Aaron
anon acq, task suggestions
did extensive analysis of mobile app traffic
mobile traffic and participation trends, effect of tablets switchover
Lila: in this chart, editor activated on mobile means what?
Dario: registered on mobile device[?], and makes 5 edits within 24h
Prediction models for survival periods (worked on this earlier this year, still needs final completion)
Lila: (question about time limitation)
Leila: this is from 2001 onwards
Lila: wonder about 5 edit threshold, what about someone who does a 100 edits in a short burst? What was the rationale to focus on this metric (active editors)?
in the end, what matters is content - whether it is being created and kept up to date
Howie: should have a session examining what this metric is good for (and what not) - there has been a lot of research about this in recent years
Lila: also, e.g. Commons will have different usage patterns
Dario: yes, we mostly do these analyses on a per-project basis
Dario: also, supported a number of other teams with smaller requests, in consultation mode
This slide is actually not comprehensive, need to get better at capturing everything
Lila: this kind of "response train" is normal, it's not bad that requests are coming
but needs to be managed
Toby: sometimes Oliver and Leila were hit by 3 things at once...
Dario: Team process:
joint standups with other teams
Suffer from lack of scrum master, project management support
Lila: where are the inefficienceies - when interacting with other teams, or within team?
Leila: btw, there are also community requests, not just other WMF teams
Lila: for capacity planning, it would be more of a project[product ?] management role which requires deeper understanding, scrum master is more for supporting workflow
Erik: this is the kind of thing I have asked Arthur to support/advise on (team practices)
Lila: and sometimes it's ok to share that role within team
Toby: but I need my people (like Dario) focus on their skills
onboarded Ellery as FR analyst
started collaboration with Morten (external researcher) on task recommendations
Aaron: Morten is processing data on task recommendations right now
collaboration with LANL on anonymizing pageview data - on hold
opened traffic research analyst position - on hold
Outreach: Wikimania, showcases, ...
Lila: coming out of standardization process, what gaps we still have?
Dario: have specifications and raw data, not yet the numbers
strategic research: close to our strategic priorities
APIs, also for external research
This is mostly about data that is public anyway
discontinue "embedded" model
understanding reader behavior - is becoming a priority
detect knowledge gaps
value-based measures - go beyond mere edit count
uplevel the FR experimental strategy
Anasuya: reader behavior - will this differentiate between mature/ small wikis?
Dario: look for definitions that work across the board
but e.g. Wikidata differs a lot
Anasuya: I'm thinking about differences between Wikipedias, too
Lila: need to wrap up Q1 work
goals on this slide make a lot of sense to me
Lila: Core team leading this? yes
Public data sources:
this is making already public data more accessible - e.g. revert metadata: would need lots of manual work now
Lab tools (stretch goal)
move toward consulting model
should allow team to allocate resources more flexibly
Lila: should require advance notice from Engineering / Product teams
avoid resetting priorities
Erik: already have a cross-team process
Toby: both Dan and Andrew have chaired this
Erik: moving from long-term (yearly) goal-setting to quarterly priority setting, have separate meetings for this
Toby: get into a data/metrics driven mindset
Lila: no project should start without having a way to measure impact
Erik: need to be aware on whether it's possible or not, which data expectation is realistic
it's already the case that all the current priorities have measurable goals
Toby: suggest separate conversation on this
some support for IEG
Leila: Aaron does a lot of this (outreach)
Lila: it's OK for experiments, but does it scale?
Anasuya: also, brings external researchers into contact with our community much faster
slide 29-35: appendix (skipped, can take this offline)
main goal was to implement dashboard - accomplished
break down by target site (desktop/mobile/apps) - not reached
blocked on ...
UV: blocked on tech proposal
Vital Signs: Value prop
demo https://metrics.wmflabs.org/static/public/dash/ , e.g. newly registered on Armenian WP, rolling surviving new editors
Lila: I'm going to play with this... can we note the definitions at the bottom?
Dario: they are on Meta, just need to cross-link
Toby: check out Wikidata, compare to ENWP
this may represent a future trend: a lot more smaller edits (some of them by machines)
Vital Signs: backlog
lots of feedback coming in
builds the data that Vital Signs uses
Wikimetrics accomplishments: ...
a lot more robust than cronjobs
Lila: allows to identify fast growing/slow growing projects? yes
new hire will take on backlog for Grantmaking
extend to CentralAuth (users with edits on several wikis)
Lila: so that's a filter by user name (instead of project)? yes
tagging cohorts - classify
Lila: tag reports too?
Kevin: yes, we could do that
Lila: just want to make sure we do this in a general way
Kevin: add ability to remove users from cohort for privacy reasons
Lila: can we also integrate usage (traffic) data?
Toby: Wikimetrics is not primarily about that
friction is between more functionality desired by Grantmaking and...
Anasuya: content metrics are really important for us too, but that's separate
Toby: this is pretty general (using off-the shelf components, e.g. Hadoop),
might be able to quickly plug it into other databases as well, e.g. pageviews
also interesting: Yuvi's Quarry (allows db queries via web interface on Labs)
Oliver (on chat): +1. Quarry is awesome.
Jonathan (on chat): incredibly useful
publicly documented schemas
maintenance, sanitize data
data processing ecosystem, addresses dissatisfaction with data quality
accomplishments: operational since July
a lot of cooperation with Ops team
turned off legacy data feeds (after contacting people) - no complaints so far
port Webstatscollector to Hadoop
significantly reduced error rate (from 3-6%)
Toby: decided to have Christian work very closely with Ops team on this
compile PVs per page and make data public
- community has been asking about this for years
Erik: we have the historical PV data, it has been used for many research projects
Oliver: We can backfill for 1.5 years, using the new definition and the sampled logs, but it'll be a bit inaccurate, because..sampled logs.
Lila: squares on the bottom mean what?
Kevin: blue = addition (e.g. Q4 : myself), red = losses
Process & Metrics
Lila: using point system?
Kevin: yes, Fibonacci
defined 34 points = one developer with little or no distraction during the sprint
7/24 outlier = Dan and Nuria on vacation
metrics show team was more focused in later sprints
Toby: visualization helped team focus
preliminary Q2 goals
Out of scope:
need to put a brake on Vital Signs
Lila: can we break this down, make a list
Toby: did a back-of the envelope velocity calculation, saw we could not do all of them
Lila: just make sure you get the things you commit to done, rest as stretch goals
Limn tried to do too much
Vital Signs as basis to make this available to community
ErikM: using "Vega" open source library
Lila: managing another tool is also work...
Erik: good to do a spike on it, if it's not worthwhile, we will find out quickly
Lila: I have worked myself with Saiku and Tableau[?]
do what you think works, be aware of "human" (work) cost
Toby: should be easy to put contractor on this
(skip "Unique Visitors: Proposed Milestones" slide)
continue with space and support - keep developers focused
recruit / staffing: need to tweak job description to attract right people
promote data informed perspective