Research:Article feedback/Clicktracking

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Article Feedback v5 Data & MetricsStage 1: Design
(December 2011 - March 2012)
Stage 2: Placement
(March 2012 - April 2012)
Stage 3: Impact on engagement
(April 2012 - May 2012)

WP:AFT5 (Talk)
Feature requirements

Dashboards

Overview
Article samples
Feature data
Clicktracking data

Volume analysis
Quality assessment
Reader survey / Team survey
Usability testing

Volume analysis
Quality assessment

Conversions and newcomer quality

Final tests

Quality assessment
Research report (2012 Q4)
Moderation tools usability study

This is the list of events tracked as of the launch of AFT5 (reference: bugzilla:32992)

Event naming scheme[edit]

The general naming scheme used in AFT5 is the following:

'ext.articleFeedbackv5@' + version + '-' + key

where version is the version number as set in the configuration file and key is the complete event string:

  • [option id]-[event]-[placement id]
  • [option id]-[trigger id]-click-overlay
  • [option id]-[cta id]-[event]-[placement id]
  • [option id]-[edit link location]-[event]

Identifiers[edit]

Option IDs[edit]

option1
Did you find what you were looking for
AFT5-Feedback-Form-Option-1-Launch-Screenshot.png
option2
Make a suggestion
AFT5-Feedback-Form-Option-2-Launch-Screenshot-12-19-2001.png
option3
Rate this article
AFT5-Feedback-Form-Option-3-Launch-Screenshot-12-19-2001.png

Placement IDs[edit]

bottom
Default placement at the bottom of the page
overlay
Widget display in an overlay triggered by a click on the feedback button

Trigger IDs[edit]

(see a screenshot of feedback links)

triggerA
Link after the site subhead ("from Wikipedia..."), or below the titlebar and left-justified if there is no subhead
triggerB
Link below the titlebar, right-justified (and below the geocoordinates, if present)
triggerC
Fixed-position vertical tab on the right side of the window, about 200px from the top
triggerD
Fixed-position tab on the bottom right corner of the window
triggerE
Fixed-position tab on the bottom of the window, at the center
triggerF
Fixed-position tab on the left side of the window, at the center
triggerG
Rounded button in the left sidebar, just below the logo
triggerH
Links at each section bar, just after [edit]
triggerTBX
Link in the Toolbox section of the left sidebar

CTA IDs[edit]

cta_edit
Edit this page
AFT-v5-CallToAction-Edit-Mockup-11-29.png
cta_learn_more
Learn more
AFTv5-CTA-Thanks-NoEdit-12-11.png
cta_survey
Take a survey
AFT4-CTA-Survey-Screenshot.png

Edit Link Locations[edit]

edit_tab_link
The edit tab above the page
section_edit_link
One of the section edit links

Full list of events[edit]

Unless otherwise specified all events are tracked at 100%.

Stage 1 (Design)[edit]

option1-init (1%)
option2-init (1%)
option3-init (1%)
option1-impression-overlay
option2-impression-overlay
option3-impression-overlay
option1-impression-bottom
option2-impression-bottom
option3-impression-bottom
option1-submit-overlay
option2-submit-overlay
option3-submit-overlay
option1-submit-bottom
option2-submit-bottom
option3-submit-bottom
option1-cta_edit-impression-overlay
option2-cta_edit-impression-overlay
option3-cta_edit-impression-overlay
option1-cta_learn_more-impression-overlay
option2-cta_learn_more-impression-overlay
option3-cta_learn_more-impression-overlay
option1-cta_survey-impression-overlay
option2-cta_survey-impression-overlay
option3-cta_survey-impression-overlay
option1-cta_edit-impression-bottom
option2-cta_edit-impression-bottom
option3-cta_edit-impression-bottom
option1-cta_learn_more-impression-bottom
option2-cta_learn_more-impression-bottom
option3-cta_learn_more-impression-bottom
option1-cta_survey-impression-bottom
option2-cta_survey-impression-bottom
option3-cta_survey-impression-bottom
option1-close-overlay
option2-close-overlay
option3-close-overlay
option1-cta_edit-close-overlay
option2-cta_edit-close-overlay
option3-cta_edit-close-overlay
option1-cta_learn_more-close-overlay
option2-cta_learn_more-close-overlay
option3-cta_learn_more-close-overlay
option1-cta_learn_more-button_click-overlay
option2-cta_learn_more-button_click-overlay
option3-cta_learn_more-button_click-overlay
option1-cta_learn_more-button_click-bottom
option2-cta_learn_more-button_click-bottom
option3-cta_learn_more-button_click-bottom
option1-cta_edit-button_click-overlay
option2-cta_edit-button_click-overlay
option3-cta_edit-button_click-overlay
option1-cta_edit-button_click-bottom
option2-cta_edit-button_click-bottom
option3-cta_edit-button_click-bottom
option1-cta_survey-button_click-overlay
option2-cta_survey-button_click-overlay
option3-cta_survey-button_click-overlay
option1-cta_survey-button_click-bottom
option2-cta_survey-button_click-bottom
option3-cta_survey-button_click-bottom
option1-cta_edit-edit_attempt-overlay
option2-cta_edit-edit_attempt-overlay
option3-cta_edit-edit_attempt-overlay
option1-cta_edit-edit_attempt-bottom
option2-cta_edit-edit_attempt-bottom
option3-cta_edit-edit_attempt-bottom
option1-cta_edit-edit_success-overlay
option2-cta_edit-edit_success-overlay
option3-cta_edit-edit_success-overlay
option1-cta_edit-edit_success-bottom
option2-cta_edit-edit_success-bottom
option3-cta_edit-edit_success-bottom
option1-triggerA-click-overlay
option2-triggerA-click-overlay
option3-triggerA-click-overlay
option1-triggerB-click-overlay
option2-triggerB-click-overlay
option3-triggerB-click-overlay
option1-triggerC-click-overlay
option2-triggerC-click-overlay
option3-triggerC-click-overlay
option1-triggerD-click-overlay
option2-triggerD-click-overlay
option3-triggerD-click-overlay
option1-triggerE-click-overlay
option2-triggerE-click-overlay
option3-triggerE-click-overlay
option1-triggerF-click-overlay
option2-triggerF-click-overlay
option3-triggerF-click-overlay
option1-triggerG-click-overlay
option2-triggerG-click-overlay
option3-triggerG-click-overlay
option1-triggerH-click-overlay
option2-triggerH-click-overlay
option3-triggerH-click-overlay
option1-triggerTBX-click-overlay
option2-triggerTBX-click-overlay
option3-triggerTBX-click-overlay
option1-edit_tab_link-click
option2-edit_tab_link-click
option3-edit_tab_link-click
option1-section_edit_link-click
option2-section_edit_link-click
option3-section_edit_link-click
option1-edit_tab_link-edit_attempt
option2-edit_tab_link-edit_attempt
option3-edit_tab_link-edit_attempt
option1-section_edit_link-edit_attempt
option2-section_edit_link-edit_attempt
option3-section_edit_link-edit_attempt
option1-edit_tab_link-edit_success
option2-edit_tab_link-edit_success
option3-edit_tab_link-edit_success
option1-section_edit_link-edit_success
option2-section_edit_link-edit_success
option3-section_edit_link-edit_success

Stage 2 (placement)[edit]

option1A-init (1%)
option1E-init (1%)
option1X-init (1%)

option1A-impression-overlay
option1E-impression-overlay

option1A-impression-bottom
option1E-impression-bottom
option1X-impression-bottom

option1A-submit-overlay
option1E-submit-overlay

option1A-submit-bottom
option1E-submit-bottom
option1X-submit-bottom

option1A-disable_button_click
option1E-disable_button_click

option1A-disable_flyover-impression
option1E-disable_flyover-impression

option1A-disable_gotoprefs_click
option1E-disable_gotoprefs_click

option1A-cta_edit-impression-overlay
option1E-cta_edit-impression-overlay

option1A-cta_learn_more-impression-overlay
option1E-cta_learn_more-impression-overlay

option1A-cta_edit-impression-bottom
option1E-cta_edit-impression-bottom
option1X-cta_edit-impression-bottom

option1A-cta_learn_more-impression-bottom
option1E-cta_learn_more-impression-bottom
option1X-cta_learn_more-impression-bottom

option1A-close-overlay
option1E-close-overlay

option1A-cta_edit-close-overlay
option1E-cta_edit-close-overlay

option1A-cta_learn_more-close-overlay
option1E-cta_learn_more-close-overlay

option1A-cta_learn_more-button_click-overlay
option1E-cta_learn_more-button_click-overlay

option1A-cta_learn_more-button_click-bottom
option1E-cta_learn_more-button_click-bottom
option1X-cta_learn_more-button_click-bottom

option1A-cta_edit-button_click-overlay
option1E-cta_edit-button_click-overlay

option1A-cta_edit-button_click-bottom
option1E-cta_edit-button_click-bottom
option1X-cta_edit-button_click-bottom

option1A-cta_edit-edit_attempt-overlay
option1E-cta_edit-edit_attempt-overlay

option1A-cta_edit-edit_attempt-bottom
option1E-cta_edit-edit_attempt-bottom
option1X-cta_edit-edit_attempt-bottom

option1A-cta_edit-edit_success-overlay
option1E-cta_edit-edit_success-overlay

option1A-cta_edit-edit_success-bottom
option1E-cta_edit-edit_success-bottom
option1X-cta_edit-edit_success-bottom

option1A-triggerA-click-overlay
option1E-triggerE-click-overlay

option1A-edit_tab_link-click
option1E-edit_tab_link-click
option1X-edit_tab_link-click

option1A-section_edit_link-click
option1E-section_edit_link-click
option1X-section_edit_link-click

option1A-edit_tab_link-edit_attempt
option1E-edit_tab_link-edit_attempt
option1X-edit_tab_link-edit_attempt

option1A-section_edit_link-edit_attempt
option1E-section_edit_link-edit_attempt
option1X-section_edit_link-edit_attempt

option1A-edit_tab_link-edit_success
option1E-edit_tab_link-edit_success
option1X-edit_tab_link-edit_success

option1A-section_edit_link-edit_success
option1E-section_edit_link-edit_success
option1X-section_edit_link-edit_success

Stage 3 (Impact on engagement)[edit]

Assumptions

  • Option4E will need to have a fallback similar to the "Learn more" CTA that we use instead of the Edit CTA on non-editable/protected pages
  • Option4E init events will be unaffected by protection status (we will capture combined Option4E and Option 4E fallback inits)
  • We won't have the actual number of Option4 inits, only a sample
  • We will capture Option4E vs Option4E fallback impressions separately
  • We will entirely hide the feedback link for the Option4E fallback, but still display the fallback form at the bottom of the page
  • We will allow registered users to disable Option4E via the user prefs even if it's not displaying a feedback form
  • On non-editable pages we will count clicks on the "View source" tab link but we won't get any section edit link events

Note Option4 events are represented as follows:

optionSE_4{E|X}
all Option4 events combined
optionSE_4E_edit
only Option4 events for editable pages
optionSE_4X_learn_more
only Option4 events for non-editable pages (fallback)
optionSE_1E-init (1%)
optionSE_4E-init (1%)   #Option4E_edit only; optionSE_4X_learn_more is always recorded as noedit
optionSE_0X-init (1%)

optionSE_1E-noedit-init (1%) #Counts inits for users in the bucket with no edit access to the page
optionSE_4X-noedit-init (1%) #Counts inits for users in the bucket with no edit access to the page
optionSE_0X-noedit-init (1%) #Counts inits for users in the bucket with no edit access to the page

optionSE_1E-impression-overlay
optionSE_4E_edit-impression-overlay #Assumes the fallback is never displayed as an overlay

optionSE_1E-impression-bottom
optionSE_4E_edit-impression-bottom
optionSE_4X_learn_more-impression-bottom   #Assumes the fallback is never displayed as an overlay

optionSE_1E-submit-overlay
optionSE_1E-submit-bottom

optionSE_4E_edit-button_click-overlay
optionSE_4E_edit-button_click-bottom
optionSE_4X_learn_more-button_click-bottom #Assumes the fallback is never displayed as an overlay

optionSE_4E_edit-edit_attempt-overlay #Edit-related events for direct CTA
optionSE_4E_edit-edit_attempt-bottom
optionSE_4E_edit-edit_success-overlay
optionSE_4E_edit-edit_success-bottom

optionSE_1E-disable_button_click
optionSE_4E_edit-disable_button_click #Assumes registered users will disable a direct call to edit via prefs referring to AFT
optionSE_1E-disable_flyover-impression
optionSE_4E_edit-disable_flyover-impression #Assumes registered users will disable a direct call to edit via prefs referring to AFT
optionSE_1E-disable_gotoprefs_click
optionSE_4E_edit-disable_gotoprefs_click #Assumes registered users will disable a direct call to edit via prefs referring to AFT

optionSE_1E-cta_edit-impression-overlay
optionSE_1E-cta_learn_more-impression-overlay

optionSE_1E-cta_edit-impression-bottom
optionSE_1E-cta_learn_more-impression-bottom

optionSE_1E-close-overlay
optionSE_4E_edit-close-overlay #Assumes the fallback is never displayed as an overlay

optionSE_1E-cta_edit-close-overlay
optionSE_1E-cta_learn_more-close-overlay

optionSE_1E-cta_learn_more-button_click-overlay
optionSE_1E-cta_learn_more-button_click-bottom

optionSE_1E-cta_edit-button_click-overlay
optionSE_1E-cta_edit-button_click-bottom
optionSE_1E-cta_edit-edit_attempt-overlay
optionSE_1E-cta_edit-edit_attempt-bottom
optionSE_1E-cta_edit-edit_success-overlay
optionSE_1E-cta_edit-edit_success-bottom

optionSE_1E-triggerE-click-overlay
optionSE_4E_edit-triggerE-click-overlay #Assumes the fallback is never displayed as an overlay

optionSE_1E-edit_tab_link-click
optionSE_4E_edit-edit_tab_link-click
optionSE_0X-edit_tab_link-click

optionSE_1E-view_source_tab_link-click
optionSE_4X_learn_more-view_source_tab_link-click
optionSE_0X-view_source_tab_link-click

optionSE_1E-section_edit_link-click #Assumes section edit links are only displayed if page is editable
optionSE_4E_edit-section_edit_link-click #Assumes section edit links are only displayed if page is editable
optionSE_0X-section_edit_link-click #Assumes section edit links are only displayed if page is editable

optionSE_1E-edit_tab_link-edit_attempt
optionSE_4E_edit-edit_tab_link-edit_attempt
optionSE_0X-edit_tab_link-edit_attempt

optionSE_1E-section_edit_link-edit_attempt
optionSE_4E_edit-section_edit_link-edit_attempt
optionSE_0X-section_edit_link-edit_attempt

optionSE_1E-edit_tab_link-edit_success
optionSE_4E_edit-edit_tab_link-edit_success
optionSE_0x-edit_tab_link-edit_success

optionSE_1E-section_edit_link-edit_success
optionSE_4E_edit-section_edit_link-edit_success
optionSE_0X-section_edit_link-edit_success


CTA1 update (October 2012)[edit]

Specs based on stage 3 events above, assuming the default form id (6) and placement id (X).

option6X-init (1%)
option6X-noedit-init (1%) #Counts inits for users in the bucket with no edit access to the page

option6X-impression
option6X-submit

option6X-cta_edit-impression
option6X-cta_learn_more-impression

option6X-cta_edit-button_click
option6X-cta_learn_more-button_click

option6X-cta_edit-edit_attempt
option6X-cta_edit-edit_success

option6X-edit_tab_link-click
option6X-section_edit_link-click
option6X-view_source_tab_link-click

option6X-edit_tab_link-edit_attempt
option6X-section_edit_link-edit_attempt

option6X-edit_tab_link-edit_success
option6X-section_edit_link-edit_success

Stage 4 (FeedbackPage usage)[edit]

Notes[edit]

  • All events in Stage 4 are tracked at 100%
  • The main metrics we are focusing on in Stage 4 are:
    • the CTR and breakdown of traffic to the FeedbackPage by referral and user category (anons vs registered vs monitors)
    • the breakdown of filter events from the Feedback Page by referral and user category (as above)
  • We won't be bucketing users in Stage 4 to study FeedbackPage
  • We won't apply clicktracking to the toolbox or the permalink view of a single post at this stage

Event types[edit]

(The prefix will be ext.articleFeedbackv5 as usual)

1. Referral events

The general naming scheme for these events should be:

[event-type]-[user_privs]

where event-type is one of the following:

cta_view_feedback-impression            <----- cta5 is displayed
cta_view_feedback-button_click   	<----- sends user to feedback page
cta_view_feedback-link_click   	        <----- sends user to individual post

talk_page_view_feedback-impression      <----- talk page button is displayed 
talk_page_view_feedback-button_click    <----- sends user to feedback page

and user_privs is one of the following:

anon		<----- anonymous
reg		<----- regular registered user
mon		<----- monitor (has the aftv5-see-hidden-feedback privilege)

2. Landing events

The general naming scheme for these events should be:

feedback_page-impression-[referral]-[user_privs]

where referral is one of the following:

ref_url		<----- from url
ref_cta		<----- from cta
ref_talk	<----- from talk page

3. Filter events

The general naming scheme for these events should be:

feedback_page-click-[filter]-[referral]-[user_privs]

where filter is one of the following:

f_most_relevant
f_all_comments
f_featured
f_helpful
f_all_visible
f_more 		<----- click on the show more button at the end of the page

NB: The additional data for the feedback page will be the page id of the article whose feedback is being shown, or null if it's the central page.

Additional data[edit]

The log includes as additional data (as the last field of each log record) a pipe-separated value: page_title|rev_id.

  • page_title: title of the article
  • rev_id: revision id of the article. Note that this is the id of the successfully created revision for *-edit_success events

Log format[edit]

Clicktracking data is logged using the following, tab-separated format:

enwiki ext.articleFeedbackv5@0-option1-impression-bottom 20120113013257 0 w4dtyfWSw714KsgmMAAkdC7isy8CmHvP9 0 0 0 0 Pokémon|274278

Log format specification[edit]

  1. database name (e.g. enwiki or ptwikisource)
  2. event name (as documented above)
  3. timestamp (in YYYYMMDDhhmmss format)
  4. user category: 1 for logged in users, NULL for anons
  5. user token: an anonymous token replacing user names or IP addresses
  6. namespace
  7. lifetime edit count (logged in only)
  8. 6-month edit count (logged in only)
  9. 3-month edit count (logged in only)
  10. last month edit count (logged in only)
  11. additional data (page title, rev_id)

Data collection[edit]

Events and deployments that affected the collection of clicktracking data are documented here


Article Feedback Version 5 Specs: HubFeature RequirementsTechnical DesignTestingExtension Page
Metrics: Data and Metrics PlanClicktrackingVolume AnalysisQuality Assessment
Tools: AFTv5 CategoryWikipedia Data StreamOpen BugsRecent Code ChangesPDF Prototype
Community: MW talk pageIdeas LogWikipedia talk page