Talk:Community health initiative/Interaction Timeline

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Welcome to our discussion!
Light-Bulb icon by Till Teenck green.svg
      When participating in this discussion, please remember the following:
  • Debate ideas, not people.
  • Avoid discussing or linking to specific examples of harassment or user misconduct.
  • To participate privately, contact our team via email. (Your email may be shared internally but never publicly.)

Sockpuppets[edit]

I added a sockpuppet investigation example to the list. This is a significant use of this tool independent of harassment, but many harassment cases also involve abuse of sock accounts.

I understand that entering more than two usernames into the tool generates an N^2 complexity and longer reports, but being able to enter multiple names can help identify which interactions warrant closer examination. It would be nice to allow as many names as feasible, without over taxing the system. Alsee (talk) 08:34, 16 May 2017 (UTC)

Thanks Alsee for your idea. I'll make sure the ideas is brought to the the attention of the team. SPoore (WMF) (talk) 13:58, 16 May 2017 (UTC)
Yes, thank you for adding that new example. I'm curious if we'll be able to design and build a tool that allows for 3+ user names and effectively displays the information in a comprehensible interface. And I totally agree that this tool could be useful for sockpuppet investigations. Also, thank you for pointing out that the numbers in blue on the Edit Interaction Analyzer show who edited first. — Trevor Bolliger, WMF Product Manager 🗨 20:43, 16 May 2017 (UTC)
A viable option would be to show a lot of information for two users, and have reduced information for more than two users. It could either be one tool that drops to the simpler mode for more than two names, or have something separate that accepts more than two names. Alsee (talk) 10:20, 18 May 2017 (UTC)
That's a great idea — no need to complicate the two-user tool just for the use case of a three-user (or multi-user) tool. Another idea that's been suggested is a simple overview page that provides statistics about how 2+ users interact without showing any granular details — e.g. total number of overlap pages, talk page interactions, average time between interactions, etc. Maybe this page should be renamed 'User Interaction History tools. — Trevor Bolliger, WMF Product Manager 🗨 17:18, 18 May 2017 (UTC)
Maybe I can offer a little insight here via a recent case that came up at en:Wikipedia:Administrators' noticeboard/Incidents. I came upon a complaint that an editor was writing racist edit summaries. It was a relatively clear-cut case that didn't require any special tools, and another admin gave the accused editor a final warning. The interesting thing, however, was the complainant speculated that the abusive editor was a likely sock puppet but couldn't come up with a sock master. The topics edited were all related to Hungarian politics.

What I did was look at the abusive editor's most edited pages in the X-Tools edit counter. From there, I scanned the article histories to find editors that were indefinitely blocked for sock puppetry. I quickly noticed that one site banned editor had several CU-confirmed sock puppets active in the same articles as the suspected sock puppet. He also had an LTA case open that described his edits as "xenophobic and racist". This gave me a potential match and the confidence to keep going.

I found a sock puppet of the banned editor that had operated long enough to have a substantial edit history and compared this account to the abusive editor using the Editor Interaction Analyser. I was kind of lost, however, because I couldn't tell what POV, if any, these two editors were pushing. The next part was tediously clicking on diffs until I finally found two that were very, very similar. After setting that aside, I found a few repeated edit summaries that were shared among the suspected sock puppet and several CU-confirmed sock puppets. Based on this evidence, I blocked the suspected sock puppet.

If I had been alerted in the beginning that a CU-confirmed, site banned sock puppeteer was active in the same topics as the suspected sock puppet, that would have made this easier and faster. Finding the LTA case page linked from the site banned editor's SPI archive was also important. The part that sounds most useful to this investigation, however, is the ability to potentially match POV pushing between two accounts who operate in unfamiliar topics. Sorry this is so long, but I figured it might help if I went heavy on details. NinjaRobotPirate (talk) 04:11, 14 June 2017 (UTC)

Thank you, @NinjaRobotPirate: for sharing this example with us! I'm about to get started on some analysis on when EIA is used, what people love about it and what functionality we could add and you've given me a great head start.
Can you share a little more about what was "tedious" during this process? And could you share how you tracked your work so you didn't get lost in all the tabs?
It's great to see that this EIA is useful for sockpuppet identification, but I do want to state that we see the primary use case for the User Interaction History feature to be for user-to-user dispute resolution. Building better sockpuppet identification tools is important to us, and in our Anti-Harassment Tools backlog we have an iticket for T166813 — Robust sockpuppet identification tools. I'd highly recommend following that ticket. I'll also make it a point to contact you when we start that project. — Trevor Bolliger, WMF Product Manager 🗨 00:35, 15 June 2017 (UTC)
A few months ago, there was a case where someone used the EIA to help resolve a harassment complaint: link. I don't know if it's useful, but it's the first thing that came to mind when you mentioned user-to-user dispute resolution.

In my case, using the EIA was tedious mostly because I didn't know what I was looking for, just that I'd know it when it found it (for example, a frequent typo or misspelling). I don't know anything about Hungarian politics, so it was a pattern matching game. As far as tracking work, I'm more methodical than organized. Typically, I keep the EIA master report in one tab, then sequentially go down the timelines one at a time in a second tab. From the tab that contains the current timeline I'm looking at, I'll open several diffs in new tabs. If all this becomes cumbersome, I'll open a new browser window to separate the EIA stuff from my other tabs. Besides that, I may take notes, such as annotated diffs. I sometimes write these in an offline text editor. NinjaRobotPirate (talk) 03:17, 15 June 2017 (UTC)

@NinjaRobotPirate: Ah yes, I've read that case. Those are exactly the kinds of cases where we think our User Interaction History tool (which could be built on ∑'s EIA tool) could help admins or other community leaders most accurately understand the full relationship between two contributors without having to open dozens of tabs. I also think there's room for shared notes/annotations so multiple users could collaborate on a single case.
One final question — how did you learn about the EIA? Did you stumble upon, did someone send it to you? Is there a list/database of useful tools? — Trevor Bolliger, WMF Product Manager 🗨 19:01, 15 June 2017 (UTC)
The EIA is linked from individual cases at en:Wikipedia:Sockpuppet investigations. That's almost certainly where I found it. The only other place I know of where it's mentioned is en:Wikipedia:Tools, but it's unlikely I'd have remembered this if you hadn't asked. NinjaRobotPirate (talk) 19:32, 15 June 2017 (UTC)
Thank you! — Trevor Bolliger, WMF Product Manager 🗨 22:07, 15 June 2017 (UTC)
Regarding shared notes/annotations: I'm not sure what you had in mind, so disregard my comment if I misunderstood. It would almost certainly be a bad idea to add a content system inside the tool, to hold shared notes/annotations. We use wikipages for shared content. You don't want to reinvent that wheel. Any shared content outside of wikipages would need to duplicate a staggering range of non-obvious wikipage functionality. Instead you could give us things like copy-paste links with info embedded in the URL, or results we can copy/paste as wikitext, or even directly dump-to or modifing a wikipage. Alsee (talk) 12:43, 28 June 2017 (UTC)

@Alsee: Great points, and I 100% agree that we shouldn't reinvent the wheel on any project we build. We may focus on building export-to-wikitext tools as you suggested or something to synchronize a specific wiki page with a specific interaction query. We'll still want to be extremely conscious about how publicly visible these notes are, in some cases the notes might be best kept private for the sake of a user's privacy. — Trevor Bolliger, WMF Product Manager 🗨 16:23, 28 June 2017 (UTC)

References[edit]

Feedback summary, Oct 16[edit]

Hello everybody. I’ve re-read all the comments here and on the talk page on English Wikipedia and wanted to provide a summary of all feedback for the Interaction Timeline so far:

  • Mention and Thanks notifications could be shown, as they are publicly visible in the wiki revision history or logs. However, sometimes they may not be helpful so we will want to consider a filter to toggle them on/off.
  • We’re very likely going to need filters for any non-essential data we represent in the timeline. These would most likely be a series of checkboxes to toggle certain elements in the timeline on/off.
  • The two-column layout could be used to monitor more than two users by bundling the others into one of the columns. (E.g. compare ‘Apples’ against ‘Bananas’+’Carrots’)
  • One vertical line (Variation 3) vs two vertical lines (Variation 2) is preferable. No explicit support for Variation 2 yet.
  • Showing the explicit time calculation between users is helpful — it avoids mental arithmetic.
  • In-line diffs would be preferable to new tabs, but beware megadiffs and slow performance.
  • Seeing the edit summary is important, as long as they don’t take up too much space.
  • Some information could be hidden under mouse-over, but consider accessibility needs.
  • The timeline could be to scale to better illustrate when/how people are overlapping in their interactions — maybe a log(interval) scale.
  • We should explore bundling multiple successive edits by one user into a single ‘block’, but will need to make sure there is a way to see the details for each specific edit.
  • Showing cross-wiki activity could be helpful, but it would need a filter to toggle on/off.
  • Showing aggregated statistics above the timeline is helpful
  • It could be helpful to…
    • …annotate the timeline, but there are also some concerns about necessity and complexity.
    • …export results from the tool to wikitext for easy on-wiki reporting.
    • …show edits made by the other users outside of their direct interaction, but would definitely need a filter toggle.
    • …color code or use icons to indicate reverts.
    • …filter certain namespaces out of the timeline.
    • …represent namespaces with different icons.
    • …highlight the different usernames with different colors.
  • The UI should warn when a filter/toggle is enabled that is hiding certain information
  • This is a utility tool — don’t add too much whitespace. Be cognizant of information density.
  • If we built a solution that allow users to specify if they want their emails logged (and visible only to admins) then this data could be shown in the Timeline.

Did I miss anything or misrepresent any feedback? And keep the thoughts coming! — Trevor Bolliger, WMF Product Manager 🗨 22:17, 16 October 2017 (UTC)

3rd Draft Wireframes[edit]

Hi all, We made our 3rd round of wireframes. I made sure to incorporate community feedback like reducing the amount of whitespace. I'd love to what you think of the differences between wireframes? We also shortened the text across all of the wireframes.

We are looking for feedback :) One wireframe has summaries that say "summary: an addition" for an example, and for another removes "summary" and just says "an addition." Is this easy to read, does this come across as a summary or should we include the word 'summary'? What do you think of the different kinds of diffs we've included? Does one look better or is more readable than the other?

let me know :)

--CSinders (WMF) (talk) 21:39, 31 October 2017 (UTC)

Test the alpha version of the Interaction Timeline[edit]

Our team is proud to announce that an alpha version of the Interaction Timeline is now available for testing! We wanted to get the tool in your hands so you can let us know if we're headed in the right direction.

The Interaction Timeline can be found at https://tools.wmflabs.org/interaction-timeline/. If you provide two usernames and a wiki, the Interaction Timeline will display a chronologic list of edits made by the two specified users on pages where both users have edited. For example, if User:A edits on Washington, Texas, and Kansas and User:B edits California, Texas, and Kansas the tool would only show their edits on Texas and Kansas.

Known limitations
  • Due to current API limitations the tool only retrieves a maximum of 1,000 edits. This is our top priority to resolve. (T179702) The tool works like this:
    1. In the background, the tool retrieves the first 500 edits by both users from the provided start date (or first 500 edits of all time if no start date is provided.)
    2. In the background, the tool then generates a list of commonly-edited pages between the two users, from all time.
    3. On the timeline, the tool only displays the edits to the commonly-edited pages within the 1,000 edits that it already retrieved.
  • Performance is not optimized, and there is no loading indicator to show when the tool is still working to generate the results.
  • There is no error message when there are no results.
  • The tool does not support IP address, only usernames.
  • Typing in the date range only allows you to type one character at a time.
Examples to test
  1. User:Test-apples and User:Test-bananas on test.wikipedia.org
  2. User:Test-carrots and User:Test-durian on test.wikipedia.org
  3. User:Derby pie and User:Sweets lover on test.wikipedia.org
  4. User:Tinker toys and Baby rattle on test.wikipedia.org
Discussion topics

Thank you, and we sincerely look forward to your thoughts on our work, and your feedback on where to take it next. — Trevor Bolliger, WMF Product Manager 🗨 21:32, 13 November 2017 (UTC)

None of the test links work in Firefox. (They do work in Chrome.) But that's not important.
Over at EnWiki I suggested that you Explicitly calculate the ratio of how many edits are visible between the two tools.[1] I suspect no one did this. Select two actual editors with a significant number of edits. Bring up a page in each tool. Scroll the header-area off the top. Count how many entries are visible in each tool. Divide, and multiply by 100 to get a percent. Depending on the length of edit summaries, I can see as few as 4 entries in the Interaction_Timeline tool, and more typically 6 entries. In Edit Interaction Analyzer I can see about 30. Depending which way you look at the ratio, that is either 13% to 20% density for the new tool, or 500% to 750% density in the existing tool.
This is not a minor issue of whitespace styling. 500% to 750% is an entirely different planet. The new tool is designed like a twitter feed. We need a data-analysis-tool design. Alsee (talk) 21:21, 15 November 2017 (UTC)
@Alsee: We'll test in more browser/OS configurations. We did not calculate the percentages because we knew it had far less information density than we would like. We used off-the-shelf Bootstrap UI elements to build this very quickly. In phab:T180090 we will use Wikimedia branding and UI elements, which should standardize the font, text-size, and spacing of elements. For this alpha version we were 100% focused on getting the skeleton built. — Trevor Bolliger, WMF Product Manager 🗨 00:33, 16 November 2017 (UTC)
@Alsee: Hi! I'm the designer on this- and if you look at the wireframes above, that's more what the final tool will look like. We know there's a lot of white space in this prototype, a lot more than we intended. This is not the final look and feel of the tool- we were interested in getting a highly functioning and working prototype out the door. But, noted, there will be a lot less white space in the final version. Can ask, have you played around with the tool yet? What do you think of how it compares interactions between editors? What could be improved there?--CSinders (WMF) (talk) 00:48, 16 November 2017 (UTC)
CSinders (WMF), the wireframes look look like any significant difference in density. I didn't comment much beyond the raw density because I assume other functionality is being built... but ok. The current version isn't usable yet. I'll start in the context of the current Editor Interaction Analyzer. An established editor may have thousands, tens of thousands, or even hundreds of thousands of edits. The Analyzer narrows that down to pages where there are mutual edits (possibly hundreds of pages), and sorts them by shortest edit-interval between the users. If the interval is weeks or months, it's probably not relevant. Then I can start at the top, or at pages-of-interest, and pull up *just* the edits by those users on that page. They may have be dozens or hundreds of edits to the page, so I want to see as many edits at once as reasonably possible. Then I can try to understand what interaction, if any, there was on that page. Then I can check other pages they both edited.
The only thing the current tool has at the moment to help sift though all of the edits is the calendar. It shows all edits by all listed users, and it shows maybe 6 edits at a time. That may be less than 5 minutes. A busy editor can make a one or two dozen edits per hour, and an editor using AutoWikiBrowser might approach a thousand edits in a day. Showing 6 edits is like peeping through a straw, and there's nothing (yet) to filter relevant edits. In most cases it's hopeless to try to scroll through unfiltered edits a few at a time. Alsee (talk) 10:07, 16 November 2017 (UTC)
@Alsee: The Timeline only shows edits on pages where both users have edited for all time, but filters what is displayed on the Timeline based on the date range parameters. Here’s a hypothetical scenario:
  • User:Apples edited the articles ‘Hawaii’ and ‘California’ in June 2017 and ‘Kentucky’ and ‘Florida’ in July 2017
  • User:Bananas edited ‘Kansas’ in June 2017 and ‘California,’ ‘Kentucky,’ and ‘Montana’ in July 2017
  • If you perform a query for User:Apples and User:Bananas for July 2017, you will see Apples’ edits to Kentucky and Banana’s edits to California and Kentucky.
After we have full date-range support (the 1,000 maximum edits limitation is very frustrating and is one of our top priorities to fix) the Timeline and the Analyzer will show the same data in different formats, because the tools are designed for two different workflows: The Analyzer is designed to be able to drill into the interactions one page at a time while the Timeline is designed to be able to follow interactions across pages. From many cases of harassment we’ve seen, conversations are not confined to one single page. Often, a conversation will begin on one talk page, carry to their user talk pages, and culminate on a noticeboard. The edits to the article pages themselves are also important pieces of information, especially in cases of edit wars or stalking.
I do share two big concerns as you:
  1. The information density, which we will address in phab:T180090 in the next few weeks
  2. The ability to sift out the ‘noise’ edits, which may clutter up the Timeline. We are planning on building filters to allow certain pages or namespaces to be hidden on the Timeline. This work will begin in January/February.--CSinders (WMF) (talk) 22:01, 16 November 2017 (UTC)

Alpha feedback[edit]

See also: w:en:Wikipedia talk:Community health initiative on English Wikipedia/Interaction Timeline#Alpha comments
  • I wouldn't link to the generic tool until it works for most test cases. Instead, the link to the test example would work.
  • Sticky headers for the username columns (link the usernames to actual user pages)
  • I personally think it's more helpful to have a brace or bracket group five posts and show that they happened within the span of five minutes than it is to see the timestamps for each individual node
  • It's hard to get a sense of the disagreement's substance without inline diffs. The edit summaries (nevertheless section headers) often won't be descriptive enough on their own to be particularly useful, so the reviewing editor is back to opening diffs manually in a bunch of tabs.
smaller suggestions
  • username search should support lowercase usernames (czar => Czar)
Ideally, the search would be case-agnostic and show even cZAr => Czar
  • Ideally, the tool would autofill whatever project in which the first listed user participates most, or shows the user's most active sites atop the list of others—saves a few clicks
It'd be nice if "enwp" and "English Wikipedia" both corrected to en.wikipedia.org

czar 22:47, 15 November 2017 (UTC)

  • Thank you for the feedback, Czar. Once we complete Phab:T179702 (hopefully next week!) the tool will support all cases better. (I also think the loading indicator and error messages will help, since long queries can take several seconds to run. We'll optimize performance once we're sure this concept holds water.)
  • We'll build a sticky header in phab:T180245 which will hopefully be built before the end of the year. Same for in-line diffs (phab:T180417).
  • The first-character case sensitivity and illogical ordering for wikis is driving me mad too. We'll address those in phab:T180084 and phab:T180087 but I love your suggestion to automatically fill the wiki field based on where they edit. Or hey — maybe the tool could show all edits across all wikis (but that's a much bigger undertaking. Maybe later if this tool is successfully useful!) — Trevor Bolliger, WMF Product Manager 🗨 00:53, 16 November 2017 (UTC)