Talk:October 2013 private data security issue

Schema incompatibility

First off, thanks for the report, it's very much appreciated. I'm curious to know what the schema incompatibility was. According to [1] there have been no changes to the `user` table since May. Thanks, Legoktm (talk) 05:14, 3 October 2013 (UTC)

It's the removal of user_options in 1.18 that cause the problem; but wikis that were created before that change still had their column even if it was no longer used, so did not break the trigger (which was attempting to redact that column along with the others). MPelletier (WMF) (talk) 05:41, 3 October 2013 (UTC)

Whitelisting

Thank you for the prompt disclosure. I thought at some point in the past things were set so that tables and columns would need to be explicitly whitelisted before they'd be visible on the Toolserver/Labs. Did that not come into play here, or is that what you're referring to when you talk about the database triggers that failed? Emufarmers (talk) 05:56, 3 October 2013 (UTC)

You are correct; there was a bug in the system that applies the redaction trigger such that it didn't report the error when in failed on those wikis for that table. MPelletier (WMF) (talk) 06:06, 3 October 2013 (UTC)

It sounds like two errors then, doesn't it? Not failing loudly (error 1), but also not being a true whitelist (error 2). Right? --MZMcBride (talk) 06:38, 3 October 2013 (UTC)

It's a whitelist; that table was cleared with a set of redactions. Had the process stopped as supposed when it failed to apply the redaction, the actual replication would have never started. MPelletier (WMF) (talk) 06:42, 3 October 2013 (UTC)

I was under the impression that the new system whitelisted on a per table column basis, not on a per table basis. --MZMcBride (talk) 16:45, 3 October 2013 (UTC)

Common password

I have changed password after recent alert to change password. Unfortunately I use the same password for few more sites (I have not accessed few of those sites for a long time and I may face trouble to find out all those sites now). Is there any chance that my those accounts are in danger? --Tito Dutta (Talk) 05:58, 3 October 2013 (UTC)

Yes, you should change your password on all sites. If the password database was stolen by a malicious LabsDB user (we have no indication that this actually occurred), then unless the password is very strong, the malicious user could extract your password from the database and use it against other sites. -- Tim Starling (talk) 06:04, 3 October 2013 (UTC)

OK, instead of starting a new section I'll post here since it's very related. So just what system were you using to store the passwords? Surely by design you shouldn't be able to easily go from the hash to the password. Given that I suspect most (if not all) user accounts would be very low priority for hackers, and it seems very unlikely that hackers would have had access anyway, it seems to me you've just inconvenienced a large number of users for a very low risk - and possibly created more risk as I suspect many users will have to write down new passwords to remember them. I'd also like to point out that if you hadn't forced SUL upon us I wouldn't be in this situation as although I've read at least one of the effected sites I don't think I've ever edited them. Maybe you should disable SUL as it's clearly a security risk if it means password hashes are stored in more places than they need to be. Dpmuk (talk) 06:12, 3 October 2013 (UTC)

We store salted hashes with two rounds of MD5: MD5(salt | MD5(password)). This is unfortunately insufficient for typical passwords used on the web. See bug 28419 for a discussion of this issue and suggested mitigations. Wikimedia does not enforce the use of strong passwords, so we expect that many of the potentially compromised hashes can be attacked by brute force.

The risk may be low, but we felt that it was most prudent to notify affected users and to force a password reset.

Wikimedia's SUL implementation does not copy password hashes into the local wiki's database when you read or edit that wiki, so if you merely read one of the affected wikis, it may be that your password was not compromised. You will be informed whether or not your password was compromised when you log in to any account attached to a compromised account via SUL. The SUL password database was not compromised -- the leak was actually the result of the legacy system of optional SUL. If SUL was fully enforced and compulsory, the leak would not have occurred. -- Tim Starling (talk) 06:32, 3 October 2013 (UTC)

Well I've been told I'm affected and I have no idea how then but I'll take your word for it. I started to lose the will to live when that bug started discussing function names and correct OO design - another example of why Bugzilla may be great for developers but is useless for end users - we really could do with some in between layer. It's pleasing to here that SUL shouldn't increase risk but rather disappointing to hear you're still using MD5 given how long it's been known to have vulnerabilities and how long much better alternatives have existed. Still think the inconvenience way outweighs the risk although I accept views will differ on that and I'm probably in the minority. Dpmuk (talk) 07:11, 3 October 2013 (UTC)

And they may or may not have your email address too... --Rschen7754 06:28, 3 October 2013 (UTC)

@DPmuk: Maybe you should consider using a Password manager (like keepass (OS) or 1password ($$)) to generate safe and unique passwords and handle them in a (more or less) convenient way. --78.49.125.126 06:51, 3 October 2013 (UTC)

Usernames exposed too?

Were usernames exposed too? --Kozuch (talk) 07:04, 3 October 2013 (UTC)

What does that mean? Special:Listusers NPKC? --Mormegil (cs) 07:10, 3 October 2013 (UTC)

I think Kozuch is asking whether usernames were linked to the (other?) data that was exposed. There's a significant difference between having a bunch of password hashes in isolation, and having each one linked to the relevant username. I presume that they were linked to usernames, based on the actions WMF has taken to prevent potential abuse, but the report is silent on this point. --Avenue (talk) 07:29, 3 October 2013 (UTC)

Yes, someone who was able to retrieve the hash would have been able to retrieve the associated username as well. -- ArielGlenn (talk) 07:58, 3 October 2013 (UTC)

The solution...

The solution seems not to have been fully thought through. If someone has stolen my password and logged in as me, they can just change my password and then my email address to lock me out from my account forever. I have obviously fixed this already, but still allowing logins with the old password and then just changing it without even forcing an email confirmation (hard if no email has been given yes, which is an argument for why that should be required) seems to amount to the same as doing nothing at all more than informing us about the situation. /Grillo (talk) 08:15, 3 October 2013 (UTC)

Informing editors was our primary concern. Given that we have no indication that the data was accessed at all, changing passwords is a strictly precautionary measure indicated in this situation; but it also will protect you in the case the data was stored for later attacks. MPelletier (WMF) (talk) 14:07, 3 October 2013 (UTC)

Accounts with same email

A minor source of confusion is that the notice, at least in my case, was sent only once: to my main username, as I gather from the To: header (I suppose unmerged accounts have received one email per each local account?). That's a nice "spam" reduction; the same email however is used for some other accounts and I'm not told which were compromised. For instance, one of my bots was but another not. Can we add a short clarification that users can check on Special:CentralAuth if their global account has a local account on one of those compromised wikis? --Nemo 08:24, 3 October 2013 (UTC)

The determining account isn't whethere there is a local account, but whether actual credentials were once stored there – this normally happens only to unmerged accounts, or on the wikis where the account was originally created. An account created automatically through SUL will never have credentials in the local database. MPelletier (WMF) (talk) 13:02, 3 October 2013 (UTC)

Yes, that's what I was told last night too: that since my SUL account had been created long before, the fact that I'd edited three of the wikis on the list was irrelevant. Unfortunately, I could not log in today without changing my password. As best I can tell, the wikis I used that are on this list are not SUL wikis, so I do not understand why I need to change my SUL password instead of just the passwords for the (non-SUL) accounts on those wikis. As best I can figure, those accounts remain vulnerable, whereas the one account that wasn't vulnerable is the one that required a password change. Risker (talk) 13:27, 3 October 2013 (UTC)

We have no way of knowing whether or not a password that was used on a local account was the same as your SUL account or not; anyone who has had an exposed hash was made to change their password, regardless of whether they were then unified or not for that reason. MPelletier (WMF) (talk) 13:58, 3 October 2013 (UTC)

Well that's fairly counterintuitive. The problem is that without changing the password on those affected non-SUL wikis, much of the information is still valid there, including password. That isn't clear on the page or in the instructions. I recommend that the instructions specify this. I'm not particularly worried about someone hijacking my accounts on those wikis (my passwords are rated "very strong" using all conventional and xkcd considerations), but someone with a weak password on one of those projects would be more susceptible to hijacking. Risker (talk) 15:51, 3 October 2013 (UTC)

You're correct, of course. Ideally we'd have wanted to tell the affected users which accounts on which wiki triggered the notice, but we wanted to get the warning out as quickly as possible and that did not leave us enough time to construct customized emails. I'll ask the LCA people to see about making clear instructions for that case. MPelletier (WMF) (talk) 15:58, 3 October 2013 (UTC)

Thanks, Marc. I agree that it was better to get things out there for people, and the majority of the affected wikis are SULs. And thanks to you personally (as well as the rest of the Ops team) for your work on this; probably not something you expected to run into when providing support to users migrating to Labs from Toolserver. :-) Risker (talk) 16:09, 3 October 2013 (UTC)

I just discovered that one of the database affected is on a non-SUL wiki that I have an account on. Similar to Risker, I was forced to change my SUL password, but was not made to do so on the non-SUL wiki where I definitely has (an affected) local account. Without specifying which specific account (and on what wiki) was affected, I could have easily (and did) forget it. Were only the SUL login session token invalidated with a password reset? Not all the affected databases are associated with a SUL wiki. -- KTC (talk) 15:13, 4 October 2013 (UTC)

If it's impossible to find out whether the credentials were leaked, then my question stands: let me rephrase, «users can check on Special:CentralAuth if their global account has a local account on one of those compromised wikis and hence their credentials were potentially leaked». Correct? --Nemo 14:38, 3 October 2013 (UTC)

If you look up your account at Special:CentralAuth when you are logged in, then you have an account on at least one of the listed wikis, unless you haven't activated SUL at all. This is because loginwiki accounts should be created for anyone who has activated SUL and who has logged in since SUL was changed earlier this year. What is your point? --Stefan2 (talk) 14:54, 3 October 2013 (UTC)

Why would one check it as logged in? My point is that I followed that way to discover what accounts of mine were affected. Nobody has provided an alternative way so far; if you know one, please share. --Nemo 10:58, 4 October 2013 (UTC)

A proper hashing scheme

A bug report was filed two and a half years ago talking about the hashing scheme. Nothing happened! Even though hashing encapsulation could have been used to avoid a mass password reset, still nothing happened. The news about LinkedIn, UbuntuForums, etc. and nothing happened.

What's even worse, the suggestions for replacement are WHIRLPOOL and SHA-2. I advise the people responsible for this to check the article at Security.StackExchange about proper password hashing. I didn't post this in the bug report because I don't think it will change anything if posted there. --Adnan Suomi (talk) 08:35, 3 October 2013 (UTC)

Password hashing is easy to get wrong, as that article points out, and that bug is something I've been working on as a low priority for a while. Obviously it's become a higher priority at this point, so do expect some activity soon. CSteipp (talk) 16:12, 3 October 2013 (UTC)

loginwiki

I just wanted to make it clear - does the fact that the "login" wiki is in the list mean that this issue potentially affects all the users who have used SUL after the launch of login.wikimedia.org took place this year, and that those users should be required to change password, regardless of the local wiki(s) they have logged in to? --whym (talk) 09:31, 3 October 2013 (UTC) If that is the case, I think this background information is better explained in the document. I was personally unsure why I was affected and required changing password, when quickly looking at the list of the wikis, none of which I have logged in to as far as I recall. --whym (talk) 11:36, 3 October 2013 (UTC)

No, the trigger is whether you had a local password on the database, SUL is not affected in any way. In general, people will have had hashes on a wiki if this is where the account was created originally, or if the account was not unified. Simply logging in on a wiki does not copy your credentials there. MPelletier (WMF) (talk) 12:40, 3 October 2013 (UTC)

Forgive me if I get it wrong, but are the wiki with my credentials stored corresponding to the entries of "home wiki" and "confirmed by passowrd" on Special:CentralAuth/Whym (ja.wikipedia and meta.wikimedia in my case)? --whym (talk) 13:33, 3 October 2013 (UTC)

Those wikis that are "confirmed by password" do have local credentials. Your home wiki doesn't necessarily have any – that is simply the one you have the most edits on – but it is likely that it does since for most people this is where their account was created. MPelletier (WMF) (talk) 14:01, 3 October 2013 (UTC)

As stated by MPelletier too, it depends on how you logged into the affected wiki that determines if your password was stored. If you were logged in and visited the wiki, you password hash is not stored on the local wiki. Your password is only stored when you create a new account on the wiki by logging in, or if the global account was unattached by a steward. In all, there were only about 700 accounts with a password hash on loginwiki, despite over 4M users using it for SUL.CSteipp (talk) 16:35, 3 October 2013 (UTC)

Thanks, now I seem to understand it. It sounds like if I created account by manually logging into one of the local wikis entering credentials there, which I probably did, then it made me affected by this issue. --whym (talk) 11:52, 4 October 2013 (UTC)

I have a similar question, but the other way round: I didn't receive the e-mail, so I assume that I'm not affected. However, I think I must have visited some of the wikis mentioned while logged in through SUL (I log in mainly through dewiki or Commons, which apparently aren't affected), and now I'm wondering whether I'm really not affected or maybe I should have received the e-mail and it got lost... or do affected users get a "you must change your password" message when logging in? Gestumblindi (talk) 12:33, 3 October 2013 (UTC)

Yes, affected user are forcibly logged off and made to change their passwords. If you weren't, then you weren't affected. MPelletier (WMF) (talk) 12:40, 3 October 2013 (UTC)

Again, not entirely correct. I was not forcibly logged out of my SUL account when this change was made or in the hour afterward; however, I could not log into my SUL account today until I changed my password. Risker (talk) 13:29, 3 October 2013 (UTC)

The two mechanisms are distinct. Any existing session on the affected wikis have been terminated (because of the session token), and the account is made to change its password (because of the hash and the impossibility to know whether you have been using the same password everywhere or not). MPelletier (WMF) (talk) 14:03, 3 October 2013 (UTC)

private data published

In your e-mail you wrote that private data may have been published and even under CC-BY license? Weird. I didn't even know you stored it where people can abuse it e.g. for sending spam, create an advertisement with my name, e-mail address etc. The fact that you have such an open structure makes it very vulnerable I'm afraid. I had myself access as DBA to data of our clients, but I signed a non-disclosure contract. Here it's the contrary. I'm in fact client and contributor together and may become a victim of your system. If this happens you lost a piece of my trust to say it carefully. Sorry for my bad English.  Klaas|Z4␟V:  10:17, 3 October 2013 (UTC)

Private data is stored on servers with restricted access. We have a process in place for removing private data before anything is replicated to the LabsDB servers, but, as the email says, there was a problem with that process for some wikis, which has now been corrected. You can be sure that we do not ever intentionally store private data where it can be generally accessed. We have an open infrastructure, yes, but not that open. -- ArielGlenn (talk) 10:46, 3 October 2013 (UTC)

Should this be published as a CentralNotice or SiteNotice, so that all registered users can, upon next login, choose whether to change their password or not? Or maybe just send out a mass email to those users who have provided one... --Lexein (talk) 12:20, 3 October 2013 (UTC)

Every user whose hash was visible was marked in the database so that they are forced to change their password on-wiki, so even if they have not gotten the email they have a much more direct mechanism of notification. MPelletier (WMF) (talk) 12:43, 3 October 2013 (UTC)

SUL

With SUL implemented, why those data were not simply removed from all the local wiki databases? It seems to me that the higher risk comes from storing those data in all different databases, and since we have now a central SUL why is it still like that for all users that had accounts created before? Private data should be deleted from the databases were they don't need to be. Amqui (talk) 13:19, 3 October 2013 (UTC)

I pretty much agree with that. Heck I only had one place that I was affected and I've had to change the password here? Is this because I'm fully SUL'd up? MIVP - (Can I Help?) - (HeadQuarters) 13:48, 3 October 2013 (UTC)

You are correct, and the operations team is looking into doing just that (remove login data that is no longer used). MPelletier (WMF) (talk) 14:05, 3 October 2013 (UTC)

D'accord, merci pour l'info MPelletier. Amqui (talk) 23:57, 3 October 2013 (UTC)

Questions from Wiki Edits Examiner

I just sent this note to the Wikimedia Foundation "accountsecurity" e-mail. In the interest of time, and the fact that some WMFers consider me part of "a list of exactly two people that the WMF feels obliged never to engage with", I'll post it here, too:

• As the Wiki Edits Examiner at Examiner.com, I will be publishing a story today about this security lapse. I have several questions, if you would be kind enough to respond on the record:

(1) How do you describe what "LabsDB" is? Is it a Wikimedia site, a community, a project, or what? The word "infrastructure" is a bit ambiguous.

(2) Is LabsDB congruent with wikitech.wikimedia.org, or with en-labs.wikimedia.org (which seems to be broken), or with some other domain? That is, how would a LabsDB volunteer "log in" to LabsDB?

(3) Were LabsDB participants vetted by the Wikimedia Foundation in any way?

(4) "This issue was discovered and reported by a trusted volunteer." How was the trustworthiness of the volunteer determined? We have all seen in the past that even Wikimedia Foundation employees (Carolyn Doran, Oliver Keyes, et al) cannot always be trusted -- how do we trust that the WMF is able to determine if a given volunteer is "trusted"? What is the name of the volunteer who reported the security flaw?

Thank you. I will hold my publication until 2:30 PM Eastern, to allow you time to respond. If you elect not to respond, please also let me know that.

Gregory Kohs

Examiner.com reporter

Voice: 484-NEW-WIKI

End transmission. -- Thekohser (talk) 13:44, 3 October 2013 (UTC)

Hi, Wikimedia Labs is where volunteers and staff alike can develop software, run bots and host tools. LabsDB is a cluster of databases within Wikimedia Labs that contain a near real-time read-only copy of the main wiki databases (e.g. pages, revisions, categories etc.). This allows those bots and tools to query this database for information to then act on (e.g. a bot that automatically finds and reverts vandalism, or a web tool that admins can use to find orphaned talk pages that should be deleted etc.). As further explained in the main post, this copy is redacted to only expose data that one is able to already access from the Wikipedia website or its API.

One can roughly compare the purpose and access restrictions of Wikimedia Labs to that of Toolserver (from the Wikimedia Deutschland chapter). Though Wikimedia Labs is bigger than Toolserver, in both environments anyone can sign up for an account and start developing software (within their own space). Both environments feature a redacted replica of the wiki databases. –Krinkle^talk 16:30, 3 October 2013 (UTC)

The reporter is listed on bugzilla:54847 Legoktm (talk) 06:35, 4 October 2013 (UTC)

It is best to ignore him, tbh. The examiner is little more than a glorified blog. --Guerillero 16:08, 4 October 2013 (UTC)

That's a rather hurtful thing to say, Guerillero. Examiner is the 217th most visited website in the United States (according to Alexa) reaching 19 million Americans last month (according to Quantcast). My National Wiki Edits Examiner stories in particular have garnered over 86,000 page views since 2010, when I started writing for Examiner. I have an editor who watches my content, and (yes) I have had to make some changes to an article or two to meet Examiner's editorial guidelines. So, it is not a "blog". If you are an expert at blogging, may I ask you how many page views to which one should aspire, so as not to be "ignored"? - Thekohser (talk) 20:04, 4 October 2013 (UTC)

For anyone interested, the article turned out as such. -- Thekohser (talk) 20:07, 4 October 2013 (UTC)

More careful

Really. Next time be more careful. PiRSquared17 (talk) 15:05, 3 October 2013 (UTC)

Per PiRSquared17. Southparkfan 16:37, 3 October 2013 (UTC)

Obviously this was a very unfortunate incident. However, shit happens. The most important aspect here, in my opinion, is that lessons must be learned and better systems must be put in place to try to ensure that an incident like this never happens again. --MZMcBride (talk) 16:52, 3 October 2013 (UTC)

We're already reviewing the systems in question, and will be adding at least a layer of redundancy to the checks so that if it fails again it will fail in the right direction (fail to replicate at all). MPelletier (WMF) (talk) 17:14, 3 October 2013 (UTC)

For starters, you shouldn't be playing in any way with any user data. Yes, mistakes happen, but messing with what you should not be messing carelessly greatly improves the odds, doesn't it? On a positive note, thank you for the warning. - Nabla (talk) 20:13, 3 October 2013 (UTC)

I'm not sure what you mean by "playing with user data". No one was just messing around. Legoktm (talk) 06:37, 4 October 2013 (UTC)

Right. Sixth, or whatever, largest site on the web lets 5 months worth of private user information available to hundreds of other users - who just happen to be the most likely be the skilled enough to make use of it - and "nothing" happened... As little as I can get from the above, only "whitelisted" information should be made available, so if private information was eventually available, it either was whitelisted, or the whitelist is poorly implemented. It seams to be the case here. Apparently they(?) do not select specific columns/tables - as in "SELECT this-column, that-columnm, ... FROM ..." and then export that copy - instead they delete ('redact') specific column and then export the data. I am only an amateur coder, but this seems like messing around carelessly, because, alas, a change in a schema would allow data to get through. That I wouldn't think of it, is understandable, but they are coding one of the largest site's on the web, we would expect much better than that. - Nabla (talk) 10:04, 4 October 2013 (UTC)

Global accounts (unified login)

It should be noted that anyone with a global account will only need to make a single password change on any Wikimedia project site (e.g. Meta, English Wikipedia, German Wikisource, Wikimania 2014, etc.) More information at Help:Unified login. --Lightsup55 ( T | C ) 19:04, 3 October 2013 (UTC)

More precisely, anyone with a single global account without local unattached accounts. ;) --Nemo 11:00, 4 October 2013 (UTC)

Informing users with no email

As users were individually informed about this issue by e-mail, I wonder if those that have no email linked to the wiki account were informed too.—Teles «Talk to me ˱_{C L @ S}˲» 23:16, 6 October 2013 (UTC)

When they try to log in, they will see the message. PiRSquared17 (talk) 23:22, 6 October 2013 (UTC)

Ah, good. That's something.—Teles «Talk to me ˱_{C L @ S}˲» 23:53, 6 October 2013 (UTC)