This is an email to shell account holders on
translatewiki.net and to
wikitech-l, so that you are informed.
Today at 08:10 UTC Niklas noticed that the
translatewiki.net server had
been compromised. We saw some suspicious files in /tmp and a few processes
that didn't belong:
elastic+ 22862 0.0 0.0 2684 2388 ? S 04:53 0:00
/tmp/freeBSD /tmp/freeBSD 1
elastic+ 31575 0.0 0.0 2684 2388 ? S 06:38 0:00
/tmp/freeBSD /tmp/freeBSD 1
elastic+ 31580 16.7 0.0 90816 724 ? Ssl 06:38 16:26
[.Linux_time_y_2]
We gathered data and looked at our recent traffic statistics. We drew the
following conclusions:
- Only the Elasticsearch account had been compromised. The intruder did not
gain access to other accounts.
- The attack could be made because the Elasticsearch process was bound to
all interfaces, instead of only the localhost interface, and dynamic
scripting was enabled, because it is required by CirrusSearch
(CVE-2014-3120).
- A virtual machine was started, and given the traffic that was generated
(about 1TB in the past 4 days), we think this was a DDoS drone. The process
reported to an IP address in China.
- A server reinstall is the right thing to do (better safe than sorry).
The compromised server was taken off-line around 10:00 UTC today.
Actions taken:
- Bind Elasticsearch only to localhost from now on:
https://gerrit.wikimedia.org/r/#/c/145262/
- Reinstall the server
Actions to be taken:
- Configure a firewall to only allow expected traffic to enter and exit the
translatewiki.net server so that something like the added virtual machine
could not have communicated to the outside world.
- As a precaution, shell account holders should change any secret that they
have used on the
translatewiki.net server in the past 7 days.
We are thankful to the people in the MediaWiki security IRC channel and
Henri Salo for helping us with data gathering on the attack, and how to
proceed.
We have re-installed the
translatewiki.net server, and are currently
re-importing the databases. We expect to be back online in a few hours.
Once we come back online, we'll still have to rebuild some non-critical
meta data stores, like populating the search database.
Cheers!
Siebrand