Mirroring Wikimedia project XML dumps: Difference between revisions

Content deleted Content added

Inline

Revision as of 10:57, 17 May 2012

This page coordinates the efforts for mirroring Wikimedia project XML dumps around the globe, on independent servers, similar to the GNU/Linux .isos mirror sites. See the list of mirrors below for the dumps.

Requirements

Space

Last 5 good dumps (most desired option): 6 TB for 5 most recent dumps, as of Jan 2012.
Last 2 good dumps: 2.5 TB, as of Jan 2012.
Only most recent good dumps : 1.3 TB, as of Jan 2012.
Historical archives (2 dumps per year from 2002 through 2010): 1.6T now (Jan 2012), missing some data, expect 3-4T total.
All dumps currently hosted: about 27TB and growing, as of Jan 2012.
Expect slow growth; the number of dumps we keep will not grow substiantially but the number and size of projects will increase steadily.

We are not very interested in selectively mirroring some projects or dumps.
"Other" (pageview and other statistics): 2.3T, as of Jan 2012, allow growth to 4-5T.

Bandwidth

Wikimedia provides about 20 MB/s via dataset2.wikimedia.org (stats) for XML dumps, as of January 2011.

Current Mirrors

Organisation	Contents	HTTP access	FTP access	rsync URL
Wikimedia	XML dumps	http://dumps.wikimedia.org	none	none
C3SL	last 5 good XML dumps	http://wikipedia.c3sl.ufpr.br	ftp://wikipedia.c3sl.ufpr.br/wikipedia/	rsync://wikipedia.c3sl.ufpr.br/wikipedia/
Masaryk University	last 5 good XML dumps	http://ftp.fi.muni.cz/pub/wikimedia/	ftp://ftp.fi.muni.cz/pub/wikimedia/	rsync://ftp.fi.muni.cz/pub/wikimedia/
Your.org	XML dumps	http://dumps.wikimedia.your.org/	ftp://ftpmirror.your.org/pub/wikimedia/dumps/	rsync://ftpmirror.your.org/wikimedia-dumps/
Your.org	media (current version only)	http://ftpmirror.your.org/pub/wikimedia/images/	ftp://ftpmirror.your.org/pub/wikimedia/images/	rsync://ftpmirror.your.org/wikimedia-images/

For an unofficial listing of torrents, see data dump torrents.

Who can we contact for hosting a mirror of the XML dumps?

If you are a hosting organization and want to volunteer, please send email to ariel -at- wikimedia.org with XML dumps mirror somewhere in the subject line.

If you are brainstorming organizations that might be interested, see discussion page.

External links

dumps.wikimedia.org

@@ Line 14: / Line 14: @@
 ===Bandwidth===
 Wikimedia provides about 20 MB/s via [[:wikitech:Dataset2|dataset2.wikimedia.org]] ([http://ganglia.wikimedia.org/?c=Miscellaneous%20pmtpa&h=dataset2.wikimedia.org&m=cpu_report&r=month&s=by%20name&hc=3&mc=3 stats]) for XML dumps, as of {{date|2011|01}}.
+==Current Mirrors==
 {| class="wikitable"
 |-
-! Organisation !! HTTP access !! FTP access !! rsync URL
+! Organisation !! Contents !! HTTP access !! FTP access !! rsync URL
 |-
-| Wikimedia || http://dumps.wikimedia.org || ''none'' || ''none''
+| Wikimedia || XML dumps || http://dumps.wikimedia.org || ''none'' || ''none''
 |-
-| C3SL || http://wikipedia.c3sl.ufpr.br || ftp://wikipedia.c3sl.ufpr.br/wikipedia/ || rsync://wikipedia.c3sl.ufpr.br/wikipedia/
+| C3SL || last 5 good XML dumps || http://wikipedia.c3sl.ufpr.br || ftp://wikipedia.c3sl.ufpr.br/wikipedia/ || rsync://wikipedia.c3sl.ufpr.br/wikipedia/
+|-
+| Masaryk University || last 5 good XML dumps || http://ftp.fi.muni.cz/pub/wikimedia/ || ftp://ftp.fi.muni.cz/pub/wikimedia/ || rsync://ftp.fi.muni.cz/pub/wikimedia/
+|-
+| Your.org || XML dumps || http://dumps.wikimedia.your.org/ || ftp://ftpmirror.your.org/pub/wikimedia/dumps/ || rsync://ftpmirror.your.org/wikimedia-dumps/
+|-
+| Your.org || media (current version only) || http://ftpmirror.your.org/pub/wikimedia/images/ ||ftp://ftpmirror.your.org/pub/wikimedia/images/  || rsync://ftpmirror.your.org/wikimedia-images/
 |}
 :''For an unofficial listing of torrents, see [[data dump torrents]].''