Talk:Data dump torrents

Archives

---

1

Translate tags

If someone could fix the translate tags and explain to me how they work, that would be great, thanks. --Thibaut120094 (talk) 15:27, 6 December 2018 (UTC)

20190120 enwiki torrent is not linked

There's no link for any 2019 English-language torrents. I hate to burden Wikimedia with downloading the 20190120 enwiki, but I don't know how to get it otherwise.

Where are the checksums

Shouldn't we have checksums here or some information on where to find them? Seems like a good idea to have some pointer to this information so people know that they are actually getting the download they think they should be getting? --Fincle (talk) 21:36, 22 April 2019 (UTC)

Yes, I think they should be included. (Also, the "dump-torrents" project on tools.wmflabs.org claims to have sha1 and md5 checksums (example), but when I click the links I get 404 errors.) PiRSquared17 (talk) 21:51, 22 April 2019 (UTC)

Guys, all checksums are available on the page of each of Wikipedia releases. Here for the english version: https://dumps.wikimedia.org/enwiki/ you'll find the checksums below the words "Dump complete", usually a few days after the dump is actually completed. 2803:9800:9017:4C35:C900:7DEA:81E6:4F5E 20:29, 12 March 2022 (UTC)

---

You can obtain the checksums from Wikimedia directly via the following process:

• Know the ISO 639 two-letter code for the language that you want, and the exact Date of the dump that you need before continuing. We'll refer to language as LL and date as YYYYMMDD in this case.
• Insert these into the appropriate places in the following URLs:
  • https://dumps.wikimedia.org/LLwiki/YYYYMMDD/LLwiki-YYYYMMDD-sha1sums.txt
  • https://dumps.wikimedia.org/LLwiki/YYYYMMDD/LLwiki-YYYYMMDD-md5sums.txt
• Use the resulting direct URLs as appropriate e.g. curl, wget, paste into your browser address bar etc.

So if you want the SHA-1 checksums for the German Wikipedia dump of the 1st January 2024, you would build the URL https://dumps.wikimedia.org/dewiki/20240101/dewiki-20240101-sha1sums.txt (Note: Wikimedia only keeps dumps for about three months after generation, so this URL won't work.)

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 03:35, 23 June 2025 (UTC)

Tutorial anyone?

Can anyone knowledgeable please include a tutorial on how to create torrents? I'm wishful to download dumps and then share them here but I wouldn't know how to properly do it.

I second this, “OlaftheMeta” and academictorrents.com have some kind of exclusive hold on this and fails to upload on the first of the month. This could become a larger issue in the future. Twillisjr (talk) 17:55, 5 June 2026 (UTC)

Blocked in UK or on Sky Broadband?

Are these bittorrents blocked in the UK or on Sky Broadband? The servers are completely non-responsive for me. Both nicdex.com and litika.com have the same IP address, as does torrage.info (90.207.238.183) :/ Why aren't these torrent files on wikipedia itself?

Just to help answer this question for anybody else who has it: Most torrent sites and trackers are blocked in the UK (And most western nations) on the grounds of the upholding of Copyright law, the notable exceptions being sites/trackers connected with specific lawful activities e.g. the distribution of Linux media and free software using sites/trackers which block attempts to access torrents that aren't in a whitelist. This does not prevent torrents from working in themselves, but does impede access to other peers who are sharing that content.

Alternatives/lawful workarounds to this are to download the desired dumps from a Wikimedia mirror, or - If the desired content is available on Archive.org - To click the Show all link on the right hand side of the items information page and try the torrent file provided there. (This might be hit and miss in some cases; See my reply further down for some reasons why.)

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 03:19, 23 June 2025 (UTC)

Entire Wikipedia Including All History?

I would like to download a copy of all of Wikipedia, preferably all languages, and complete with history and deleted pages in a way that I can stand up my local copy. How much storage do I need to get this up and running, and where do I get the data from, please? I have a Mediawiki server already, so something like "SQL including everything plus a media tarball" or so would be preferred. The description on the website https://dumps.wikimedia.org is a bit vague and suggests that the SQL dump only contains the SQL for one day. 2A02:C7C:399B:9700:77F6:96ED:55D3:23BB 22:08, 2 February 2023 (UTC)

I'm a bit late to answer this question (I only joined Wikipedia about a week or two ago, though I've been editing anonymously for years) but my understanding is that at point of my reply, a complete copy of just the current Wikipedia in all languages with all content from Commons would exceed 400TiB. Add the entire meta subset (All page revisions and edit histories) and you're potentially looking at 1,5-2x that amount = 600-800TiB.

Conversely; My collection (Current pages only in De, Es, Fr and Brexit) runs to about 43GB compressed or 190GB uncompressed, and I find that's more than good enough for encyclopædic reference and language improvement.

Are you running a data centre there? (:

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 03:52, 23 June 2025 (UTC)

Link "Other list of torrents in the same style as the dump list" problem

Clicking on it, I get

“

No web-service

The URL you have requested, https://dump-torrents.toolforge.org/, is not currently serviced. If you have reached this page from somewhere else... This URI is managed by the dump-torrents tool, maintained by Legoktm . That tool might not have a web interface, or it may currently be disabled. If you're pretty sure this shouldn't be an error, you may wish to notify the tool's maintainers (above) about the error and how you ended up here. If you maintain this tool You have not enabled a web service for your tool, or it has stopped working because of a fatal error. Please check the error logs of your web service.

” Playmobil111 (talk) 09:17, 13 February 2023 (UTC)

Remove sketchy link for 2024-07-01 dump

That site is full of popups, can't find the file. AlwaystheVictor (talk) 10:52, 24 July 2024 (UTC)

I've started adding links for Academic Torrents, which should be more trustworthy. Willbeason (talk) 18:17, 3 September 2024 (UTC)

Are there index files for those torrents? (Maybe I missed this somewhere?) Mnegus01 (talk) 15:35, 23 March 2025 (UTC)

Uncompressed size

Do you think it'd be possible to list he uncompressed sizes? Especially for Wikidata as I know it's over 1TB and I genuinely don't know if I have the space to uncompress it lol Asdfghjohnkl (talk) 22:47, 21 December 2024 (UTC)

I uncompressed the 20250401 multistream dump of enwiki and the compression ratio is 4.3/1. I'll add a section about that. OloffTheMeta (talk) 12:34, 3 May 2025 (UTC)

I've run test decompressions of the French, German and Spanish dumps from 2025-06-01 plus the English dump from 2025-05-01 today (See my post below for a simpler method) and they come out as follows:

• dewiki-20250601-pages-articles-multistream.xml.bz2: 7,30GB compressed, 30GB decompressed.
• eswiki-20250601-pages-articles-multistream.xml.bz2: 4,70GB compressed, 21GB decompressed.
• frwiki-20250601-pages-articles-multistream.xml.bz2: 6,40GB compressed, 30GB decompressed.
• enwiki-20250501-pages-articles-multistream.xml.bz2: 24,87GB compressed, 109GB decompressed.

I've also made a slew of edits to tidy up the page and make it more uniform/easier to use (N.B: This means translateability of the page may need re-checking) and where I have the decompressed sizes I've added those to the appropriate entries.

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 11:58, 10 June 2025 (UTC)

Mirrored non-English dumps/torrents on Archive.org

Being bothered by the absence of up-to-date torrents for some non-English Wikipedia versions, I've just cloned the latest dewiki, eswiki and frwiki multistream dumps to Archive.org (See collections wikipedia-20250601-%LANG%-pages-articles, where %LANG% is the ISO-639 code for the desired language, and add -index to the end to get a clone of the dump index file) which generates web-seeded torrents for uploaded material. These have been added to the Data dump torrents page with magnet links, along with some tidying-up of the page sections I've worked on. Although Archive.org supports hosting and distribution of material via BitTorrent it also adds its own small metafiles to each collection, and these are also included in the torrents.

I can't see this being an issue (The additional metafiles are few, and small with the exception of one pad file employed for technical reasons to do with torrents) as it allows people to access the dump by torrent, but it does mean that other torrents of the same file without the Archive.org metafiles won't share seeds or magnet links with those downloading the Archive.org torrents. For this reason I'd recommend encouraging use of the latter as Archive.org torrent framework is very robust, even if the additional files are marginally irksome and cause Archive.org torrents not to gel/share with independent torrents of the same underlying file.

Could anybody with write access to the wikimediadownloads and wikicollections collections on Archive.org please propose/add the following Archive.org articles to them:

• wikipedia-20250601-de-pages-articles-index
• wikipedia-20250601-de-pages-articles
• wikipedia-20250601-es-pages-articles-index
• wikipedia-20250601-es-pages-articles
• wikipedia-20250601-fr-pages-articles-index
• wikipedia-20250601-fr-pages-articles

And can someone also please double-check that my use of German comma decimal in the dewiki listing conforms to Wikipedia policy and I'm not breaking translation functionality, please?

Thankyou :-)

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 07:43, 10 June 2025 (UTC)

(A long-time Wikipedia user, but who lost access to his account many years ago. A curse on passwords! ;-)

Extensive Page Tidy-ups and Improvements for Clarity

Separate to my thread above about using torrents hosted on Archive.org, I've also gone through most sections on the page today to make the listings clearer and easier to read/use for English, French, Spanish, Portugese and German. The principles/editing style I've employed for this are as follows:

• Each item should occupy a single line to produce lists that are easier to read. Where multiple items have the same date, these are grouped together and indented under that date. The benefits of this are very clear when comparing the appearance of the English Wikipedia section as it appears now compared to before any of my edits.
• The most relevant item to an ordinary user in each section - i.e. The latest/most up-to date item that visitors are most likely to be looking for - Is highlighted in bold, and any other entries have the emboldening removed. This facilitates quicker identification of what is likely to be the most relevant item to the majority of visitors so they can access it directly. This makes longer sections (e.g. enwiki) considerably easier to use.
• Section headings and historical torrent boxes have been headed in a multilingual fashion for increased usability by visitors seeking torrents in those languages, noting some cultures are put-off by any requirement to use or work in English. Full dependence has been placed on Google Translate for the Hebrew translation, and may need checking.
• Torrents are listed in reverse chronological order, with the newest item appearing at the head of each section.
• Torrents appear in the Older torrents box if they are older than six months (i.e: More likely to be "dead" torrents that are no longer seeded) and more recent torrents appear above the box for ease of identification.
• Where possible/known (See below) the decompressed size of each item is also given, using a [Compressed size] --> [Decompressed size] syntax e.g. « 6.4Go --> 30Go » for frwiki-20250601-pages-articles-multistream.

The decompressed size of a Wikipedia dump (And most other compressed individual files) can be ascertained without having to extract the entire file to disk. On Linux and MacOS computers which support bzip2 and dd, the following command may be used:

• bzip2 -dc ./pages-articles-multistream.xml.bz2 | dd bs=1048576 iflag=fullblock of=/dev/null

And produces output similar to the following:

• user@host:/Wikipédia$ bzip2 -dc ./dewiki-20250601-pages-articles-multistream.xml.bz2 | dd bs=1048576 iflag=fullblock of=/dev/null
• 28894+1 records in / 28894+1 records out / 30297913646 bytes (30 GB, 28 GiB) copied, 1866.67 s, 16.2 MB/s

• user@host:/Wikipédia$ bzip2 -dc ./eswiki-20250601-pages-articles-multistream.xml.bz2 | dd bs=1048576 iflag=fullblock of=/dev/null
• 20301+1 records in / 20301+1 records out / 21287906551 bytes (21 GB, 20 GiB) copied, 1202.21 s, 17.7 MB/s

• user@host:/Wikipédia$ bzip2 -dc ./frwiki-20250601-pages-articles-multistream.xml.bz2 | dd bs=1048576 iflag=fullblock of=/dev/null
• 28257+1 records in / 28257+1 records out / 29630113497 bytes (30 GB, 28 GiB) copied, 1749.54 s, 16.9 MB/s

• user@host:/Wikipédia$ bzip2 -dc ./enwiki-20250501-pages-articles-multistream.xml.bz2 | dd bs=1048576 iflag=fullblock of=/dev/null
• 104098+1 records in / 104098+1 records out / 109155593548 bytes (109 GB, 102 GiB) copied, 6012.4 s, 18.2 MB/s

This has the effect of decompressing the file into a memory buffer, passing it through the dd utility which counts the amount of uncompressed data that has been extracted by bzip2 (In this case; In 1MiB blocks) before discarding the extracted data. The output (Which may take a long time to be produced) will give the decompressed size of the file in real megabytes minus one. (i.e: An output of 20301 records will be 20,302MiB in size. The presence of a +1 in the record count shows the number of incomplete records - Almost always 0 or 1 - And any number higher than one may indicate a damaged or failing storage device.)

I need to note that having had to strip out translation markers here and there (The <--T:1234-> variety) before my edits have been accepted (Not something I've ever encountered in a Wiki environment before) someone else with more experience of this may want to run the translations bot over the page so it has the opportunity to screen, re-catalogue and restore any markers that need to be there. (The usage guide for translation tags explicitly instructs users NOT to insert these markers by themselves; That the translation bot will do this as a part of its ordinary function.)

Anyhow...I hope the entire morning I've spent trying to help out here comes in useful and appreciated in this case. :-)

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 11:17, 10 June 2025 (UTC)

More non-English dumps, plus hiccup with Archive.org torrents

After being off for a spot of illness I've just uploaded the Polish Wikipedia dumps for 2025-06-01 to Archive.org and added them to this page. I also have jawiki (Japanese) and ruwiki (Russian) for the same dates coming soon, but tonights experiment with plwiki shows that transloading dumps over my 3G connection is not at all practical and really necessitates a faster, fixed-line connection.

I've also discovered a problem with the way Archive.org generates and handles the torrents it provides: If the metadata for an item is changed in any way (Even just capitalising a single letter) it re-generates the torrent. This changes the torrents infohash and magnet links, meaning torrents downloaded after the change won't connect and share with peers that downloaded and shared the torrent before the edit took place. For this reason I'm going to look at using the same Academic Torrents site that has been used for the torrents of the English Wikipedia, but this depends on a few factors. In particular; I do not have anywhere I can seed Wikipedia torrents from at present.

I'm also considering changing how I do these dumps in the future (Planned for every three months, automated if possible) to have the index and dump files in the same Archive.org collection, which makes more sense overall; Most users will want to download both in 99% of cases, so having them in the same collection makes sense and reduces collection clutter on Archive.org. Any separate torrents done for Academic Torrents will probably follow that same format, and there might be scope for a Wikimedia user policy on how such torrents should be laid out and what they should include (e.g. Index file, dump file, checksum files, anything else?).

At present I prefer torrents that are formed as;

• ./Wikipedia-??wiki-yyyyMMdd-pages-articles-multistream/??wiki-yyyyMMdd-md5sums.txt
• ./Wikipedia-??wiki-yyyyMMdd-pages-articles-multistream/??wiki-yyyyMMdd-sha1sums.txt
• ./Wikipedia-??wiki-yyyyMMdd-pages-articles-multistream/??wiki-yyyyMMdd-pages-articles-index.txt.bz2
• ./Wikipedia-??wiki-yyyyMMdd-pages-articles-multistream/??wiki-yyyyMMdd-pages-articles-multistream.xml.bz2

This arranges each torrent as a folder that alphabetically groups with other Wikipedia dumps in the same download location - Automatically sorted by language then date - Contains what's generally needed to use the pages-articles-multistream dumps, plus the checksum files taken directly from the Wikimedia dump tree. This arrangement also puts the shortest files at the head of the torrent and larger files toward the end.

Any comments or critique on the above would be greatly appreciated. :-)

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 03:01, 23 June 2025 (UTC)

Just to update the above in adding that...

• New torrents with webseeds¹ have been generated and uploaded to Academic Torrents,
• ¹ To avoid extra data burdens for Wikimedia these employ official and known stable Wikimedia mirrors for their web-seeds, with the copies I uploaded to Archive.org employed as a lasting fallback. As these should remain reliable well past relevance, they have been marked as web-seeded torrents.
• The torrent files and Academic Torrent listings have been provided with appropriate usage, license information and links in the torrent comment fields, in the language of the torrent subject matter and UK English.
• I intend to add directly accessible (i.e: Obtainable without leaving this domain) copies of the torrents I've generated to the Wikipedia data dump torrent archive to cover for edge cases where Academic Torrents (Or any other domains conforming to *torrent*) is inaccessible to the user, perhaps owing to ISP or network applied restrictions.

ᛒᚱᛟᚴᛂᚾ ᚢᛁᚴᛁᚾᚷ (Broken Viking|T·C) 01:27, 25 June 2025 (UTC)