Static content group
This is a group and set of projects dedicated to gathering and sharing static content from Wikimedia projects, including CDs and DVDs, single-file databases for use with specific browsers and readers, and pdf and html exports.
- 1 Background
- 2 Goals
- 3 Status
- 4 Food for thought
- 5 Why offline matters
- 6 Projects that involve Wikipedia Offline
- 7 User Feedback
- 8 Features & Tech specifications
- 9 Get involved
- 10 Get in touch
- 11 References
- 12 Guidelines and coordination
- 13 Updates and recommendations
- 14 See also
In 2004, the first Wikipedia CD was released, and the concept of WikiReaders for offline printing and reading was refined. In 2006, the Special Projects Committee authorized the creation of a subcommittee dedicated to static snapshots of Wikimedia content, to identify the groups working on such projects, and to help them work effectively together and share their results. This page is derived from those efforts and the work of all who have pursued similar goals.
- content quality/vetting for "Wikipedia 1.0"
- content on paper
- topical subsets
- selections for different audiences
- Maintain, offer and assist interested parties in getting
- static content of Wikimedia projects in a variety of formats
- metadata about its completeness, complexity/age level, and quality.
- Coordinate efforts to produce and distribute snapshots, and software for viewing snapshots.
- Stimulate research about content and making content accessible in an offline or semi-online environment.
As of mid-2006, the Wikimedia Foundation served to the world outdated version of static content of Wikipedias (November 2005 snapshot - ) and offered the same content for download (). There seeem to be persistent small problems with static content installation (categories,search, perhaps other).
There were plans to set up an up-to-date server with current static content of Wikipedias.
MediaWiki 1.5 included routines to dump a wiki to HTML, rendering the HTML with the same parser used on a live wiki.
There have been several separate attempts at producing software to convert SQL dumps into data formats that are suitable. Directmedia Publishing GmbH in Germany had by then a successful history of distributing Wikipedia content on DVD for Windows and MacOSX (Linux is beta).
Needs updating since 2007!
See List of Offline Projects for updated list of offline projects in the community.
Food for thought
- gateway to Wikipedia content publication on WP-De
- PDA-friendly content hosted by wikipedia:Axel Schäfer, member of parliament ( and  )
- /Germany - Introduction to the current situation of "Static content in Germany and/or in Germany"
- En: Organised by the w:Wikipedia:Version 1.0 Editorial Team with help from a bot. Release of w:Wikipedia:Version 0.5 (a test CD) is planned for late 2006.
- En: Earlier in 2006 the UK children's charity SOS Children released "A World of Learning" containing 2200 articles suitable for kids. They are now working here to expand the selection.
- Polish Wikipedia DVD: Major release planned for autumn 2006 by Wikimedia Polska in collaboration with Helion SA. See also this report in English.
- French Wikipedia: Work beginning on software, now collaborating with en and po.
- En/Fr Mandriva project: Wikimedia and Mandriva (very little activity)
- No: See no:Wikipedia:Wikipedia_1.0 (put on hold).
- Sv: See sv:Wikipedia:Wikipedia_1.0 (little activity).
- Bn: See bn:উইকিপেডিয়া:উইকিপ্রকল্প সিডি প্রকাশনা (Target: A 2000 article CD by the end of 2007)
Other project languages : Chinese? Dutch? Italian? Russian?
Other languages as of 2013
The following languages are working on creating offline files for some subset of the Wikimedia projects.
|Language||Project type||Project name||Project status|
|English||General collection||Wikipedia 1.0||0.8 completed; waiting for 1.0 update 2011|
|English||Schools collection||Wikipedia for Schools||2008-09 version completed; waiting for the updated 2011|
|English||Schools collection||Offline Wikipedia for Indian Schools||under progress|
|Spanish||General collection||Wikipedia 1.0||Est. August release date; for articles see ToolServer|
|Portuguese||Schools Collection||Ongoing! Project Page; List of articles|
|Malayalam (Wikipedia)||General collection of selected articles||Wikipedia 1.0||Released on 2010 April 17|
|Malayalam (Wikisource)||General collection of selected books||Wikisource 1.0||Released on 2011 June 11|
|Malayalam (Wikisource)||General collection of selected books||Wikisource 2.0||Released on 2013 Coctober 14|
In addition to these aggregate selections, there is also need and room for the selection of niche collections: e.g., the creation of articles regarding to Chemistry, India, etc. Some of these are accessible by the independent creation of books via the PediaPress book extension tool on Wikipedia. These creations are independent projects and can be found at Wikipedia:Books.
Since before 2004:
- The en:Webaroo Wikipedia and related packs
- OEPC : en:Wikipedia:One_Encyclopedia_Per_Child (simple, laptopwiki)
Formats and readers
Distribution formats: plain (x)HTML, PDF, TomeRaider, Plucker, Webaroo pack, proprietary formats
- Directmedia (free, if not open source, on Linux and Mac platforms - Linux version (digibux) is GPL)
- Gearswiki and google gears efforts
- Other (See below)
Kiwix brings internet contents to people without internet access. It is free as in beer and as in speech.
As an offline reader, it is especially thought to make Wikipedia available offline, but technically any kind of web content can be stored into a ZIM file (a highly compressed open format) and then read by the app: there are currently several hundred different contents available in more than 100 languages, from Wikipedia, Wikiquote, the Wiktionary to TED conferences or the Gutenberg library.
Why offline matters
We're featuring a quote here from the UN Broadband Commission from their September 2013 report, because it's the easiest, most pragmatic and straight-forward way to show you the importance of disseminating knowledge - and information - offline, complementary to all activities that we do online: "While more and more people are coming online, over 90% of people in the world’s 49 Least Developed Countries remain totally unconnected.”
Projects that involve Wikipedia Offline
Kiwix is mostly installed in schools that cannot afford broadband internet access. In these cases, it's so much faster to use Wikipedia offline.
Wikimed is a free android app that collates all medicin-related articles form Wikipedia and makes them available offline. It is currently available in English, French, Arabic, Farsi, German, Spanish, Chinese, Odia, and Portuguese. You can download it here.
Wikipedia offline in jails
- Main article: Wikipedia in jails
Since March 2013, prisoners in the prison Bellevue in Gorgier (western Switzerland) who request it can have access to Wikipedia offline, because Swiss prisoners have very restricted access to the Internet. The idea is to stimulate or to support the interest for education of prisoners who were, for a large majority, condemned to long-time sentences. After a three month pilot phase, the project was proven very successful. Among the 36 prisoners of the Bellevue’s prison in Gorgier, 18 possess or rent a computer. All of them requested the upload of Wikipedia offline on their PC.
The feedback is unanimously positive: it reveals that access to Wikipedia is seen as an improvement of education and/or information activities in jail.
The followup of the project aims to use Wikipedia in the training program of the prisoners. The use of Wikipedia in the classes, the organization of general culture contests, and even the training of new Wikipedia editors. The partnership between Wikimedia CH and the direction of the prison aims to be durable. Wikimedia CH installed the Kiwix files and trained the IT team of the prison, who can now upload the software for every new prisoner who requests. Detention Centers for minors are excluded from this program in Switzerland as they get access to the Internet and don't have the need to access Wikipedia offline.
In 2014, WMCH started to collaborate with the Swiss Insitute for Education in Detention Centers to expand the coverage of Wikipedia offline in prisons all over Switzerland. As of May 2014, all prisons in the German-speaking part of Switzerland have access to Wikipedia offline, thanks to the Swiss Institute for Education in Detention Centers.
Canada, Germany, the US, France, Belgium and Italy (jail in Pavia, where a Kiwix server runs in a dedicated computer room, led by http://www.informaticisenzafrontiere.org) also have similar projects in prisons that involve Wikipedia offline.
Wikipedia for Schools
"At SOS Children, we wanted to bring this fantastic resource to children without internet access around the globe. So we began work on an ambitious project to get the very best content from Wikipedia into a self-contained selection which could be distributed on a CD. We checked every article for child friendliness and structured the content around the national curriculum. Today, Wikipedia for Schools is in its fourth incarnation, and the new version is ready to go - this time on USB. At EduWiki 2013, we will show you how the project has benefited students and teachers here in the UK, and in countries across the developing world. With the help of others, we have distributed copies globally, and we have had an amazing response from the people who count. In the UK, Wikipedia for Schools has been a great classroom companion for students and teachers alike.” 
Mesh Sayada is a collaboratively designed and built wireless network. The town of Sayada is located in Tunisia. The network serves as a platform for locally-hosted content, such as Wikipedia Offline in Arabic and French thanks to Kiwix software, free ebooks and Open Street Maps. The Mesh is serviced and maintained by a local NGO, CLibre with the help of local volunteers.
- "Very important and helpful source of information" (User from Bahrain)
- "Thank you for your help! Now my school can use Wikipedia offline."' (User from Mexico)
- "I like to browse my favorite encyclopedia even when there is no network" (User from Yemen)
- "I have no internet in my house. Kiwix is such a help, because I need Wikipedia for my study."' (User from Cuba)
Features & Tech specifications
- Open-source: all code is stored on GitHub;
- Available on Windows, GNU/Linux, macOS, iOS and Android;
- Works like your a regular browser;
- Allows searching of articles and within articles;
- Web Server: you can share content on your LAN.
There are many ways to participate and to work with us in order to develop the Kiwix - Wikipedia offline project. The following list features many topics where help would really be appreciated:
- Translate: the Kiwix user interface is translated into more than 100 languages. We still have some more work to do here: see translatewiki:Translating:Kiwix to help.
- Share: Kiwix has a broad user community - we need to care for it and share news. See below to become an ambassador;
- Deploy: if you want to deploy Kiwix anywhere, let us know! You can also file for a [Grant] with the Foundation.
- Develop: if you are a coder, feel free to join us on GitHub - /OpenZim for scrapers, and /Kiwix for the app itself. Look for tickets labeled "Good First Issue" (to get started) or "Help wanted" (for real challenges).
Become an ambassador
As an ambassador, you are going to spread the word about Kiwix in different ways:
- Mention Kiwix when talking about Wikimedia: e.g., add a slide about Kiwix when making a public presentation about Wikipedia. If you are interested in giving a talk at a meet-up or conference or organizing a Kiwix event, Stephane (Kiwix) can provide you with slides, flyers and other material you might find useful. Don't forget to add the Category:Kiwix presentations at your uploaded file on Commons.
- Represent Kiwix in conferences and workshops.
- Answer questions about the project within your community.
- Use social media to get in touch with users.
- Help implementing mirror websites for Kiwix in different countries to decrease downloading time. This can be done by contacting servers owners (e.g. Universities, Telecom operators, ..). (See existing mirror websites list). Don't forget to include technical instructions.
- Ask your language community to add a link in the sidebar of the wiki inviting readers to download the content throw Kiwix. (See example).
Official country and language ambassadors
The Ambassador program lists people who are familiar with both Kiwix and a language-related project: they will try to assist with questions and requests for presentations. An ambassador is a trusted volunteer that has been vetted by existing ambassadors and/or their local chapter. Contact them if you need help!
- Countries and languages
- Algeria : bachounda (ar; fr; en)
- Belgium : Lionel Scheepmans (fr, pt, en)
- Canada : Helmoony (ar; fr; en)
- Cameroon : Yanstv (fr; en)
- Dem Rep of the Congo: BamLifa (en; fr)
- Ghana : Flixtey (en)
- Jordan/Levant: مصعب (ar; en)
- Mali : rgaudin (fr; en; es; bm)
- Nigeria : Blossom Ozurumba (en; ig)
- Southern Africa : Michaelgraaf (en; af; zu)
- Switzerland: Stephane (Kiwix) (fr; en; es; de) ; Kelson (fr; en; de)
- Tunisia : Dyolf77 (ar; fr)
- Russia : Reda Kerbouche (ar; fr; ru; en)
Are you a Kiwix user? Do you want to help? Contact us!
Get in touch
- http://www.broadbandcommission.org/Documents/bb-annualreport2013.pdf Annual UN Broadband Commission Report 2013
- https://wiki.wikimedia.org.uk/wiki/EduWiki_Conference_2013/Abstracts#Workshops by Jamie Goodland, who works with the international children’s charity SOS Children
- Case Study: Mesh Sayada by Ryan Gerety, Andy Gunn and Will Hawkins Open Technology Institute
- Association pour la culture numérique Libre
- Tero-dump wikipedia to html converter
- Convert Wikipedia's SQL dump to static HTML for local installation only a prototype, very limited, but usable
- Wikimedia content for Sharp Zaurus
- TomeRaider and TomeRaider database
- Wikipedia in ipodlinux.org
- Static HTML tree dumps for mirroring or CD distribution
- Dynamic HTML generation from a local XML database dump
Initial members of the subcommittee:
Other interested people:
- Eric Astor (working on OEPC)
- Erik Garrison (working on related statistics)
- Eyu100 (asked the get involved)
Guidelines and coordination
Style guidelines for each project should be written down, for the benefit of projects to come after them. Coordination across projects of aspects such as script writing can also be quite helpful -- in catching mistakes and corner cases, and in avoiding repeated effort. Some specific ideas follow.
Registering a new snapshot
Needed : a process for announcing and registering a snapshot for others to see in progress, contribute to, or download and use. Start with finished projects to date in German, English, and Polish.
- En:wp: SOS CD project; Andrew wrote some scripts for this. 2006 articles on a CD.
- Initially distributed to benefit a children's charity in early 2006
- Wikiwizzy's version of the above for distribution in S.Africa
- De:wp: Directmedia CD, then DVD.
- Pl:wp: ?? DVD, planned for completion in October (slightly new deadline).
- How should snapshots recognize authors? What's the best way to attribute WP as a project as well as individuals, not simply to satisfy the GFDL?
- How can snapshots share algorithms? Part of snapshot design is choosing content.
- How can snapshots get updated? Scripting the creation so they don't get stale; minimizing editorial time needed.
- Style : different ways to handle
- templates (navigational, other)
- foul language
- images (size, content)
- text (when too long, for balance)
- interlanguage, interwiki links
- Enhancing content : how to handle
- delicate subjects (warning templates)
- conflicted subjects (pov templates)
- fast-changing subjects (news / current event templates)
- (note: all this can update the dynamic database directly)
- Interactivity and interfaces : Front-ends to read and interact with different snapshot formats.
- Reducing text : summarizing, auto-excerpting
- Ranking text : bot-assisted reviewing/vetting/rating, metric analysis (apsp, grank, hit-popularity, edit-popularity, expertise, writing style, &c)
- Metadata : bot-assisted annotation (audience, type, categorization)
- Spellchecker, grammar checker
- Copyvio checker
- Image resizing & compression
- Metadata extraction
- History metadata (list of users, freshness, &c)
- Image/media metadata
- Index generation (for browsing)
- Category tree generation
General steps to completion
Just as for the online resource, the offline versions of Wikimedia content are only as strong as their contents. The main constraint with an offline product is the data size restrictions: the entirety of Wikipedia (and/or other Wikimedia projects) must somehow be condensed so that it fits on a CD, DVD, or USB stick which then has to be housed on a computer, mobile phone, or e-reader. The goal of carefully selecting content, then, is to provide the best and most appropriate static content.
The goal is:
|To provide a library of the highest quality and most relevant offline content collections.|
This implies having content options most relevant for the different offline audiences. Many factors must be taken into consideration, including
- Article complexity
Want to create your own custom offline collection? See:
Created in partnership with PediaPress, the Wikipedia Book Creator enables users to easily select specific articles from Wikipedia and create a book. This "book" can then be downloaded for free in PDF or openZIM format, or for a fee the content can be ordered as a physical book.
Content selection for schools
The Schools Content project page shows the list of articles included in the distribution, and provides a workspace for suggestions for adding articles.
Some articles are excluded from the collection as they are non-appropriate / essential for schools. A list of such articles intentionally excluded by OLPC can be seen here: http://mad.printf.net/blacklist2.
Updates and recommendations
- Add links to new moulin and kiwix projects, update links from local wikipedias re: 1.0 (via interwiki links). +sj | help with translation |+
- Add links to review ideas, including content stamping (since 2000) +sj | help with translation |+ 20:15, 26 February 2008 (UTC)
- set up a cron job with a script to update static content whenever content dump is done
- This actually sounds like a project for the Wikimedia Toolserver. -- Mathias Schindler 09:19, 3 June 2006 (UTC)