WMDE Technical Wishes/Move files to Commons
Correctly move files to Wikimedia Commons from other wikis[edit]
The FileImporter and FileExporter implement a new functionality that allows moving files with all their original data intact, while documenting the move in the version history.
| Background |
|---|
BackgroundThe wishBeing able to move files from Wikipedia to Commons including the version history and information about who moved the file was a wish from the 2013 survey of the German-speaking communities: Files in Wikimedia Commons can be used by all wikis. That’s why files from local wikis should be transferred to the central repository if their license allows it. But the current transfer process has limits: The version history page isn’t kept on Commons, e.g. it doesn’t show who originally uploaded the file. Further, former revisions of the file have to be added separately to the file page. Situation beforeThere were several tools that helped users move files from local wikis to Commons. But keeping all file information intact could not be done by tools, as this has to be programmed in the backend. This concerned the history of the file and of the file page, including information about who originally uploaded the file and who moved the file to Commons. As a workaround, people added this information manually, e.g. with the {{Original Upload Log}}. |
How it works[edit]
FileExporter is now a default feature on all wikis.
 | === Configure the feature for your wiki ===
Each wiki needs a configuration file that defines
and more. These configuration files are maintained by the communities. Many files already exist, based on the files from the CommonsHelper2 tool. We're asking users to have a look at them and update or create them if needed to make sure the imports are done right. For more info, read the documentation how the configuration files work and what they are needed for. |
Custom categories can be calculated from the source URL or timestamp. If a custom message is provided for MediaWiki:fileimporter-post-import-revision-annotation, it will be included in the imported file info, with message parameter $1 set to the full URL of the source file, and $2 set to an ISO 8601 timestamp for when the import began. For example, the message "{{#invoke:SourceCategory|urlToDomainCategory|url=$1}}{{#invoke:MonthCategory|main|date=$2}}" might be substituted into the file info as "{{#invoke:SourceCategory|urlToDomainCategory|url=http://dev.wiki/wiki/File:Mittens.jpeg}}{{#invoke:MonthCategory|main|date=2020-07-06T11:21:49+00:00}}".
Workflow[edit]
You need to be logged in in order to use the feature.
-
1: On a local file page, click on the "Export to Wikimedia Commons" link. -
2: The FileImporter checks if the file can be imported and if replacements need to be made during the import. This step is based on the wiki's configuration file. Read more. -
3: A preview page on Commons shows you what will be imported. You can edit the file title and page information, and ask the FileImporter to clean up the source wiki in your name, by adding a template like NowCommons to the source file, or deleting the source file if you have admin rights. Clicking on “Import File” copies the file to Commons. -
4: The file is imported, including the name of the original uploader and name of the importer, the previous history of the file and the file page, and any changes that were made during import.
Limitations[edit]
The focus of this functionality is on moving local files to Commons correctly in a technical way. What the functionality doesn’t do is:
- automatically move files to Commons
- move several files to Commons at the same time
- match categories on local wikis with Commons categories
- replace the necessary legal check regarding correct licenses etc.
User rights[edit]
| Action | required right on Commons | required right in the source wiki |
|---|---|---|
| Importing files | right to upload files (logged-in user) | auto-confirmed user |
| Reverting file imports on Commons | admin (because this action is equivalent to file deletion) |
/ |
| Deleting files from the source wiki | / | admin rights
Without admin rights, you can add a template like {{nowCommons}}) to the local file during the import so that the wiki can do whichever cleanup process they decided on. |
Technical implementation[edit]
- The functionality contains two MediaWiki extensions:
- The FileExporter provides a link to Commons on the local wiki and hands over an URL to the extension FileImporter.
- The FileImporter is a Commons extension which imports the file including all data when all prerequisites are met.
- Technically, the files aren't "moved", but "copied".
- JavaScript: The feature works without JavaScript. JavaScript elements might be added in a later version so that file name and file info can be changed on the same page.
Development[edit]
- To understand the requirements and the scope of the wish, several conversations with users took place.
- The team presented the planned workflow for this feature here on the project page in March 2017. First mocks were presented in July 2017.
- In March 2018, Commons users were asked if they consider it necessary to keep the Original Upload Log when using the new feature.
- Since April 2018, users can test an alpha version of the feature.
- In June 2018, a beta version of the feature was deployed to first wikis in order to test it in a smaller group first before making it a beta feature everywhere. This included, as always, one left-to-right wiki. With this small beta approach, we could take care of issues before rolling it out as a beta on all wikis. For example, we've added some features and fixed some bugs during the small beta phase.
- In January 2019, a beta version of the feature was released to all remaining wikis, inviting everyone test and share their thoughts. This full beta feature already gave a good idea of what the default feature would look like in the future, even though development wasn't fully finished yet. The purpose of the beta phase is to find out how this feature works for anyone who might want to use it in the future.
- Since September 2019, the FileExporter is a default feature on a few first wikis. Deployment on more wikis is planned for later this year.
Deployment roadmap[edit]
| FileExporter | FileImporter | |
|---|---|---|
| 2018-06-12
Files from mediawiki.org and test2.wikipedia.org can be exported to test.wikipedia.org. |
beta feature on mediawiki.org and test2.wikipedia.org
For this deployment, the link on the file page is labelled "Export to Test Wiki". To test the export on these wikis, you need to be an auto-confirmed user. |
test.wikipedia.org |
| 2018-06-25 | beta feature on German, Farsi and Arabic Wikipedia (small beta deployment) | Wikimedia Commons |
| 2018-07-11 | beta feature on sourceswiki (Wikisource for multilingual books) | |
| 2018-09-17 | beta feature on Korean Wikipedia | |
| 2018-10-08 | beta feature on Marathi Wikipedia | |
| 2019-01-16 | beta feature on other wikis (full beta deployment) | What's new in this release – some examples:
|
| April - July 2019 | several improvements. The feature now
and more | |
| September 24, 2019 | default feature on first wikis:
|
Some examples of what’s new:
plus some bugfixes. |
| August 5, 2020 | deployment of the FileExporter as a default feature on all remaining wikis | 'Now Commons' template can be added to the file on the source Wiki by clicking the corresponding check box on the import page. |
Links[edit]
- the project on Phabricator
- the central help page on Mediawiki.org
- the central feedback page on Mediawiki.org
- the FileExporter and the FileImporter extension manuals on Mediawiki.org
- Information on the current transfer process, infos about which local files shouldn't be moved to Commons and the tools that currently can be used to move files to Commons (on English Wikipedia, on German Wikipedia)
FAQ[edit]
Are all revisions imported, or only the latest revision of a file?[edit]
All revisions are imported. (example)
Files with hidden file versions can't be imported, see #Can I export deleted files?.
Can I export deleted files?[edit]
The FileExporter can't import
- deleted files
- files with hidden (revision deleted) file versions (example file)
It can import files with other hidden parts. In this case, the following hidden parts are replaced:
- Hidden page text (example file)
- Hidden edit summary (example file)
- Hidden user / IP (example file)
It can import partially undeleted files. In this case, the deleted revisions/file versions (not marked as hidden nor otherwise listed in page/file history) are not imported.
How can I find files that were imported with FileExporter?[edit]
- In the revision history of a file, the import is shown as a null revision (i.e. an entry in the revision list that doesn’t actually change the page). If changes were made during import, an additional text revision is created.
- The revision that is created during import is tagged with
Modified with FileImporter. This way it can be filtered on the Recent Changes page. - For the import, an upload log entry and an import log entry are created and associated with the null revision and with the importing user, and timestamped with the import date. Having both the null revision and the upload log entry is the default for uploads, although it leads to "doubled" entries on the history page, see this example file and its log entries.
- For (old) imported file revisions, upload log entries are created with the date and user of the original upload to make all file revisions appear in Commons' upload log. These revisions are also tagged with
Imported with FileImporter. (an example file from the beta cluster)
This is true for files that were exported after January 16, 2019. Files that have been imported with FileExporter before this date don't have the tag or the log entries.
What if I want to import a file to Commons that has been imported before (without the history)?[edit]
The decision if you import an already imported file again or not is up to you. The advantage: You don't lose the file and version history and have more transparency about who uploaded the file or edited the file description page. How to do this:
- 1. If necessary undelete the local file in the wiki
- 2. Delete the already imported file on Wikimedia Commons
- 3. Import the file with FileExporter.
- 4. Delete the local version.
Attention: Perform steps 2 and 3 in a timeframe of 10 minutes to avoid that the User:CommonsDelinker starts it work and removes the file usages from the wiki pages.
If the file page of the old imported file on Commons has been changed in the meantime, preserving both version histories can be tricky. You can try this:
- 5. Undelete the file that you have deleted in step 1, to merge the version histories.
- 6. Restore the file description on Commons and refine it.
This requires some manual work (example). It's up to you if you want to do this or not. There's a request on Phabricator for a follow-up project to build a tool that makes such imports easier.
There's a license missing in my wiki's configuration file. What do I do?[edit]
Each wiki is responsible for their own configuration file. If you think your wiki's file is incomplete or wrong, it's recommended that you discuss it within your community and then change the configuration page.
What do I have to do after the import?[edit]
This depends on the processes in your local wiki. One possibility is to make sure the old file name is not used any more, and then delete it. FileImporter supports this cleanup process by adding templates like NowCommons to the source file in your name, if you want it, and by allowing you to delete the source file during import, if you're an admin on the source wiki.
I'm an admin in my local wiki and I'd like to delete the original file directly from Commons. Can I do this?[edit]
Yes, that’s possible. As an admin you can choose during the import if you want to have the file automatically removed in your name after the import or not.
I want the feature to be in my wiki's language, for example the name and description on Special:Preferences. How do I do that?[edit]
Texts both for FileExporter and FileImporter can be translated. Any translation is very much appreciated!
- FileExporter on Translatewiki: https://translatewiki.net/wiki/Special:Translate/ext-fileexporter
- FileImporter on Translatewiki: https://translatewiki.net/wiki/Special:Translate/ext-fileimporter
Can I use the FileExporter to export files to other Wikimedia wikis than Commons?[edit]
No, this is currently not possible. Requests on Phabricator:
Can files be moved back to the local wiki?[edit]
There's a manual way to do this:
- 1. Delete the file on Commons. This requires admin rights on Commons.
- 2. Restore the original file on the local wiki. This requires admin rights on the local wiki.
There's a request on Phabricator to make this a built-in feature.
My source template has more parameters than the template on Commons. How can I transfer their contents?[edit]
Example: The Information template on English Wikipedia has the parameters "Location" and "Additional_information" that the Information template on Commons doesn’t have.
Additional parameters are imported to the new file description page on Commons, but they are only shown in wikitext, not on the rendered page. You can manually put this wikitext into the right position in the Commons template.
There’s a request on Phabricator for a built-in feature which merges additional parameters from a source wiki into a Commons parameter.
My wiki allows different spellings for the same parameter. How can I make sure they both get replaced by the same Commons parameter?[edit]
FileImporter can now match different spelling variants of a parameter if they have been defined in the configuration file.
Example: The Russian Information template allows the same parameter to be spelled |Описание= or |описание= (see configuration file). FileImporter matches both to |Description=.
How can I add categories during import?[edit]
Categories on local wikis aren’t the same as on Commons. FileImporter does not match categories during import, so you have to manually change category names from your local wiki into the corresponding Commons categories.
The FileImporter does show which categories will be imported. Missing categories are shown as red links, giving an indication which categories need to be changed. To change them, you have to
- click on “edit file info”,
- change the categories in the wikitext.
Feedback[edit]
Feedback is always welcome and needed to improve the feature! Please use the central feedback page on MediaWiki.org, in case you have any questions, comments, advice ... Thanks :-)