WMDE Technical Wishes/Move files to Commons

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

Correctly move files to Wikimedia Commons from other wikis[edit]

Logo for the beta feature FileExporter.svg

The FileImporter and FileExporter implement a new functionality that allows moving files with all their original data intact, while documenting the move in the version history.


Background[edit]

The wish[edit]

Being able to move files from Wikipedia to Commons including the version history and information about who moved the file was a wish from the 2013 survey of the German-speaking communities:

Files in Wikimedia Commons can be used by all wikis. That’s why files from local wikis should be transferred to the central repository if their license allows it. But the current transfer process has limits: The version history page isn’t kept on Commons, e.g. it doesn’t show who originally uploaded the file. Further, former revisions of the file have to be added separately to the file page.

Situation before[edit]

There were several tools that helped users move files from local wikis to Commons. But keeping all file information intact could not be done by tools, as this has to be programmed in the backend. This concerned the history of the file and of the file page, including information about who originally uploaded the file and who moved the file to Commons. As a workaround, people added this information manually, e.g. with the {{Original Upload Log}}.

How it works[edit]

Information

Two prerequisites for using the feature on your wiki:

  1. You need to activate the beta feature in your preferences.
  2. There needs to be a configuration file for your wiki.

Getting started[edit]

Activate the beta feature[edit]

Beta experiments icon - transparent.png

The functionality is currently available as a beta feature. This means: Interested users can opt-in to use it before it becomes a default feature. Let us know what you think to help us fine-tune it.

If you want to activate the beta feature, check “FileExporter” in your preferences on your wiki. You don’t need to activate anything on Commons.

Configure the feature for your wiki[edit]

Each wiki needs a configuration file that defines

  • which templates a file must have in order to be exported to Wikimedia Commons,
  • which templates in a file should block a transfer to Wikimedia Commons,
  • how information in the file should be replaced during import (e.g. templates and headings),
  • and more.

These configuration files are maintained by the communities. Many files already exist, based on the files from the CommonsHelper2 tool. We're asking users to have a look at them and update or create them if needed to make sure the imports are done right.

For more info, read the documentation how the configuration files work and what they are needed for.

Workflow[edit]

You need to be logged in in order to use the feature.

Limitations[edit]

The focus of this functionality is on moving local files to Commons correctly in a technical way. What the functionality doesn’t do is:

  • automatically move files to Commons
  • move several files to Commons at the same time
  • match categories on local wikis with Commons categories
  • replace the necessary legal check regarding correct licenses etc.

User rights[edit]

Action required right on Commons required right in the source wiki
Importing files right to upload files (logged-in user) auto-confirmed user
Reverting file imports on Commons admin
(because this action is equivalent to file deletion)
/
Deleting files from the source wiki / admin rights

Without admin rights, you can add a template like {{nowCommons}}) to the local file during the import so that the wiki can do whichever cleanup process they decided on.

Technical implementation[edit]

  • The functionality contains two MediaWiki extensions:
    • The FileExporter provides a link to Commons on the local wiki and hands over an URL to the extension FileImporter.
    • The FileImporter is a Commons extension which imports the file including all data when all prerequisites are met.
  • Technically, the files aren't "moved", but "copied".
  • JavaScript: The feature works without JavaScript. JavaScript elements might be added in a later version so that file name and file info can be changed on the same page.

Development and deployment[edit]

  • To understand the requirements and the scope of the wish, several conversations with users took place.
  • The team presented the planned workflow for this feature here on the project page in March 2017. First mocks were presented in July 2017.
  • In March 2018, Commons users were asked if they consider it necessary to keep the Original Upload Log when using the new feature.
  • Since April 2018, users can test an alpha version of the feature.
  • In June 2018, a beta version of the feature was deployed to first wikis in order to test it in a smaller group first before making it a beta feature everywhere. This included, as always, one left-to-right wiki. With this small beta approach, we could take care of issues before rolling it out as a beta on all wikis. For example, we've added some features and fixed some bugs during the small beta phase.
  • In January 2019, a beta version of the feature was released to all remaining wikis, inviting everyone test and share their thoughts. This full beta feature already gave a good idea of what the default feature would look like in the future, even though development wasn't fully finished yet. The purpose of the beta phase is to find out how this feature works for anyone who might want to use it in the future.
FileExporter FileImporter
2018-06-12

Files from mediawiki.org and test2.wikipedia.org can be exported to test.wikipedia.org.

beta feature on mediawiki.org and test2.wikipedia.org

For this deployment, the link on the file page is labelled "Export to Test Wiki". To test the export on these wikis, you need to be an auto-confirmed user.

test.wikipedia.org
2018-06-25 beta feature on German, Persian and Arabic Wikipedia (small beta deployment) Wikimedia Commons
2018-07-11 beta feature on sourcewiki.org
2018-09-17 beta feature on Korean Wikipedia
2018-10-08 beta feature on Marathi Wikipedia
2019-01-16 beta feature on other wikis (full beta deployment) What's new in this release – some examples:
  • Log entries and tags are added to file imports. (read more)
  • The matching and replacement of templates was extended. (e.g. T194505, T198317, T198711)
  • If no config file exists, files can't be moved. (T199108)
  • If a user's file is moved to Commons, their Commons account is auto-created (T209346)
  • Bug fixes and smaller changes were made (e.g. T198321, T198681, T194642)
April - July 2019 several improvements. The feature now
  • shows on the import page which categories will be used
  • makes it easier to declare on the source file that is was moved to Commons
  • allows deletion of the file on the source wiki during import (requires admin rights on the source wiki)
  • can match different spellings for the same parameter (T213955)
  • use language typing on template parameters if requested in the configuration (T198607)
  • has interwiki links working with all Wikimedia projects

and more

August 2019 planned deployment of the FileExporter as a default feature to first wikis

Links[edit]

FAQ[edit]

Are all revisions imported, or only the latest revision of a file?[edit]

All revisions are imported. (example)

Files with deleted/suppressed revisions can't be imported, see #Can I export deleted files?.

Can I export deleted files?[edit]

The FileExporter can't import

It can import files with suppressed parts. In this case, the suppressed parts are hidden. Examples:

How can I find files that were imported with FileExporter?[edit]

  • In the revision history of a file, the import is shown as a null revision (i.e. an entry in the revision list that doesn’t actually change the page).
  • For the import, an upload log entry and an import log entry are created and associated with the null revision and with the importing user, and timestamped with the import date. Having both the null revision and the upload log entry is the default for uploads, although it leads to "doubled" entries on the history page, see this example file and its log entries.
  • For (old) imported file revisions, upload log entries are created with the date and user of the original upload to make all file revisions appear in Commons' upload log.
  • Imports are tagged with the tag fileimporter, so you can filter them in Recent Changes.

This is true for files that were exported after January 16, 2019. Files that have been imported with FileExporter before this date don't have the tag or the log entries.

What if I want to import a file to Commons that has been imported before (without the history)?[edit]

The decision if you import an already imported file again or not is up to you. The advantage: You don't lose the file and version history and have more transparency about who uploaded the file or edited the file description page. How to do this:

  • 1. If necessary undelete the local file in the wiki
  • 2. Delete the already imported file on Wikimedia Commons
  • 3. Import the file with FileExporter.
  • 4. Delete the local version.

Attention: Perform steps 2 and 3 in a timeframe of 10 minutes to avoid that the User:CommonsDelinker starts it work and removes the file usages from the wiki pages.

If the file page of the old imported file on Commons has been changed in the meantime, preserving both version histories can be tricky. You can try this:

  • 5. Undelete the file that you have deleted in step 1, to merge the version histories.
  • 6. Restore the file description on Commons and refine it.

This requires some manual work (example). It's up to you if you want to do this or not. There's a request on Phabricator for a follow-up project to build a tool that makes such imports easier.

There's a license missing in my wiki's configuration file. What do I do?[edit]

Each wiki is responsible for their own configuration file. If you think your wiki's file is incomplete or wrong, it's recommended that you discuss it within your community and then change the configuration page.

What do I have to do after the import?[edit]

This depends on the processes in your local wiki. One possibility is to make sure the old file name is not used any more, and then delete it. FileImporter supports this cleanup process by adding templates like NowCommons to the source file in your name, if you want it, and by allowing you to delete the source file during import, if you're an admin on the source wiki.

I'm an admin in my local wiki and I'd like to delete the original file directly from Commons. Can I do this?[edit]

Yes, that’s possible. As an admin you can choose during the import if you want to have the file automatically removed in your name after the import or not.

I want the feature to be in my wiki's language, for example the name and description on Special:Preferences. How do I do that?[edit]

Texts both for FileExporter and FileImporter can be translated. Any translation is very much appreciated!

Can I use the FileExporter to export files to other Wikimedia wikis than Commons?[edit]

No, this is currently not possible. Requests on Phabricator:

Can files be moved back to the local wiki?[edit]

There's a manual way to do this:

  • 1. Delete the file on Commons. This requires admin rights on Commons.
  • 2. Restore the original file on the local wiki. This requires admin rights on the local wiki.

There's a request on Phabricator to make this a built-in feature.

My source template has more parameters than the template on Commons. How can I transfer their contents?[edit]

Example: The Information template on English Wikipedia has the parameters "Location" and "Additional_information" that the Information template on Commons doesn’t have.

Additional parameters are imported to the new file description page on Commons, but they are only shown in wikitext, not on the rendered page. You can manually put this wikitext into the right position in the Commons template.

There’s a request on Phabricator for a built-in feature which merges additional parameters from a source wiki into a Commons parameter.

My wiki allows different spellings for the same parameter. How can I make sure they both get replaced by the same Commons parameter?[edit]

FileImporter can now match different spelling variants of a parameter if they have been defined in the configuration file.

Example: The Russian Information template allows the same parameter to be spelled |Описание= or |описание= (see configuration file). FileImporter matches both to |Description=.

How can I add categories during import?[edit]

Categories on local wikis aren’t the same as on Commons. FileImporter does not match categories during import, so you have to manually change category names from your local wiki into the corresponding Commons categories.

The FileImporter does show which categories will be imported. Missing categories are shown as red links, giving an indication which categories need to be changed. To change them, you have to

  • click on “edit file info”,
  • change the categories in the wikitext.

Feedback[edit]

Feedback is always welcome and needed to improve the feature! Please use the central feedback page on MediaWiki.org, in case you have any questions, comments, advice ... Thanks :-)