Community Wishlist Survey 2022/Multimedia and Commons/Mass uploader

From Meta, a Wikimedia project coordination wiki

Mass uploader

  • Problem: No Mass uploaders like Vicuna, Pattypan are functional for months, and the problems are not being addressed seriously even though links to the same are shown at the main page under Upload.
  • Proposed solution: Please revive or patch or repair the issue with Pattypan, they may not be big issues to rectify. The Foundation may have to fund a bit to take back it to working at the earliest. It seems that the people who should address the issue are not serious enough to make it right.
  • Who would benefit: All media uploaders who want to share their hard gained images and media for the world for free, without ever giving to non-free platforms.
  • More comments: See also:
  • Phabricator tickets: phab:T293543
  • Proposer: Vinayaraj (talk) 03:01, 12 January 2022 (UTC)[reply]

Discussion

Yes, that is the issue.--Vinayaraj (talk) 14:59, 12 January 2022 (UTC)[reply]
A web-based mass uploader that can process spreadsheets like Pattypan does would certainly be nice, but for the time being, I would like to see Pattypan fixed as fast as possible, because there are stuck ongoing GLAM projects that rely on it, and then think about future tools (see also my comment below). Gestumblindi (talk) 14:43, 15 January 2022 (UTC)[reply]
@Gestumblindi:A web-based mass uploader is not nice. The webpage will be killed or unresponsive if you load a number of images(which are very easy on pattypan or vicuna). Just try the UploadWizard with a 50 or some images then you can directly experience the issue. If we can develop a much more stable mass uploader then that will be good. --Ranjithsiji (talk) 05:49, 20 January 2022 (UTC)[reply]
  • Support for the requirements of an available mass uploader, e.g. for media in general. Before implementing a new uploader it might make sense to look at possible problems at the API. May be the proposal addresses a backend problem, that blocks upload clients to work as espected - e.g. that the upload clients like Commonist, Pattypan, Vicuna, ... cannot upload files due to backend modification of the API must be blocked due to a specific cause or challenge. --Bert Niehaus (talk) 10:09, 12 January 2022 (UTC)[reply]
  • I fear that putting this on a "Wishlist" implies it's just something that's nice to have or that the community would like. Bulk uploads are a central part of the partnerships Wikimedia has built up with galleries, libraries, archives, museums and other partner organisations. Often those partner organisations and the relevant Wikimedia chapter have put funding and staff time into those relationships, and had difficult conversations to persuade those partners to adopt free licensing. When bulk uploads are put on hold because the suitable tools aren't working, that halts the work of those partnerships, undermining the paid staff and the work done to build the partnership. The more months the situation goes on, the more damage is done and the less impact funders are seeing for their money. To a crucial part of the Wikimedia movement, this is a critical problem requiring an urgent fix, not something we'd like to have eventually. MartinPoulter (talk) 12:37, 12 January 2022 (UTC)[reply]
  • For context Pattypan has uploaded more that 1.1 million images to Commons, without tools like this available all my mass upload projects are on hold indefinately and partners may loose interest, same for everyone else. MusikAnimal (WMF) I have more background info on why PattyPan etc are broken if helpful. John Cummings (talk) 12:43, 12 January 2022 (UTC)[reply]
  • Its kind of disconcerting that no one has been able to fix these apps apparently. The fix is relatively simple. If no one was able to deliver these fixes, are the apps even viable at all long term any longer ? —TheDJ (talkcontribs) 12:57, 12 January 2022 (UTC)[reply]
  • I know several GLAMs who used Pattypan and now have problems with mass uploads since months. This is a very important issue from my point of view. --Hadi (talk) 15:02, 12 January 2022 (UTC)[reply]
  • This is a crisis. We need a tool for batch uploading. As of now we have a half-functioning patch that works only in Ubuntu and that I have to operate on behalf of various other Czech Wikipedians. The global GLAM seems to be paralyzed to a great degree as large amount of people simply have no way how to upload large groups of images. There are GLAM users who try to virtualize Ubuntu to be able to send images to Wikimedia Commons. 🤦‍♂️ There is only one volunteer in the Github project who simply does not have resources to fix the issue, despite a huge amount of work he did. Github shows questions from various people from all around the world asking "when the thing will start working"? This should be a priority issue and should be fixed fast. Aktron (talk) 17:40, 12 January 2022 (UTC)[reply]
  • I never liked Pattypan and Vicuna, and I hate the built-in uploader with all my heart, so please please please make something simple and really usable like Commonist work again. The Commons are dead without it. --Anvilaquarius (talk) 08:59, 13 January 2022 (UTC)[reply]
  • @Vinayaraj @Vysotsky I agree with the longing for Pattypan above, thanks Vinayaraj for sounding the alarm.
While the classic UploadWizard continues to work for small numbers of images, a Mediawiki software change should not stall excellent tools as Pattypan for more substantial uploads with varied metadata, which for instance Commonist cannot handle. (The cumbersome but very effective GlamWikiToolset is out of business as well by the change in Mediawiki software?)
At the African Studies Center of Leiden University we rely on Pattypan for a Master thesis project by a student and for longer term projects of thousands of photos by the local Wikimedian in residence, supervised by the librarian.
So please repair Pattypan! thank you, Hansmuller (talk) 14:13, 13 January 2022 (UTC) Wikimedian in residence ASC Leiden[reply]
  • The root cause of this is that a MediaWiki change broke my bot framework, which all three of these tools use a legacy version of. This was fixed three months ago but cannot be easily backported - I rewrote that part of the code to use the new HTTP client in Java 11 some time ago. The solution is to port all three tools to Java 11 (and 17) so that they can use the latest version. MER-C 20:37, 13 January 2022 (UTC)[reply]
  • There is an experimental version of Pattypan (21.10-experimental-2, "This package migrates Pattypan to Java 11+, OpenJFX, and mainstream Wiki.java.") that apparently works, but only under Linux. Using Windows, the login step seems to work, but the actual upload fails with a "GOAWAY" message from the server. As this is a tool that is widely in use by GLAMs and they are accustomed to its workflow (and often use only Windows), if it's fairly easy to fix, which I would assume it should be, it would be a better approach to first fix Pattypan fully, so that people can continue with their projects in the way they're familiar with, and then think about possible other tools. See also the discussion from here and in the sections below that. In my opinion, a quick fix that just makes the familiar Pattypan working again for now is more important than more time-consuming additional ideas/projects. Gestumblindi (talk) 12:28, 15 January 2022 (UTC)[reply]
  • Everyday uploading of media to various places like, Facebook, Instagram or say Google photos are becoming very easier. For any commoner it is more than sufficient to keep their memories to themselves or share with the world. Not many are bothered about the license. But for those who insist to upload to Wikimedia Commons, they have a special aim: to share their media forever, to anyone seek knowledge from anywhere, anytime without the slightest of limitations or login. These are the class of people who do not want any profit or mention. And these are the very people find it most difficult to upload their photos. It is the duty of the foundation to rectify the existing glitches of uploading, a day sooner so that this class of people will not go away. Sincerely desire to see Pattypan back into action at the earliest.--Vinayaraj (talk) 13:50, 15 January 2022 (UTC)[reply]
  • @Vinayaraj: Curious if you're familiar with the work currently being done to build a new structured data/batch uploading tool for Commons that leverages the OpenRefine data cleaning/manipulation software – does this address what you're looking for? MPinchuk (WMF) (talk) 20:17, 17 January 2022 (UTC)[reply]
    I have no technical know-how about these things. I have been using Pattypan for long and it is exactly what I needed, missing it heavily. Vinayaraj (talk) 13:19, 18 January 2022 (UTC)[reply]
    I am leading the development of the OpenRefine batch upload functionality. In my experience, people can learn to use OpenRefine in one or two hours. We'll do our best to organize trainings and create good documentation. Spinster (talk) 09:23, 23 January 2022 (UTC)[reply]
    Thank you, looking forward to the release Vinayaraj (talk) 15:22, 23 January 2022 (UTC)[reply]
  • I think it's a very important issue for mass uploading files in Commons; and I thinks licensing of the files are the problems. Thingofme (talk) 15:47, 18 January 2022 (UTC)[reply]
  • Please bring back Pattypan for Windows users! I work for Dumbarton Oaks and we have an established workflow that was functional with Pattypan but has been stalled out for months now. You cannot underestimate the lead time that it takes institutions like ours which don't have Wikimedians in residence to be able to start a process like this, and having that process stymied is causing us to lose a lot of momentum. I recognize that we owe a great debt of gratitude to the people who have the knowhow to build these tools, especially when they are doing it on their own free time, so I am not putting this on them, but if there is anything the Wikimedia Foundation can do to help, institutions like mine would be extremely grateful. Bettinche (talk) 03:33, 19 January 2022 (UTC)[reply]
  • Just adding my support to fix the issues with Pattypan, specifically the issues with the Windows version as a minimum. This is identified as the current problem on the GitHub page. For libraries looking to upload large collections with their metadata, it does seem the best tool out there. I also found it useful for mapping our metadata schema to Wikimedia's. I would certainly be interested in the proposal to have Wikimedia Commons functions in OpenRefine mentioned by @MPinchuk (WMF):, as I do use OpenRefine a lot for cleaning metadata records. However, I think in the short-term fixing the issues with Pattypan may be more achievable. --Wjbfarrell (talk) 11:51, 19 January 2022 (UTC)[reply]
  • I agree that it's imperative to fix Pattypan, even if it's just implementing a stop-gap measure. GLAMs rely on it, and there's no equivalent tool out there. The OpenRefine Commons upload sounds interesting, but it's not expected to be operational until June 2022 at the earliest. In the meantime, this bottleneck is going to be frustrating for many, especially because Pattypan wasn't deprecated or phased out, it just stopped working. brwz (talk) 16:20, 19 January 2022 (UTC)[reply]
    Yes; GLAMs should be able to at least finish their current Pattypan-based upload projects that are now suddenly stuck in the midst of their established workflow. Then, certainly, if there are some even better tools someday, a phase out date for Pattypan could be announced, so that everyone has enough time to adapt, but people need a working Pattypan first. Gestumblindi (talk) 23:53, 19 January 2022 (UTC)[reply]
  • Also the Image Archive of the ETH-Library is very hugely depending on the functionality and functioning of Pattypan! We are planning to upload more than 100'000 Swissair areal images and other. We used to work with GWToolset but as it was communicated that this tool is no longer being maintained, we changed our whole workflow to work with Pattypan and also helped other GLAM-institutions to get used with it. We would be very greatful if the tool would work again soon! ^FH ETH-Bibliothek (talk) 07:24, 20 January 2022 (UTC)[reply]
  • The Central Library of Zurich is planning to load more image collections up to Wikimedia Commons, currently a collection of ~2'000 images, and more collections in the future. For doing this, we really depend on Pattypan. If it could run again on Windows, that would be a big plus. ^AW --Zentralbibliothek Zürich (talk) 13:08, 20 January 2022 (UTC)[reply]
  • I also ask for the resolution of the problem. The project I am following has stopped uploading images for over 3 months due to the Pattypan malfunction. --Spinoziano (BEIC) (talk) 09:16, 22 January 2022 (UTC)[reply]
  • Bring back Pattypan soon. I support Vinayaraj. There is no substitute for Pattypan. UploadWizard has its own demerits. Moreover mass unloading should be promoted. Shagil Kannur (talk) 09:39, 23 January 2022 (UTC)[reply]
  • Chipping in on this in support for restoring Pattypan a.s.a.p. while we wait for OpenRefine to take over. We in the Netherlands also have several GLAM institutions that are affected by this. MichellevL (WMNL) (talk) 09:32, 26 January 2022 (UTC)[reply]
  • Info: 4 days ago, a new experimental Pattypan release (pattypan-21-10-experimental-3) was made by Abbe98 and I can happily report that it does work under Windows 10! We're currently uploading images using that release. You still need to manually install and load OpenJFX modules (starting Pattypan from commandline with the options as in the example), but it does work. @Vinayaraj, MusikAnimal (WMF), Hadi, Aktron, Hansmuller, ETH-Bibliothek, Zentralbibliothek Zürich, Spinoziano (BEIC), and MichellevL (WMNL):. Zentralbibliothek Solothurn (talk) 15:19, 27 January 2022 (UTC)[reply]
    Unfortunately, there are still some issues with pattypan-21-10-experimental-3 and pattypan-21-10-experimental-4 - in my case, the test load gets stuck at the first image, as reported here: Issue #145. ^AW --Zentralbibliothek Zürich (talk) 16:06, 31 January 2022 (UTC)[reply]
    @Zentralbibliothek Zürich and Abbe98: Well, we just uploaded a batch of 192 files with experimental-4 and the upload never got stuck in the way it did previously. There was a "Connection reset" error message after 119 images and the next image was skipped, but that was after I left the machine unattended for a while and then the upload continued successfully without the need to restart Pattypan, so I think that's a different issue. Zentralbibliothek Solothurn (talk) 18:10, 2 February 2022 (UTC)[reply]
  • Some time after Vicuna, Commonist and Pattypan stopped working, Vicuna 1.24.8a portable appeared, which does work for me, albeit partially. It has three bugs: 1) (random) often one or more files do not get uploaded wihtou obvious reason, 2) (random) it forgets the previously entered coordinates and one has to scroll across the globe to get again to a specific location, 3) (persistent) no category checking. If these issues are fixed, I think many users will be happy. JiriMatejicek (talk) 09:03, 1 February 2022 (UTC)[reply]
  • Pattypan version 22.02 is now available. You can find the download and the release notes here. Abbe98 (talk) 15:41, 7 February 2022 (UTC)[reply]
  • Just a link to our project report regarding VicuñaUploader improvement. --Juandev (talk) 04:12, 7 May 2022 (UTC)[reply]

Voting