Community Wishlist Survey 2023/Bots and gadgets/A more performant bot to replace ListeriaBot

Random proposal ►◄ Bots and gadgets The survey has concluded. Here are the results!

A more performant bot to replace ListeriaBot

Problem: Since the release of ListeriaBot version 2, there is a problem with lists with many links to big entities (for example, links to countries), which make the list generation fail. See some previous discussions: 1, 2. The issue is reported in the official repository: magnusmanske/listeria_rs#66
Proposed solution: Create a new, more performant bot or fix the memory issues with ListeriaBot. An enhanced version of ListeriaBot, one that can carry us into the 2030s.
Who would benefit: ListeriaBot is used to automatically maintain lists of Wikidata items in various Wikipedias. It is used intensively by Women in Red in various languages, as well as other projects (3,204 lists in English, 1,152 in French).
More comments: A similar proposal by MarioGom was merged into this one.
Phabricator tickets:
Proposer: Edelseider (talk) 18:13, 23 January 2023 (UTC)[reply]

Discussion

I'm proposing again this year to fix the memory issue: Community Wishlist Survey 2023/Bots and gadgets/Fix ListeriaBot memory issues. But I'll support this proposal too. Whether it's fixing ListeriaBot or rewriting it, we need a solution. MarioGom (talk) 15:17, 28 January 2023 (UTC)[reply]
For transparency, per our conversation I've merged that proposal with this one. I think in the eyes of the voters, it doesn't matter whether it's a rewrite or just fixes, so long as it's fixed :) If either of you have an issue with the new wording of the proposal, please feel free to edit the translation subpage and we will get that marked for translation for you. Regards, MusikAnimal (WMF) (talk) 05:35, 31 January 2023 (UTC)[reply]
Eventually ListeriaBot should be replaced with automated list generation. GZWDer (talk) 22:57, 8 February 2023 (UTC)[reply]
Yes. What @GZWDer says. The ticket WMDE has for it is phab:T67626. Lydia Pintscher (WMDE) (talk) 11:21, 10 February 2023 (UTC)[reply]

Tracked in Phabricator:
Task T329380

Also see ListeriaBot returns "Last line: ERROR: Login failed":

M2k~dewiki (talk) 18:18, 10 February 2023 (UTC)[reply]

The events of the last days (the bot completely inactive since 7 February, its creator looking into the problem without any result yet) have shown that ListeriaBot has become a liability, plain and simple. Too many lists across too many Wikis depend too much on it. --Edelseider (talk) 14:15, 12 February 2023 (UTC)[reply]
@GZWDer:@Lydia Pintscher (WMDE):@MusikAnimal (WMF): Okay, ListeriaBot is working again. Except that it is not working correctly at all. For some reason, it now deletes (for lack of a better word) the line "image", thus creating a list where the labels do not correspond to the content of the fields: see here. The stuff of nightmares, that tool is.--Edelseider (talk) 12:12, 13 February 2023 (UTC)[reply]

I think I fixed the problem that the "bot passwords" login did not work via API on some WMF wikis (any insight into that?) by switching the login to OAuth. I also fixed the table column problem, which was caused by a Pull Request that I foolishly merged. Right now the bot is somewhat limited by the Toolforge kubernetes constraints; relaxed constraints (more RAM, more CPUs per pod) would help there. Barring serious malfunction, the bot should now update every list on every wiki once every two days. If that does not work for some tasks, TABernacle provides a possible alternative. --Magnus Manske (talk) 14:37, 13 February 2023 (UTC)[reply]

I often get a Killed by Memory Overload message, if I dont limit the number of entries to 1000 (for example, depending on the number of columns) whith LIMIT 1000. For 1500 or 2000 entries I get a Killed by Memory Overload message. M2k~dewiki (talk) 14:40, 13 February 2023 (UTC)[reply]

@Magnus Manske:@M2k~dewiki: Even a slight change of query (see here), and it's "killed by OS for overloading memory". It has been like this and it still is like this. --Edelseider (talk) 15:50, 13 February 2023 (UTC)[reply]

What is concerning is the bot will load the entire item for each items involved. This is really not a good thing. GZWDer (talk) 18:27, 17 February 2023 (UTC)[reply]

Indeed. I haven't looked into this for a while, but the problem used to be not so much about number of results, but about number of big items present in some fields, such as some geographical entities. MarioGom (talk) 12:38, 18 February 2023 (UTC)[reply]

Voting

Support--Frettie (talk) 18:14, 10 February 2023 (UTC)[reply]
Support--M2k~dewiki (talk) 18:15, 10 February 2023 (UTC)[reply]
Support عُثمان (talk) 23:53, 10 February 2023 (UTC)[reply]
Support Poslovitch (talk) 00:27, 11 February 2023 (UTC)[reply]
Support --NGC 54 (talk｜contribs) 00:37, 11 February 2023 (UTC)[reply]
Support Akbarali (talk) 08:55, 11 February 2023 (UTC)[reply]
Support Matěj Suchánek (talk) 10:56, 11 February 2023 (UTC)[reply]
Support RVA2869 (talk) 13:17, 11 February 2023 (UTC)[reply]
Support RPI2026F1 (talk) 13:19, 11 February 2023 (UTC)[reply]
Support Bluerasberry (talk) 14:59, 11 February 2023 (UTC)[reply]
Support I often modify lists on Wikipedia, adding missing elements. This would be really helpful in order to have updated lists and contributores could spend more time improving records on Wikidata instead, so that all Wikipedias will benefit from that. FinixFighter (talk) 15:12, 11 February 2023 (UTC)[reply]
Support Ayack (talk) 15:12, 11 February 2023 (UTC)[reply]
Support Updating Wikipedia lists based on Wikidata information will certainly be faster. Thingofme (talk) 15:26, 11 February 2023 (UTC)[reply]
Support Thomas Kinz (talk) 19:40, 11 February 2023 (UTC)[reply]
Support--WomenArtistUpdates (talk) 17:32, 12 February 2023 (UTC)[reply]
Support Bencemac (talk) 20:24, 12 February 2023 (UTC)[reply]
Support Husky (talk) 20:55, 12 February 2023 (UTC)[reply]
Support --PaFra (talk) 22:51, 12 February 2023 (UTC)[reply]
Support Izno (talk) 07:12, 13 February 2023 (UTC)[reply]
Support β₁₆ - (talk) 11:15, 13 February 2023 (UTC)[reply]
Support Tacsipacsi (talk) 13:38, 13 February 2023 (UTC)[reply]
Support JAn Dudík (talk) 17:00, 13 February 2023 (UTC)[reply]
Support Wikidata lists are used on a large scale for maintenance and task list generation. It is crucial to have a reliable and scalable system. Fuzheado (talk) 11:05, 14 February 2023 (UTC)[reply]
Support Quiddity (talk) 21:15, 14 February 2023 (UTC)[reply]
Support cyrfaw (talk) 11:41, 15 February 2023 (UTC)[reply]
Support Sadads (talk) 01:10, 16 February 2023 (UTC)[reply]
Support Jon Harald Søby (talk) 12:37, 16 February 2023 (UTC)[reply]
Support For core bots, revamp required every 5 years or so. Yug (talk) 10:25, 17 February 2023 (UTC)[reply]
Support —The Editor's Apprentice (talk) 02:49, 18 February 2023 (UTC)[reply]
Support Kpjas (talk) 07:34, 18 February 2023 (UTC)[reply]
Support MarioGom (talk) 12:15, 18 February 2023 (UTC)[reply]
Support Rosiestep (talk) 12:33, 18 February 2023 (UTC)[reply]
Support Ipigott (talk) 20:42, 18 February 2023 (UTC)[reply]
Support PamD (talk) 23:49, 18 February 2023 (UTC)[reply]
Support Jklamo (talk) 11:56, 19 February 2023 (UTC)[reply]
Support Tommy Kronkvist (talk) 09:09, 20 February 2023 (UTC)[reply]
Support Nashona (talk) 14:30, 20 February 2023 (UTC)[reply]
Support a necessary piece of work Watty62 (talk) 17:42, 20 February 2023 (UTC)[reply]
Support Nikki (talk) 16:16, 21 February 2023 (UTC)[reply]
Support - PKM (talk) 18:47, 21 February 2023 (UTC)[reply]
Support Althair (talk) 04:01, 23 February 2023 (UTC)[reply]
Support TMg 12:32, 23 February 2023 (UTC)[reply]
Support CmdrDan (talk) 17:25, 24 February 2023 (UTC)[reply]