User talk:Beetstra/Archives 2021

From Meta, a Wikimedia project coordination wiki


linkwatchers ugh

The linkwatchers have been failing miserably and not particularly talking in IRC. In my inexpert way I have done some restarts and have some responsiveness back. though looking at the syslogs, I do see the troubling (replace user with xxxxx in case it needs to be quiet)

  • LINKANALYSER 000006 @ 2021-1-16 @ 21:44:30 - Dying painfully because of faulty liwa mysql handle - error Can't connect to MySQL: User 'xxxxxx' has exceeded the 'max_user_connections' resource (current value: 10) at LinkAnalyser.pl line 508.

I think that I have it back and running; I am see it generate a report or two. Maybe it was just db issues, <shrug> my problem-solving/understanding of what exactly was wrong is still substandard.  — billinghurst sDrewth 21:52, 16 January 2021 (UTC)

@Billinghurst: Hi, good morning. Yes, there is a cap to the number of connections the bot can have into the linkwatcher database. It once started to eat too many away, and that has consequences on the database. However, sometimes processes do not die cleanly and leave a connection open, upon which the bot reaches the cap. You'll have to kill everything completely, and then restart the bot. And I think that both LiWa3 and COIBot eat from the cap of 10, so one bot may give problems on the other. --Dirk Beetstra T C (en: U, T) 05:53, 17 January 2021 (UTC)
kkkk Probably had a quite a few caught as there was an absolute swag of processes, and had no idea of what was a normal/expected count. Not certain that I am aware of a graceful means to kill everything completely, so I did a number of ugly things that worked in the end. ¯\_(ツ)_/¯  — billinghurst sDrewth 11:14, 17 January 2021 (UTC)
@Billinghurst: I wouldn't know either, I think I just kept killing processes until it was silent ... then restart linkwatcher. --Dirk Beetstra T C (en: U, T) 12:53, 17 January 2021 (UTC)
Wow, that is my magic. "killall -9 perl" had a good impact after the shell was stopped.  — billinghurst sDrewth 12:56, 17 January 2021 (UTC)
@Billinghurst: you did not tell me that that also did not solve your problem. I have applied my ultimate motto: "If violence did not solve your problem, you did not use enough" ... I have restarted the whole instance, and started the script clean. --Dirk Beetstra T C (en: U, T) 18:56, 19 January 2021 (UTC)

Comment Comment killed all processes coibot, xlinkbot, liwa3; restarted processes. After a day and a half of the analysis of the same couple of domains, I decided that it was a boring loop. I am guessing that it was a database issue, but couldn't tell you. Diversity is back. If there was a simpler set of restarts then please let me which. Size 12 fire boots seemed best.  — billinghurst sDrewth 12:37, 27 January 2021 (UTC)

@Billinghurst: There seems to be something wrong with some forms of access, I notice a lot of join and leaves going on. Maybe a reset of the instances is again needed (strange, just did that ...). --Dirk Beetstra T C (en: U, T) 06:11, 31 January 2021 (UTC)

LiWa3 loop <= kicked; and new WPs

Seems that we had the linkwatchers cycling the same reports, so I have stopped and restarted coibot and liwa3. Got a big list of backlogged reports. Went to add a linkanalyser and COIBOT told me to FRO. Thought that it might processing the backlog, but not at this point. Will look at it again on my tomorrow, though probably cannot do a lot of kicking for 18-20 hours. We seem to be out of the loop anyway.

Do we need to add new WPs to the settings? I can see that the linkwatchers are in the feeds of new WPs like skr.wikipedia so I have no idea what adding them to the settings actually does. Thanks.  — billinghurst sDrewth 13:35, 17 February 2021 (UTC)

@Billinghurst: I don't know what causes that first issue. Seems to be happening over and over. Last time I tried to kill most processes on linkwatcher, it looked like there were 2 or 3 instances running at the same time (but .. why?).
Linkwatcher, on boot, reads a special page (I think the api version of Special:SiteMatrix), and from that creates a list of all wikis (the regular ones, it ignores the 'special' ones in the sitematrix, many of which are not accessible anyway). It adds to that list the ones in the variable special in the settings. Then it takes some aside (because of volume; set in owndiffreader - these wikis get their individual diffreader, currently en.wikipedia and wikidata), and divides the rest over 10 (? number seems to be hardcoded) DiffReaders (it saves files, 'channellist#' which are 'feed' lists for each the DiffReaders). Then the DiffReaders start (1 for en.wiki, one for wikidata, and the 10 for the 'rest'). Except for the ownwiki-ones, all have about 85 wikis to read (most of them rather silent). So any that are not in the sitematrix need to be put into special and in principle need a restart of the whole bot to read those variables at start (but you can manually add one to one of the channellist-files in the , and then reboot only that diffreader that reads that channellist ...). If one DiffReader keeps reconnecting it likely means that one wiki is too high volume (which is then likely es.wikipedia or de.wikipedia??) and that one then probably should move into ownwiki followed by a restart of the bot. --Dirk Beetstra T C (en: U, T) 11:07, 18 February 2021 (UTC)

Seems COIBot account is not AUTHing for freenode

Seeing both COIBot and Xlinkbot come and go in IRC. Tried lots of things including a reboot, to no success. Zppix believes that the account is not AUTHing, and I have no idea how to resolve that aspect, nor even how to problem solve or check that is the case, so I am handing that bit back to you. Happy to be further educated on what I should be doing. Regards arms and legs  — billinghurst sDrewth 23:53, 26 February 2021 (UTC)

Something worked automagically at some point. No idea why, and after liwa3s had disappeared. All back and functioning.  — billinghurst sDrewth 04:34, 27 February 2021 (UTC)

Sigh, bots have issues. Needs a break? COVID-19? Maybemaybe I have a bit of time end of this week to keep an eye on it and see if I can figure out what the issue is. --Dirk Beetstra T C (en: U, T) 05:40, 28 February 2021 (UTC)

bd808 asks ...

Whilst I was seeking some help today BDavis asked two things re COIBOT. I can do the second, but not the first due to access. 

<bd808> it would probably also be a good idea to clean up some/all of tools.coibot and add a README there saying that it is all on another Cloud VPS project now

and I went to do it and found that I don't have the access to the tool's group on toolforge to do it. 

<bd808> billinghurst: write some system admin procedure docs for this project pretty please. :)
...
<bd808> billinghurst: the most basic info would be something like https://wikitech.wikimedia.org/wiki/Tool:Stashbot#Maintenance

which I will do.  — billinghurst sDrewth 11:32, 27 February 2021 (UTC)

@Billinghurst: you should get access to tools.coibot ... if only I knew how to give you that. --Dirk Beetstra T C (en: U, T) 05:42, 28 February 2021 (UTC)
https://toolsadmin.wikimedia.org/tools/id/coibot login and you can maintain maintainers  — billinghurst sDrewth 11:23, 28 February 2021 (UTC)

FYI: Racking up files

Seems that there is some DB type issue as linkwatcher is racking up link files and not processing a lot of links. I have been through both a restart of linkwatcher and a reboot of the liwa3 instance, though it keeps slowly increasing with only occasional file loading. I also did a hard restart of coibot, though no reboot, just in case. 

<sDrewth> !backlog
<COIBot> No reports waiting. On Wiki: 0 open XWiki reports and 30 open Local reports.
<COIBot> Syslogs: 20: - coibot: -1 secs. commander: 0 secs. diffreader: 14658 secs. linksaver: 8 secs. parser: 4093 secs. readbl: - script: -1 secs.
<LiWa3_2> Syslogs: 20: - diffreader: 14 secs. linkanalyser: 697 secs. linkparser: 2688 secs. linkreporter: 12365 secs. linkwatcher: 125 secs. output: - script: -1 secs.
<LiWa3_3> LW: 03 hours 26:08 minutes active; RC: last 1 sec. ago; Reading ~ 864 wikis; Queues: P1=222; P2=1687; P3=90 (101 / 399); A1=1111; A2=0 (1194 / 868); M=0 - In LinkWatcher edit-backlog: 4094 files (0 lines) and in analyser backlog: 8 files (0 lines).

 — billinghurst sDrewth 02:46, 7 March 2021 (UTC)

@Billinghurst: syslog.linkparser throws strange errors - it seems that somewhere the system changed and it misfires on regexes which then results in mis-assigned edits. It looks like all edits are on 'Mw::' (as if it does not read the diffurl good), and then it tries to read diffs from en.wikipedia that were somewhere else. I don't understand yet what is the issue (whether it is DiffReader.pl that misinterprets stuff, or if linkwatcher.pl is doing something with the data). Note that also in the backlog files things are wrong, so it goes wrong between reading the diff from the feed by DiffReader.pl, and storing them by linkwatcher.pl (i.e. before it hits the LinkParser.pl). --Dirk Beetstra T C (en: U, T) 06:01, 7 March 2021 (UTC)

@Billinghurst: is there somewhere an extremely slow or throttled server? So that the bots lag because they have to wait for info/response? —Dirk Beetstra T C (en: U, T) 15:06, 7 March 2021 (UTC)

User:Sic19 thwocking WD with urls

I can see that User:Sic19 is racking up edits at WD adding official websites, and up to about 10M edits (unsure how any are recent). I have undertaken wl add Sic19 * and I hope that is the right solution. If there is something better that I can do, then please let me know.  — billinghurst sDrewth 23:33, 13 March 2021 (UTC)

@Billinghurst: yes, I guess that is it. Flooders on WD are an issue. Not parsing them and not getting the data is not really an option either (it sets a record for official websites, if I have time I could program something to remove ‘official site to subject’ from stats and become more precise). Crap ...
We should be whitelisting/ do-no-count-ing more links on WD though. Would be great if suchs flooders could inform linkwatcher and coibot beforehand ... —Dirk Beetstra T C (en: U, T) 05:50, 14 March 2021 (UTC)

Can COIBot/LinkReports be generated manually?

It sometimes happens that I'm not sure whether there is enough evidence to report a domain at en:Wikipedia talk:WikiProject Spam. It would be nice if I could generate a preview of the domain's LinkReport without it being saved. This would tell me if the domain is worth reporting. —Bruce1eetalk 09:12, 5 April 2021 (UTC)

@Bruce1ee: I have granted you access to request reports at user:COIBot/Poke. Or if you use IRC, then connect to Freenode and #wikimedia-external-links and you can request reports and run some analytics (Small Wiki Monitoring Team/IRC).  — billinghurst sDrewth 13:53, 5 April 2021 (UTC)
Thank you billinghurst, that will help a lot. I'll be using COIBot/Poke as I don't use IRC. —Bruce1eetalk 14:00, 5 April 2021 (UTC)
@Bruce1ee and Billinghurst: I was earlier today planning to check whether you had that capability. Done. --Dirk Beetstra T C (en: U, T) 14:16, 5 April 2021 (UTC)
Thanks. —Bruce1eetalk 14:31, 5 April 2021 (UTC)

IRC/Migrating to Libera Chat

We are going to need to do COIBot and LiWa3.

Tell me what you would like me to do to assist. Happy for whatever drudgery tasks you need done. First question is do you want a phab ticket generated, or is coordination going to be here?  — billinghurst sDrewth 13:54, 21 May 2021 (UTC) 

# freenode settings
freenodeserver=irc.freenode.net
freenodeserverport=8001

...

# freenode settings
freenodeserver=irc.libera.chat
freenodeserverport=8001
... IRC client at irc.libera.chat on ports 6665-6667 and 8000-8002 for plain-text connections, or ports 6697, 7000 and 7070 for TLS-encrypted connections.

Painful.

I’ll be around tomorrow to make a start, after that very limited. I first want to run a backup of current, then linkwatcher (easiest), then COIBot and XLinkBot. Problem is going to be channel modes and users in the channels (but that needs a cleanup anyway). —Dirk Beetstra T C (en: U, T) 07:19, 22 May 2021 (UTC)

The IRC stuff I can do; especially in association with CVN group. I have already started on a couple. Create list of your hierarchy, and I can work with the group coordinator and the CVN network to get things in place. Happy to be the legs.  — billinghurst sDrewth 11:12, 22 May 2021 (UTC)

@Billinghurst: See User:Beetstra/LiberaMove. --Dirk Beetstra T C (en: U, T) 07:02, 23 May 2021 (UTC)

Encoding error

User:COIBot/LinkReports currently shows "12:22:51, Tue Jun 01, 2021 - highlevelsound.blogspot.de - XWiki link additions by КиноФан2021". The linked page shows "КиноФан2021" encoded correctly. Any way you could fix that? ~~~~
User:1234qwer1234qwer4 (talk) 17:48, 3 June 2021 (UTC)

@1234qwer1234qwer4: yes, I am aware of that problem. Encoding and de-coding is sometimes tricky in Perl, and it appears that there is a double loop (or even worse) somewhere. Sometimes you see also 2 users in the list of users where one is the encoded/decoded version of the other one. It is on my list (User:COIBot/Wishlist, item 9 basically), I will also put this one in the list to re-parse and see if I can resolve it. I do have some time end of this month to do so.
The thing that is never mangled is the diff-link, so in case of doubt that is the way to use. --Dirk Beetstra T C (en: U, T) 05:27, 6 June 2021 (UTC)

About os.wiki

Znauri (Qornisi) is part of Georgia and now is occupied by Russia, and there is non-recognized republic South Ossetia, I have made change in this page, I have just written that it's territory of Georgia but administrator of os.wiki has reverted my edit and now I am not able to edit this page because this user has defensed this page, please help me https://os.wikipedia.org/w/index.php?title=%D0%97%D0%BD%D0%B0%D1%83%D1%8B%D1%80&type=revision&diff=494522&oldid=494521 --ჯეო (talk) 10:29, 1 July 2021 (UTC)

@ჯეო: I am not sure what you want to do - I don't understand the language, I do not understand what happened, and I am not an administrator on os.wikipedia. I am not even sure where to refer you to. Sorry. Dirk Beetstra T C (en: U, T) 10:52, 1 July 2021 (UTC)
I have asked you for help because it always work against this separatism in os.wiki, ok if you don't want you can do nothing but, you have to know that os.wiki is full of separatism and admin of this project is fighting about neutral edits in articles about occupied territories, good bye, have a nice day!--ჯეო (talk) 10:55, 1 July 2021 (UTC)
You should avoid "forum shopping" by posting the same message on multiple talk pages and instead have a centralised discussion. Requests for comment might be the place you are looking for? ~~~~
User:1234qwer1234qwer4 (talk) 12:01, 1 July 2021 (UTC)

"link removed"

In Special:Permalink/21697737#Selected additions, the first point says "top edit, link removed", but I just had to revert it. Is that a problem with the bot's compatibility with Wikibase? ~~~~
User:1234qwer1234qwer4 (talk) 20:02, 6 July 2021 (UTC)

Yes. The last edit indicator is correct, the visibility is not. Use the d: linksearch component in the LinkSummary template  — billinghurst sDrewth 00:25, 7 July 2021 (UTC)

@1234qwer1234qwer4 and Billinghurst: I think that it is an issue with parsing wikidata .. probably WD works a bit different and hence detection of links in 'top edit' does not work properly. I may have a look at it, will put it into the todo-list. --Dirk Beetstra T C (en: U, T) 05:30, 7 July 2021 (UTC)

I don't fuss it. Knowing that it does it, and that the "top edit" is correct is sufficient to me. Indicators are indicators, and that WD has O X, rather than O O is so manageable.  — billinghurst sDrewth 07:57, 7 July 2021 (UTC)

User:COIBot/XWiki/themexriver.com

Is there any reason why c:Special:Diff/551169321, which I just reverted, was not listed in the report (not at the LinkReports page either)? ~~~~
User:1234qwer1234qwer4 (talk) 11:29, 13 July 2021 (UTC)

I don’t know. Although LiWa3 is optimized for connections to the wikimedia feed, it does occasionally disconnect. Dirk Beetstra T C (en: U, T) 13:02, 13 July 2021 (UTC)
I think Special:Diff/21719775, which was the reason why I requested a report, should also have been listed. ~~~~
User:1234qwer1234qwer4 (talk) 14:54, 13 July 2021 (UTC)

Category:Local COIBot Reports

This seems redundant with Category:COIBot Local Reports. Should they be merged? ~~~~
User:1234qwer1234qwer4 (talk) 10:02, 10 June 2021 (UTC)

@1234qwer1234qwer4: Looks like it. It looks like Category:Local_COIBot_Reports only contains reports from 2013, 2014, 2015. Just a set of forgotten reports. Yes, they can be merged and then we will have to monitor for a bit whether COIBot is not adding stuff. --Dirk Beetstra T C (en: U, T) 10:16, 10 June 2021 (UTC)
I have redirected the category and moved its contents with the flood flag. ~~~~
User:1234qwer1234qwer4 (talk) 13:51, 10 June 2021 (UTC)
@1234qwer1234qwer4: thanks, lets keep an eye if COIBot is not refilling. —Dirk Beetstra T C (en: U, T) 11:35, 11 June 2021 (UTC)
Huh? ~~~~
User:1234qwer1234qwer4 (talk) 22:15, 12 June 2021 (UTC)
FYI, there were three new pages in the redirected category again, which I just moved to the main one: User:COIBot/Local/amz10.com, User:COIBot/Local/hulu.com, User:COIBot/Local/moddude.com. ~~~~
User:1234qwer1234qwer4 (talk) 11:07, 19 September 2021 (UTC)
That is what I meant. See User:COIBot/Wishlist (I will go through the code somewhere next week). Dirk Beetstra T C (en: U, T) 05:45, 13 June 2021 (UTC)
@1234qwer1234qwer4: Actually, this diff that you cited above suggests that the bot mistakingly takes the en.wikipedia settings from en:user:COIBot/Settings and applies them here instead of user:COIBot/Settings. Maybe it is a failed settings load that is not caught by the bot. I recall that I have seen a similar error on en.wikipedia, where COIBot did not save use the path 'user:COIBot/LinkReports/' on example.com (to save User:COIBot/LinkReports/example.com) but instead used the en.wikipedia path 'wikipedia:WikiProject Spam/LinkReports/' to save Wikipedia:WikiProject Spam/LinkReports/example.com on meta (which the mediawiki software funnily enough recognizes as 'save WikiProject Spam/LinkReports/example.com on en.wikipedia'.
I will try to check for failed settings loading, probably by setting a fake parameter on each. Dirk Beetstra T C (en: U, T) 08:34, 15 June 2021 (UTC)
Do you have an update on this? ~~~~
User:1234qwer1234qwer4 (talk) 19:51, 6 July 2021 (UTC)
@1234qwer1234qwer4: No, did not manage yet to solve this. I am very limited in time. --Dirk Beetstra T C (en: U, T) 11:14, 19 September 2021 (UTC)

COIBot and blacklisted links

I am seeing COIBot having edits rejected as it is attempting to add blacklisted IP/urls addresses [1]. In numbers of cases this is stopping the reports from being created. I know that this will partly reflect my quickcreate additions and blacklist. Is there an easy solution around this, or do we just live with it?  — billinghurst sDrewth 02:00, 9 July 2021 (UTC)

@Billinghurst: probably through the template. I suspect that the problem is links like with archive.org: archive.org/blah/data/https://pornographicfireparrot.com .. liwa3 has stripped the protocol from the beginning, but there is a sub-link that has the protocol and is blacklisted. The whole set should be in nowiki-tags and most will be solved.
Note that template-magic will also solve the problem in the previous thread, ‘if link=top then dont show removed or there’. Dirk Beetstra T C (en: U, T) 04:22, 9 July 2021 (UTC)
👍 Thx  — billinghurst sDrewth 04:43, 9 July 2021 (UTC)
Do you have an update on this? COIBot is still getting quite a number of spam blacklist hits. ~~~~
User:1234qwer1234qwer4 (talk) 13:50, 18 September 2021 (UTC)
@Billinghurst and 1234qwer1234qwer4: I now see that all these missed reports are missed due to COIBot's programming of adding the server IP to the reports (i.e. the server the website is hosted on). All these servers have been blacklisted. I think I will have to disable that function in COIBot (we anyway have the server IP track through the {{LinkSummary}}. I will see if I can do this through the settings. --Dirk Beetstra T C (en: U, T) 11:01, 19 September 2021 (UTC)

Database as a Service in Cloud VPS

Just in case you didn't see the email in cloud-announce

 — billinghurst sDrewth 13:11, 22 July 2021 (UTC)

@Billinghurst: thanks, I will pass for now until it stabilizes. Moving a 10-year worth of edits database and then crashing it may not be worth it. Dirk Beetstra T C (en: U, T) 10:41, 12 August 2021 (UTC)

Monitor list

Is there any reason why the bot writes

[...]

(example)? The first link appears to lead to nowhere, and I'm not sure what the difference between these two would be at all. ~~~~
User:1234qwer1234qwer4 (talk) 13:56, 18 September 2021 (UTC)

@1234qwer1234qwer4: COIBot has different monitorlists (one manual redlist, one autoredlist filled with mentions on wikis, on from LiWa3 statistics), but I think there is something wrong in the header there. Not an issue, but worth to put on my to-do list. Thanks for noticing! --Dirk Beetstra T C (en: U, T) 10:51, 19 September 2021 (UTC)
If there are multiple monitorlists, it might not be the best idea to call it "the" monitorlist every time. ~~~~
User:1234qwer1234qwer4 (talk) 11:08, 19 September 2021 (UTC)

COIBot for dewiki

User:Beetstra

(commons: userpage | talk (←links) || Edits: +/- | logs ← Beetstra ← logs || search)(Reports: COIBot)
(Tools: GUC | stalktoy | CA) (Search: Google | → links ←User talk )
(enwiki: userpage | talk (←links) || Edits: +/- | logs ← Beetstra ← logs || search)

hi user:Beetstra!
is COIBot running for dewiki anywhere at libera.chat? (at freenode the bot spoke to #cvn-wp-de.)
i'd like to post several queries, because we found a spamming network at dewiki and want to collect more information about it, see w:de:WP:SBL#betrugstest.com,_onlinecasinosdeutschland.com,_besteonlinecasinos.com,_besteonlinecasinos.co. -- seth (talk) 10:32, 19 September 2021 (UTC)

@Lustiger seth: de.wikipedia is not in ownchannel in User:COIBot/Settings. Nonetheless, COIBot should reply to all your questions in any channel where it is, regardless of the 'source' of the information. Do you want to see if we can get COIBot into libera's #cvn-wp-de?
Note that there is also always User:COIBot/Poke. --Dirk Beetstra T C (en: U, T) 10:39, 19 September 2021 (UTC)
Additionally, you could set up {{LinkSummary}} on de.wikipedia for those discussions, and I can make COIBot easily report automatically then. I've not done this yet, but we could even enable COIBot on de.wikipedia and make it report there for de.wikipedia specific reports (it is all controlled through settings). --Dirk Beetstra T C (en: U, T) 10:44, 19 September 2021 (UTC)
hi!
"COIBot should reply to all your questions in any channel where it is": ah, ok. that might be sufficient. which channel do you suggest to use?
User:COIBot/Poke: that might be sufficient, too. so could i just add about 20 entries there? (i'm i in the grantedusers group already as i'm a "sysop"?)
using LinkSummary at dewiki and generate de-specific reports sounds interesting, too. -- seth (talk) 18:11, 19 September 2021 (UTC)
and it would be great, if i could generate user reports such as w:en:Wikipedia:WikiProject Spam/UserReports, but for de. -- seth (talk) 20:35, 19 September 2021 (UTC)
@Lustiger seth: - I think that #wikimedia-external-links is rather silent, the real feed channels are often to noisy and you might miss (parts of) the bot's answers.
Poking works for all of the users with granted rights, it is just to 'poke' the bot to report, it will not track those like it would on other pages. The settings that define what COIBot follows/reads for templates is defined in m:User:COIBot/Settings variables: pokepages, followpages, autopokepages and linksummarypages, and regex-lists in regexpages; if you seem to be continuously ignored, add yourself to grantedusers; note that the bot sometimes disconnects from the wikimedia feeds and misses edits through that, and that it sometimes takes time before settings take effect as they are not continuously read with every action that it might need to take (it only reads the settings before it saves to see whether it has editing_on set to '1'). (feel free to adapt the settings, that is the reason they are on-wiki, user:Billinghurst can help as well, he has access to console and IRC as well).
You can just copy {{LinkSummary}} and {{UserSummary}} to de, and adapt the links a bit. It is then just a matter of changing the settings on meta to have it parse those as well (it requires a restart of the core of the bot since that is where that information is parsed). I am not sure whether you can translate the name into German or that I then have to adapt the code (or that I have caught that through localised 'i18n' settings). (to have the reports saved to de.wikipedia we need to have bot-rights on de.wikipedia and set up the local settings and path/category structure - I am rather time limited so I am not sure when/if I will have time to get to the depths of that, it should work mostly through adapting the settings though). (Please let me know what names that you would like to use on de.wikipedia for the templates and I can see if I can get it to work, time allowing).
Poking works on {{UserSummary}} just like for {{LinkSummary}} on all 'poke enabled' pages. Note that sometimes pages are there missed for poking, or that Wikimedia may make COIBot fail saving (blacklisted links, edit filters, etc.) or that the saving module crashes due to a too large report size. Sometimes it takes two or three tries, but sometimes it will never save the report and you will have to use IRC tools to get some data. --Dirk Beetstra T C (en: U, T) 06:16, 20 September 2021 (UTC)
hi!
i used the poking yesterday and it seemed to have worked: User:COIBot/LinkReports/onlinecasinosdeutschland.com. :-)
so i added some more domains there. should i delete the entries in the poke list by myself after the bot has generated the pages?
the LinkSummary template is great as it is. the UserSummary template fits to enwiki only. so i created a german version now: w:de:template:UserSummary. another possibility would be to create a third param such that one could use {{UserSummary|lustiger_seth|de}} in order to replace the default language (en). then i even would not need a local version of that template.
by the way: some links in the UserSummary template don't work, because internal links and GET params are mixed up there. -- seth (talk) 08:28, 20 September 2021 (UTC)
@Lustiger seth: Forgot to say that you also need {{IPSummary}} (for reporting users that are an IP, it has different tools).
I am happy that poking works. You can blank them at will, claim an own section there, the bot looks at the added part of the diff and extracts the added templates from that text. What is on the page has no function afterwards. I know some people use it by blanking and self-reverting the set that they want to refresh sometimes.
For the templates, I would expect that you would want to rename it to your own language (w:de:Template:GebrauchersAbstrakt; is this Abstrakt the right translation of Summary, I am not even sure whether I would use abstrakt or Samenvatting in Dutch)?? Problem there is that I think that 'UserSummary'/'IPSummary' and 'LinkSummary' are hard-coded regexes (I will put te German translations if de.wiki prefers as soon as I can). What the template displays and what links should work is irrelevant for the bot. The templates also exist here on meta, they have different links and different display format (I like the meta-format more).
Thanks for noticing some links are broken - I will have a look (but apparently then these links are not used a lot, no-one noticed before). --Dirk Beetstra T C (en: U, T) 08:49, 20 September 2021 (UTC)
(talk page stalker) "Abstrakt" does not exist as a noun in German ("Abstract" is used but only in context of papers); you would use "Zusammenfassung". ~~~~
User:1234qwer1234qwer4 (talk) 09:50, 20 September 2021 (UTC)
@1234qwer1234qwer4: I was expecting that, Dutch 'Samenvatting' is also for me a better term. Dirk Beetstra T C (en: U, T) 10:06, 20 September 2021 (UTC)
hi!
"user summary" would be something such as "User-Zusammenfassung" (as "user" can be used in german, too[2]) or "Benutzerzusammenfassung". but imho that does not fit to what the template really is.
and actually we have already w:de:template:user which redirects to w:de:template:Benutzer which gives similar but less output. so in theory i could add the link to COIbot reports there. however that would probably lead to long and unnecessary discussions. so a separate template that's primarily used in context of link additions should be fine. the name could be "Nutzerverlinkungen" or "Benutzerinfo" or alike, but as the template should be somehow crosswiki-usable, i'd prefer the somewhat canonical name UserSummary. (in fact i find the localization of such functional names totally counterproductive, but that's another topic.)
nevertheless, i'm still not sure, whether we need COIBot at dewiki or whether it would be sufficient or even better to store all the information here at meta. storing the information at meta might avoid some redundancy and might lead to more cross-wiki exchange of information. what do you think? and what would be easier for you? -- seth (talk) 21:46, 20 September 2021 (UTC)
@Lustiger seth: OK, the current names are the easiest anyway as the software is handling that already. Just let me know if there is ever a German language version and I will adapt the software accordingly then.
There is some local functionality that may be of use to people. E.g. the watchlisting of category-changes on the right categories shows you in your watchlist when COIBot saves a report of local interest (and you can act on it quickly, sometimes while the spammer is still on the current IP or at least active). Similarly the xwiki-reports would be reported to de.wikipedia, and you might be able to react on the additions to the German wikipedia of the links. Also on en.wikipedia the /Local reports are handled by a script which handles closing/addition/logging of blacklist handles (like we do here with SBHandler on meta as well). In short, it gives you some advantages/possibilities. The local reports will anyway be here on meta (see Category:COIBot_Local_Reports_for_de.wikipedia.org and its subcategories). It will however be a bit of tinkering to get it up and running (setting up the userpage and local settings, getting bot rights, setting up a tree for the reports, categorisation schemes, etc.) which may, due to my low availability, result in some messy times (badly formatted reports, settings not correct, and if there is anything that I did not properly catch in the coding it may stay even longer). (it would however make a nice testcase and I could catch the process in a 'manual' for next times). --Dirk Beetstra T C (en: U, T) 10:07, 21 September 2021 (UTC)
hi!
ok, so a local version would have some benefits. however, let's first cope with this here at meta, because this seems to be less time-expensive.
for users and for some links the poking does not seem to work, see User:COIBot/Poke#current_dewiki_seo_spam. what could be the reason? -- seth (talk) 07:30, 24 September 2021 (UTC)
┌─────────────────────────────────┘
@Lustiger seth: That the reports did not get made can have several reasons: the bot can be blocked for page creation for some reason (title blacklist - unlogged; spam blacklist), the bot can have missed the edit (instability of the network), or reports are too big (which results in the procedure to crash due to memory issues). I have tried to re-poke, I hope they come up. --Dirk Beetstra T C (en: U, T) 07:30, 26 September 2021 (UTC)
hi!
i tried to re-poke already.[3] it did not seem to help. -- seth (talk) 07:08, 27 September 2021 (UTC)
@Lustiger seth: I re-tried those as well. I have to see the bot logs (which I can't access from here), there are some which seem to be consistently missed. Dirk Beetstra T C (en: U, T) 10:52, 27 September 2021 (UTC)
hi!
at the moment the poking does not seem to work at all for me: see User:COIBot/Poke. has the interface changed? -- seth (talk) 11:09, 27 December 2021 (UTC)
@Lustiger seth: hmm, I notice that now as well. It sees pokes ([4]), but seems to miss others. Parts of the (relevant) Wikimedia coding (of the API) changed, but I think I resolved all of that. But I also changed the Wikimedia-channel reading capabilities of COIBot .. and that is another question: does COIBot lurk in the right channels?
I will try to have a look, otherwise just use en:User:COIBot/Poke (as the example above was poked there, and it worked). -- Dirk Beetstra T C (en: U, T) 12:46, 27 December 2021 (UTC)
It was an issue with the code, it picked up my latest edit so it should work now (it was ignoring ALL poking on both meta and commons, but not on en.wiki and de.wiki). Thanks for spotting this.
@Lustiger seth: forgot to poke. -- Dirk Beetstra T C (en: U, T) 08:47, 28 December 2021 (UTC)
hi!
great, thanks! now only "umzug-berlin.de" is missing, but this seems to be the other problem (that i mentioned in september). -- seth (talk) 09:21, 28 December 2021 (UTC)
@Lustiger seth: hah, I found it. umzug-berlin.de is not reported because '\bberlin\.de\b' is whitelisted. I have created a datadump:
<Beetstra> datadump umzug-berlin.de
<COIBot> dumped 5 records to m:user talk:COIBot/LinkReports/umzug-berlin.de @ 1640691571
I hope that helps. Dirk Beetstra T C (en: U, T) 11:40, 28 December 2021 (UTC)
oh, that's interesting indeed! but where is '\bberlin\.de\b' whitelisted? -- seth (talk) 23:16, 28 December 2021 (UTC)
@Lustiger seth: User:LiWa3 has an internal whitelist to improve its speed. Links with that match the regex on the internal whitelist are not further analysed and reported (they just 'obscure' the list of things that are bad for those who monitor the feeds). COIBot also does not save those reports as they tend to be way too big anyway, but that now turns out to be a bad decision. Maybe the '\bberlin\.de\b should be removed and replaced with a '^\-berlin\.de\b' or something similar ('link wl del <regex>' and 'link wl add <regex>' @LiWa3 to solve it; a quick way of whitelisting domains on both LiWa and COIBot is 'clear <domain>' @COIBot, which executes 3 or 4 commands in arow.
The command 'datadump <domain>' ignores everything, it just does a SQL statement for <domain> and then dumps the records to the talkpage. It also has a much higher limit - I think I set it to 10s of thousands (see settings, 'dbdumplimit - it needs a limit to avoid accidental dumps of youtube.com or doi.org, and getting absurdly big pages). --Dirk Beetstra T C (en: U, T) 07:13, 30 December 2021 (UTC)
ah, ok, it's a bot-internal whitelist. then i understand it. -- seth (talk) 20:56, 30 December 2021 (UTC)

repoke: protected or open command

I have added command. Is it protected or open to all?  — billinghurst sDrewth 23:09, 3 November 2021 (UTC)

@Billinghurst: it is open. —Dirk Beetstra T C (en: U, T) 03:45, 4 November 2021 (UTC)

template:deferlocalwhite

we have this one to poke under user's noses. Same construct as {{deferlocal}}  — billinghurst sDrewth 09:40, 14 November 2021 (UTC)

@Billinghurst: we need a Template:Blacklist-toolbox somewhere at the top. -- Dirk Beetstra T C (en: U, T) 10:23, 14 November 2021 (UTC)
Do you mean for listing the available templates, like is done in SRCU? If yes, can we wait until the allow and disallow rename is done in the next weeks? We can take notes of what we want to build in the list in the meantime.  — billinghurst sDrewth 11:24, 14 November 2021 (UTC)

liwa3 "link nr search"

<sDrewth> LiWa3_1 link nr add \beurogenes\.blogspot\.\w{2,4}
<LiWa3_1> Item '\\beurogenes\\.blogspot\\.\\w{2,4}' added to nr (INSERT INTO linkwatcher_noredirlist (rule,cloak,reason) VALUES ('\\beurogenes\\.blogspot\\.\\w{2,4}','wikisource/billinghurst',)).
<wm-bot> Change on meta.wikimedia.org a page User:COIBot/XWiki/eurogenes.blogspot.nl was modified, changed by Billinghurst link https://meta.wikimedia.org/w/index.php?diff=22360114 edit summary: /* Discussion */ Closed
<sDrewth> link nr search eurogene
<LiWa3_1> Searching linkwatcher_noredirlist (nr) for rules matching eurogene gives no results
<sDrewth> link nr search eurogene.*
<LiWa3_1> Searching linkwatcher_noredirlist (nr) for rules matching eurogene.* gives no results
<sDrewth> link nr search .*eurogene.*
<LiWa3_1> Searching linkwatcher_noredirlist (nr) for rules matching .*eurogene.* gives no results
<sDrewth> LiWa3_1 link nr search \beurogenes\.blogspot\.\w{2,4}
<LiWa3_1> Searching linkwatcher_noredirlist (nr) for rules matching \beurogenes\.blogspot\.\w{2,4} gives no results
<sDrewth> LiWa3_1 link nr search \\beurogenes\.blogspot\\.\\w{2,4}
<LiWa3_1> Searching linkwatcher_noredirlist (nr) for rules matching \\beurogenes\.blogspot\\.\\w{2,4} gives no results
<sDrewth> LiWa3_1 link nr search \\beurogenes\\.blogspot\\.\\w{2,4}
<LiWa3_1> Searching linkwatcher_noredirlist (nr) for rules matching \\beurogenes\\.blogspot\\.\\w{2,4} gives no results

I am not managing to get the search right for the noredir components. <shrug>  — billinghurst sDrewth 02:26, 21 November 2021 (UTC)

@Billinghurst: Nah, I'll put it on the todo list to have a look at. --Dirk Beetstra T C (en: U, T) 04:56, 21 November 2021 (UTC)

adsense analysis going wild <=> web.archive.org

The xwiki reports of COIBot has kicked up so that reports are now three times the size that they were see classicalmusicianstoza.blogspot.ca history. Seeing this on predodminantly blogspot reports, though am seeing it elsewhere too, and seems to be related to 207.241.237.3 Special:WhatLinksHere/User:COIBot/LinkReports/207.241.237.3 which is the wayback machine web.archive.org.  — billinghurst sDrewth 11:43, 27 November 2021 (UTC)

Probably happens because the archive has the original website's AdSense, and then that gets assigned to archive.org. There is some data in there, but maybe I should code against storing adsense for archive.org (we may have that already). Please put it in User:COIBot/Wishlist and maybe I will get to it at some point. Dirk Beetstra T C (en: U, T) 10:41, 28 November 2021 (UTC)
@Billinghurst: I deleted 10588 rows of adsenses ... but I am afraid there are more to kill. Dirk Beetstra T C (en: U, T) 12:19, 14 December 2021 (UTC)
Retrieved from "https://meta.wikimedia.org/w/index.php?title=User_talk:Beetstra/Archives_2021&oldid=23920033"