User talk:Beetstra/Archives 2019

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Archive This is an archive of past discussions. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.

monitoring of problematic server /24?

Hi. Seasons greetings. I know that I can monitor an IP address at a time, however, is there the means to monitor an IP range? I can see some spambots have a nice target range in and I am uncertain the best way to keep an eye on these, beyond monitoring each single IP address. Ideally I would love to kill these early and without intervention, though I have no idea how we can do that beyond blacklist on sight. Thanks for all you do.  — billinghurst sDrewth 01:50, 25 December 2018 (UTC)

Another problematic is the host range, see some at User:COIBot/XWiki/  — billinghurst sDrewth 11:18, 13 January 2019 (UTC)
Hmm the last is just spotty, though it probably is server farm that has a range spamservers in among others, and they have a range of spam domains hosted among others. Is this something that is a negative count, though not necessarily a complete black mark? :-(  — billinghurst sDrewth 11:34, 13 January 2019 (UTC)
@Billinghurst: didn't we have a possibility to check usernames .. You can probably just check for editors who have a conflict-of-interest with '209.124.86' and '85.13.141'... --Dirk Beetstra T C (en: U, T) 12:25, 13 January 2019 (UTC)
spambots, so disposable one-use usernames. :-(  — billinghurst sDrewth 13:34, 13 January 2019 (UTC)

Command alias

Hi and happy new year. Would it be possible to create quickreport as an alias for quickcreate. I often end typing quickreport and noticing that it is not a command understood by the bot (my fault I know). Thanks for your help. Best regards, —MarcoAurelio (talk) 16:27, 19 January 2019 (UTC)

@MarcoAurelio: I will do that when I have time. There a couple of things that I need to work on when I have time. —Dirk Beetstra T C (en: U, T) 17:34, 19 January 2019 (UTC)
There's no urgency at all. Thank you. Best regards, —MarcoAurelio (talk) 17:37, 19 January 2019 (UTC)

What can we do to prevent problematic IP webserver targets?

Looking at

just shows spam homes where our spambots point. Do you have a better idea of how we can do some prevention rather than just maybe having a better target to check?  — billinghurst sDrewth 05:14, 9 February 2019 (UTC)

@Billinghurst: for starters, poke the ip of the server in a LinkSummary template, if it is not too much, COIBot will list all added domains on that server IP. Second, you may want to look for a reverse IP lookup service and collect all domains on that server, and preemptively blacklist them. (and maybe look at the whole /24 of IPs). Pinging User: MER-C, I guess he has more techniques. —Dirk Beetstra T C (en: U, T) 04:08, 10 February 2019 (UTC)
Thanks Beetstra. The whole range isn't an issue, it looks like it is just these two from my review, though the lag of COIBot makes it a little trickier for the while. I had pushed xwiki reports, though from IRC it seems that push failed for IP addresses, I will try again. I will consider the pre-emptive blacklisting.  — billinghurst sDrewth 10:01, 10 February 2019 (UTC)
@Billinghurst: I have just poked them. Bot seems to be busy with something else .. --Dirk Beetstra T C (en: U, T) 10:18, 10 February 2019 (UTC)
Yep it seems to spit out reports in batches, I was presuming that it has quiet times between imports, and spat out the reports. It is fine, it makes me do other things. FWIW I have asked Maggie Dennis if the WMF still has some domain lookup subscriptions available, they did years ago. Noting that IP Address Reverse IP 9,346 websites use this address.  — billinghurst sDrewth 10:29, 10 February 2019 (UTC)

Linkwatcher reporting issues

Hi. LiWa3s are reporting Started linkanalysers - None seemed to be alive

Status is reported as …
<LiWa3_2> Syslogs: diffreader: 8940 secs. linkanalyser: 69 secs. linkparser: 225 secs. linkreporter: 5327470 secs. linkwatcher: 261 secs. output: -1 secs. script: -1 secs.
<LiWa3_3> LW: 165 days, 07:54:53 hours active; RC: last 34 sec. ago; Reading ~ 830 wikis; Queues: P1=0; P2=1; P3=0 (738 / 16759); A1=83; A2=106 (0 / 0); M=0 - In LinkWatcher edit-backlog: 0 files (0 lines) and in analyser backlog: 330 files (82536 lines).

It is collecting a few files. Thanks.  — billinghurst sDrewth 02:54, 16 February 2019 (UTC)

and d'oh, no COIBot in IRC  — billinghurst sDrewth 10:11, 16 February 2019 (UTC)

I hope I remember tomorrow morning to solve it. —Dirk Beetstra T C (en: U, T) 17:34, 16 February 2019 (UTC)

I see reports about the labs servers appear resolved, XLinkBot and UnBlockBot have reppeared a few days ago. Can we speculate on the time for recovery of COIBot and linkwatchers?  — billinghurst sDrewth 23:28, 22 February 2019 (UTC)
Something is back and the wheels are starting to rumble, though there does appear to be an issue with monitor list regex. ... see User:COIBot/XWiki/ I am not near IRC to be able to investigate.  — billinghurst sDrewth 03:31, 25 February 2019 (UTC)
@Billinghurst and MarcoAurelio: I have restarted coibot and linkwatcher on the new servers this morning (with less memory requested than I did on the old server - they might crash regularly now). The new servers have newer software, and some of the hardcoded regexes (the ones filtering the links) needed to be updated as the new perl seems more strict on regexes. As far as I could see, all necessary modules are however running fine.
The regexes that you mention, SDrewth, seem to be in the db. Those regexes are however obviously broken (missing a 'b'?). We would need to try and track those down and replace them. --Dirk Beetstra T C (en: U, T) 05:17, 25 February 2019 (UTC)
To restate my IRC comment, I searched for these in monitor list (per SWMT IRC bot guidance) and couldn't find them, though I couldn't find anything in any search of monitor list, so wonder what else I should/can be doing to resolve.  — billinghurst sDrewth 21:57, 25 February 2019 (UTC)
I’ll try to access from console one of these days. No big deal. —Dirk Beetstra T C (en: U, T) 03:19, 26 February 2019 (UTC)

Not sure what COIBot is currently doing. It will be fed, it will churn through its food, then it doesn't submit its homework. It is still in wiki and irc, and will do a quickcreate, though that seems all.  — billinghurst sDrewth 11:46, 1 March 2019 (UTC)

Movement. A few reports have been generated. Will continue to watch.  — billinghurst sDrewth 07:02, 3 March 2019 (UTC)
@Billinghurst: I guess it has been busy backparsing some blacklistlog .. --Dirk Beetstra T C (en: U, T) 07:57, 3 March 2019 (UTC)
(the thing is .. I did not touch it .. unlikely that it is doing the same now, and it is again MIA for 2 hours). --Dirk Beetstra T C (en: U, T) 08:13, 3 March 2019 (UTC)
It didn't seem to be backparsing as then it accumulates reports as waiting (!backlog shows this), whereas on this occasion they were being processed through, though not writing to the wiki. Who knows!  — billinghurst sDrewth 12:38, 3 March 2019 (UTC)
Back to quirkiness. Reports not being written to meta, quickcreate starts report though no follow-up. IRC shows things queued and apparently ticking down, though nothing to meta.  — billinghurst sDrewth 10:42, 7 March 2019 (UTC)
I am curious why this happens, it seems to work in bursts. It has a lot of quirks now on the new server. Will have to dedicate some time soon on it .. --Dirk Beetstra T C (en: U, T) 11:08, 7 March 2019 (UTC)
FWIW this is what is queued
<COIBot> 9 records waiting: 8 XWiki, 1 Local, 0 Redirect, 0 Poked, 0 Meta, 0 IP, 0 requested
<COIBot> Waiting:,,,,,,,, koprubas
 — billinghurst sDrewth 11:36, 7 March 2019 (UTC)
and now quickcreate fails. COIBot is mute when the page does not exist and does not queue the page, though it does responds to the command when the target page exists. Report command queues successfully though nothing writes.  — billinghurst sDrewth 10:03, 9 March 2019 (UTC)

For IRC, is COIBot far away? It disappeared from IRC a few days ago.  — billinghurst sDrewth 09:57, 18 March 2019 (UTC)

Guaranteed! When you ask IT support it suddenly works perfectly. IOW COIBot is back.  — billinghurst sDrewth 10:17, 19 March 2019 (UTC)
@Billinghurst: From tomorrow I have a bit more time to look and to follow the logs. The bot quite often seems to think it is logged out, and for some reason it does not want to update the linkreportlist. Then it seems to crash from nowhere. I suspect some memory issue on the new server. (and as a side, I hope to have some time to work on some old requests/bugs and work on the spamblacklistlog-parsing). --Dirk Beetstra T C (en: U, T) 10:23, 19 March 2019 (UTC)


You created your filter at enWP and tested here. Get your own your tricks ;-)  — billinghurst sDrewth 12:22, 20 March 2019 (UTC)

@Billinghurst: No, the filter is for mainspace-en.wikipedia only. You could consider to make it a global filter if we can't get the blacklist to work. Your last edits to the blacklist did not seem to work (or did you tweak them further?). --Dirk Beetstra T C (en: U, T) 12:32, 20 March 2019 (UTC)
The edits worked fine, though if you test plain text addition, rather than a url, one's expectation that it will work are exaggerated! special:log/spamblacklist/billinghurst shows success.  — billinghurst sDrewth 12:35, 20 March 2019 (UTC)
@Billinghurst: I tested it as well, it is dead now. I noticed that COIBot is decoding the url, and does not search appropriately in the db then. Maybe I need to have a look at that as well. But first lets see if I can fix the saving issues with COIBot that it has lately. Will be online in a bit. --Dirk Beetstra T C (en: U, T) 12:39, 20 March 2019 (UTC)