Developer Log/Archive 4

= 2011 =

December 28

 * 20:07 Seahorse: Test

October 16

 * 18:26 Krinkle: WikiaRC has been back up for a while now. Status OK
 * 18:26 Krinkle: KrinkleBot4 restarted, seems OK
 * 18:26 Krinkle: KrinkleBot4 crashed: mmap(...PROT_NONE...) failed \ Stacktrace: at (wrapper managed-to-native) string.InternalAllocateStr (int) at SWMTBot.SWMTUtils.replaceStrMax (string,char,char,int)

October 3

 * 20:27 Seahorse: All bots are properly functioning, except for WikiaRC which is missing and can not be started due to a login issue.

September 21

 * 06:29 Krinkle: cvn-srv2 bots have been started. Should all be OK again now.

September 16

 * 22:35 Krinkle: cvn-srv2 went down yesterday as a result of an internet connection drop at Alex's ISP. The server was also recently moved after which remote authentication was reset and has not been restored yet. Staff is locked out of the Bot Coordination panel. Awaiting server admin to reset authentication.

September 13

 * 00:27 Krinkle: Cluekie has it's own SGE cronsub entry now for longrunning jobs at the Toolserver (every 5 minutes, like cLogger)

June 17

 * 21:20 Seahorse: CVNBot8 was broadcasting a SQL logic error every time a list broadcast was made by another bot. The bot has been kicked from #cvn-broadcast and devoiced on #cvn-sw (MelosBot3 was voiced to replace it temporarily)
 * 00:15 Krinkle: #cvn-broadcast went crazy, caused bots to run in a loop and lag. All bots were forced quit. ts-willow bots have been started, cvn-srv2 can't be started due to control panel issue.
 * 00:15 Krinkle: Control panel on cvn-srv2 is inaccessible. Login rejects credentials.

June 16

 * 20:13 Seahorse: MelosBots started, covering #cvn-sw and bRCBots 1 & 2 started covering #cvn-simplewikis and #cvn-meta
 * 20:08 Seahorse: Login denied when attempting to login to srv2. Several bots missing or lagging severely probably due to a bad connection.

June 12

 * 13:30 Krinkle: Domainname countervandalism.net unavailable. Cause unknown. Could be a DNS problem, or domain expired.

June 8

 * 22:35 Krinkle: Cluekie started
 * 21:54 Krinkle: irc.wikimedia.org is back up. Staying tuned for possible side-effects.
 * 16:34 Krinkle: irc.wikimedia.org is down. CVNBots are okay. #wikimedia-tech is notified, but body systemadmin has responded yet.

June 6

 * 11:33 Krinkle: Toolserver maintenance is over; ts-willow bots back up
 * 10:12 Krinkle: Toolserver maintenance untill 12AM UTC. ts-willow bots may be unavailable during this time.

May 31

 * 23:07 Krinkle: CVNBot7 wasn't running on cvn-srv2. Started now.

May 27

 * 15:40 barras_: Restarted CVNbot2, CVNBot5, SWBot3, KrinkleBot4 and CVNBot14. All of them were online but did not report anything. Restarting fixed the issue.

May 24

 * 19:55 Krinkle: cvn-srv2 is back up. Lauched 22 bots from Webmin.

May 23

 * 18:57 Seahorse: srv2 bots have gone down, and the webpanel is not loading. It is assumed that the entire server is either shut off or simply not responding.

May 21

 * 18:38 Krinkle: cvn-srv2 went down again. Back up now. 22 bots online.

May 20

 * 00:07 Seahorse: srv2 seems to be functioning properly, all bots now online with the exception of WikiaRC

May 19

 * 23:17 Krinkle: cvn-srv2 went down again. Looks like internet connection drop. ETA unknown

May 14

 * 13:24 Krinkle: cvn-srv2 is back up. 22 bots restarted.
 * 13:24 Krinkle: cvn-srv2 is back up. 22 bots restarted.
 * 13:14 Krinkle: cvn-srv2 just went down. Looks like internet connection drop. ETA unknown

May 3

 * 23:25 Krinkle: CVN server back up. Bots back up.
 * 22:24 Krinkle: CVN server down. AlexZ / cbrown unreachable.

April 25

 * 22:56 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

March 30

 * 21:18 Krinkle: Toolserver was doing fine. One of the main Freenode servers failed, back up now. Toolserver hosted cvnbots are back up.
 * 16:30 Krinkle: Toolserver is having network issues.
 * 16:09 Krinkle: Cluekie is running again
 * 12:36 barras_: Cluekie lagged and I quited it then.... now I see it is on none of our servers :s Krinkle help me ;)

March 27

 * 00:32 Krinkle: Network restart. 22 bots were online. No lags before restarting. SWBot & SWBot3 didn't come back after restart, started manually. All 22 bots successfully restarted.

March 19

 * 09:33 AlexZ_: CVN-wiki fixed and upgraded to 1.17alpha (r82950)

February 25

 * 23:21 Seahorse: Issue resolved, and bots restarted.
 * 23:15 Seahorse: CVN server unreachable, webmin panel not loading, and bots are down.

February 22

 * 00:13 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

February 12

 * 02:53 Seahorse: CVN Server's bots all have disconnected for an unknown reason, bots are currently being started again.

February 11

 * 15:07 barras_: SeahorseRC2 quitted; currently running an old bot in #cvn-zh-scan from me. quiet it when the normal one comes back.
 * 13:51 barras_: Restarted KrinkleBot7. Bot didn't show every event. Restart seems to have fixed it.

February 7

 * 23:26 Krinkle: Random dropout/restart/disconnecting of bots on cvn-srv2. Processes in /home/cvn/bots/ terminated. Network restarted. 22 bots up now.

February 1

 * 02:27 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 31

 * 01:57 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 27

 * 01:05 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 24

 * 20:03 AlexZ_: .dyndns.org addressed has expired

January 23

 * 11:20 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 22

 * 18:03 Krinkle: WikiaRC was in #cvn-bots but not in #cvn-wikia. Complaints about relay bots to local channels of individual wikia wikis not working followed. A restart fixed it for now.
 * 04:53 AlexZ_: rebooted RG after becoming unresponsive and reporting various DNS errors. "ERR P0000-00-00T00:01:09 sntpc: Failed to resolve ntpserver 'ntp.ucsd.edu': Host name lookup failure."  Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 21

 * 00:54 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 18

 * 19:29 Krinkle: New channel: #cvn-wikia-zelda (Zeldapedia countervandalism ). Local op(s): Furry

January 17

 * 22:04 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.
 * 01:47 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 16

 * 22:54 AlexZ_: rebooted srv2 after 58 days
 * 13:58 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 13

 * 23:49 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.
 * 02:23 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 12

 * 01:52 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 11

 * 05:44 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 10

 * 02:57 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 9

 * 00:50 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 8

 * 03:23 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 7

 * 02:16 Krinkle: Daily network restart. 23 bots were online. No lags before restarting. All 23 bots successfully restarted.

January 6

 * 02:13 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 5

 * 00:13 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 4

 * 14:53 Krinkle: Added cLogger as longrun cronsub that auto-restarts in crontab @ krinkle @ toolserver
 * 02:08 Krinkle: Daily network restart. 23 bots were online. No lags before restarting. All 23 bots successfully restarted.

January 3

 * 04:56 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 2

 * 03:30 Krinkle: Daily network restart. 21 bots were online. No lags before restarting. All 21 bots successfully restarted.

January 1

 * 20:11 Krinkle: The new fa.wikinews is now monitored as well. Loaded in CVNBot15 (#cvn-wp-fa : Scan-channel for all Persian (fa) wikis)
 * 18:09 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

= 2010 =

December 31

 * 04:22 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 30

 * 04:06 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 29

 * 21:36 Seahorse: CVNBot10 became extremely slow and laggy due to an unbotlisted bot doing extremely rapid constant page deletions on ksh.wikipedia. The reason it may have been unlisted is because it was on the whitelist which I had to remove from it. Problem solved and bot restarted.
 * 03:53 AlexZ_: updating webgui URL to https://srv2.countervandalism.net:1568/

December 28

 * 01:45 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 27

 * 17:09 Krinkle: Setting feedFilterEventDelete:2 back to feedFilterEventDelete:4 for CVNBot14 @ Toolserver (too many deletions flood the channel, expection of requester was lower)
 * 15:55 Seahorse: CVNBot10 Started by Krinkle, was down for almost a day and failed to start first attempt.
 * 04:03 Krinkle: Setting feedFilterEventDelete:4 to feedFilterEventDelete:2 for CVNBot14 @ Toolserver (showing delete actions, per IRC request)
 * 03:48 Krinkle: Daily network restart. 20 bots were online. No lags before restarting. All 20 bots successfully restarted.

December 26

 * 16:41 Krinkle: KrinkleBot7 apparantly missed the addition of a proxyblockbot to the botlist and added 12,000 rows. Rows deleted and database 'vacuum'ed. Bot back up.
 * 16:25 Krinkle: KrinkleBot7 down for database optimalization. Keeping in the terminal for a short run to keep an eye on the debug log.
 * 16:19 Seahorse: KrinkleBot7 lately has been showing 3+ min lag, restarted twice but the problem has not fixed. This may be SWMTBot's general lag-for-big-wikis problem.
 * 10:16 AlexZ_: cvnwiki seems to be working again (check w/ webhost) & botserver needs eventual reboot to apply some updates.
 * 10:12 AlexZ_: rebooted bots
 * 10:06 AlexZ_: cvnwiki is dying 126: Incorrect key file for table '/ramdisk/mysql/tmp/#sql_2f54_0.MYI'; try to repair it (127.0.0.1)
 * 10:04 AlexZ_: Network restored after submitting reboot request to RG via UI.
 * 09:59 AlexZ_: 2010-12-26T01:39:58-08:00	lmd: ipnet0: UP on bridge1 with 75.37.205.255/22 GW:75.37.204.1
 * 09:58 AlexZ_: 2010-12-26T01:39:16-08:00	vrsip: Broadband connection lost
 * 09:22 AlexZ_: Running a broadband link test on RG, this may cause network issues.
 * 04:40 Krinkle: Daily network restart. All bots were on. No lags. CVNBot6 didn't boot back up. WikiaRC did leave but not come back. Started both manually.
 * 02:56 Krinkle: Issue with #cvn-unifications solved, SULWatcher's are back up. It was down since 01:50, 26 December 2010 according to logs.
 * 02:52 Krinkle: #cvn-unifications has been empty for a few hours. Bots seem to be functioning properly. Investigating possible relation to WMF issues earlier today and/or the toolserver replication issues because of that.
 * 00:18 Krinkle: Cluekie back up. Runs version 0.0.4-beta (latest trunk)

December 25

 * 23:55 Krinkle: Cluekie will be down for maintaince and back shortly!
 * 21:21 J-FelizNavidad: Restarted SULWatcher
 * 18:38 Krinkle: Banned Jules from #countervandalism, no known operator, not approved in the network.
 * 18:22 Krinkle: First publicly logged message
 * 17:57 Krinkle: Testing Clogger - From toolserver
 * 16:20 Krinkle: Testing with right log enabled
 * 16:04 Krinkle: Alright, we're up !
 * 00:00 Krinkle: Demo..