Forum instability

Let us know when something isn't working correctly, or if you find a typo. Do not post complaints or suggestions here.

Moderator: Mods

User avatar
fluffy
Boo
Posts: 9483
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Location: The Plaidlands (also, Seattle)
Contact:

Forum instability

Postby fluffy » Sun Nov 25, 2018 7:26 pm

Lately it seems that the forum is generating Internal Server Errors a lot, and I suspect this is simply Dreamhost having overallocated our server, as they tend to do. I'll see if there's anything we can do on our end but most likely we're going to need to get Dreamhost to migrate our site to a different server or something.
User avatar
fluffy
Boo
Posts: 9483
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Location: The Plaidlands (also, Seattle)
Contact:

Re: Forum instability

Postby fluffy » Sun Nov 25, 2018 7:33 pm

Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.
Lunkhead
Owney
Posts: 5647
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Location: Berkeley, CA
Contact:

Re: Forum instability

Postby Lunkhead » Sun Nov 25, 2018 7:50 pm

fluffy wrote:Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.


Ugh, that stinks. Thanks for handling that. Where would I go to see that kind of info? Is that in the boards admin/mods area somewhere or is that in the Dreamhost account admin?
User avatar
fluffy
Boo
Posts: 9483
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Location: The Plaidlands (also, Seattle)
Contact:

Re: Forum instability

Postby fluffy » Sun Nov 25, 2018 7:56 pm

Neither, I went into the server logs:

Code: Select all

[scott]$ cd logs/songfight.net/https
[scott]$ cut -f1 -d\  access.log | sort | cut -f1-3 -d. |  uniq -c | sort -n | tail -n 20
    121 162.233.200
    127 109.70.100
    131 78.46.176
    156 40.77.167
    156 54.36.148
    160 76.121.51
    181 199.249.230
    223 176.31.101
    233 167.114.1
    298 144.76.38
    301 178.63.87
    356 144.76.14
    763 52.83.219
    831 157.55.39
    928 91.242.162
    993 188.138.188
   2275 207.46.13
   4241 46.229.168
 115257 220.243.136
 115335 220.243.135


And I blocked them by adding them to the songfight.net/.htaccess file.

Anyway the forum's already feeling WAY more responsive, go figure.
Lunkhead
Owney
Posts: 5647
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Location: Berkeley, CA
Contact:

Re: Forum instability

Postby Lunkhead » Sun Nov 25, 2018 8:09 pm

Awesome, thanks again. I always wondered if there were access logs somewhere. I don't suppose we're running a log analyzer are we?
Lunkhead
Owney
Posts: 5647
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Location: Berkeley, CA
Contact:

Re: Forum instability

Postby Lunkhead » Sun Nov 25, 2018 8:20 pm

Looks like there is an analyzer running, cool.
User avatar
fluffy
Boo
Posts: 9483
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Location: The Plaidlands (also, Seattle)
Contact:

Re: Forum instability

Postby fluffy » Sun Nov 25, 2018 8:29 pm

Yeah, although it doesn't really do much useful (it's a crusty old instance of awstats that only does the same analysis that's trivial to do from the command line) and I don't know what the password to the dashboard is anyway.
User avatar
jb
Scrappy Dappy
Posts: 3591
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Location: WASHINGTON, DC
Contact:

Re: Forum instability

Postby jb » Mon Nov 26, 2018 7:17 am

Odd, I didn’t see any of the errors.
User avatar
fluffy
Boo
Posts: 9483
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Location: The Plaidlands (also, Seattle)
Contact:

Re: Forum instability

Postby fluffy » Mon Nov 26, 2018 8:42 am

They mostly happened when searching or posting. Ken first reported it a couple days ago and it seemed transitory, then I started getting them a LOT. I could dig into the logs to see when this Chinese bot swarm started; the logs show there were hundreds of machines constantly requesting every forum URL at a rapid rate, purpose unknown.

Return to “Quality Control”

Who is online

Users browsing this forum: No registered users and 1 guest