500 Internal Server Error triggered by specific text string

[Sober]

I posted this in discord, but I'll post it here and clean up the discord mess.

When posting a long set of reviews, I kept getting the 500 Internal Server Error whenever I'd hit preview or submit. By trial and error, I trimmed down the post to find that a text string that consistently gives the error. I can't paste it here obviously, even in code tags, so here's an image:



In case imgur is blocked/unavailable for anyone trying to read this, you can recreate the error by typing a semicolon, a space, the word "more," a space, and the letter t. Maybe other, similar combinations will also return a 500 error. Who knows?

Try it yourself!

[Lunkhead]

fluffy identified this issue already. There is some security software installed (the "ModSecurity" module for the Apache http server) which is matching posts against patterns and incorrectly flagging harmless snippets like that as security risks, then throwing the generic 500 internal server error. Specifically it thinks you're attempting to perform remote execution of a Unix or Windows command. I think the upshot is to try to avoid using semicolons, unfortunately, or at least to try to avoid following them up with any word that's also a common Unix or Windows terminal command like "more".

[Sober]

But I like semicolons; even if I don't always use them correctly; they're neat!

[fluffy]

fluffy identified this issue already. There is some security software installed (the "ModSecurity" module for the Apache http server) which is matching posts against patterns and incorrectly flagging harmless snippets like that as security risks, then throwing the generic 500 internal server error. Specifically it thinks you're attempting to perform remote execution of a Unix or Windows command. I think the upshot is to try to avoid using semicolons, unfortunately, or at least to try to avoid following them up with any word that's also a common Unix or Windows terminal command like "more".

Yes, that's it; more or less.

I think it's possible to turn mod_security off but I'd really rather not, for obvious reasons. This has always caused a bunch of edge-case problems on Dreamhost, though.

[Lunkhead]

"Yes, that's it; mo[b][/b]re or less."

I was wondering how you got away with that part, but when I quoted it I saw what you did. That's a good hack. Sober, you can also work around by putting an empty tag (e.g. [b][/b]) inside the word after the semicolon.

[fluffy]

Yeah, it's also a great way to get around the "not enough characters" error that phpBB makes.

That said I have absolutely no idea why "; more" would be a mod_security trigger. Any situation where a semicolon can wreak that kind of havoc has much bigger problems than someone being able to interactively page through a large file.

[ujnhunter]

There has to be more to it than just the ";" perhaps the regular old ":" is an offender too? I cannot for the life of me edit my Song Fight Liner Notes post (since Jan 19, 2021) without getting the 500 Internal Server Error. It has nothing to do with "refreshing the page and trying again" like the other thread suggests...

Edit: Nothing to do with ":" as editing all of the ":" out and underlining the dates instead still give the error... Even hitting "Preview" instead of "Submit" gives the error.

[fluffy]

Unfortunately I don't think dreamhost publishes their mod_security match rules anywhere, to see what might be causing the problem. I can see if I can track something down though.

[Sober]

"Yes, that's it; more or less."

I was wondering how you got away with that part, but when I quoted it I saw what you did. That's a good hack. Sober, you can also work around by putting an empty tag (e.g. ) inside the word after the semicolon.

This is awesome to know!

[fluffy]

Oh also if you're on Windows you should be able to pepper your text with invisible space characters by holding alt, typing +FEFF, then releasing alt, which is easier than typing [b][/b] a lot. There's a similar thing you can do on Mac but it's significantly more involved.

[fluffy]



[Lunkhead]

fluffy I think the error message in the logs has the regex that's being used? Or at least, it prints out some super long regex.

[ujnhunter]

After a couple hours of painstakingly editing/previewing to update my Liner Notes post... I've got it down to a couple of things where I would get the 500 Server Error. The Song Fight! & Nur Ein titles "Shutdown" & "Sleep Tight" would trigger the error in my post, until I renamed them as "Shutdxwn" & "Slxxp Tight" and then it's possible there is a limit to how "long" your post can be because once I got to my "Cover Art" section, I couldn't type a single character after "Let's Get Bloody" or it would 500 Server Error out again which I fixed by making my Liner Notes have less line breaks. Not sure if this helps, but at least I was finally able to edit my post which I hadn't been able to do in over a year because of the 500 Server Errors.

[Lunkhead]

"shutdown" and "sleep" are both common commands that would be bad to allow a hacker to remotely execute by way of an Apache instance running on a server. What's the link to your liner notes post?

[fluffy]

fluffy I think the error message in the logs has the regex that's being used? Or at least, it prints out some super long regex.

Ah, so it does. So here's one such log entry:

Code: Select all

[Tue Apr 26 13:07:04.431164 2022] [:error] [pid 230823:tid 3690573842176] [client 96.92.148.2:54914] [client 96.92.148.2] ModSecurity:
Warning. Pattern match "(?i)(?:;|\\\\{|\\\\||\\\\|\\\\||&|&&|\\\\n|\\\\r|`)\\\\s*[\\\\(,@\\\\'\\"\\\\s]*(?:[\\\\w'\\"\\\\./]+/|[\\\\\\\\'\\"\\\\^]*
\\\\w[\\\\\\\\'\\"\\\\^]*:.*\\\\\\\\|[\\\\^\\\\.\\\\w '\\"/\\\\\\\\]*\\\\\\\\)?[\\"\\\\^]*(?:s[\\"\\\\^]*(?:y[\\"\\\\^]*s[\\"\\\\^]*(?:t[\\"\\\\^]*
e[\\"\\\\^]*m[\\"\\\\^]*(?:p[\\"\\\\^]*r[\\"\\\\^]*o[\\"\\\\^]*p[\\"\\\\^]*e ..." at ARGS:message. [file "/dh/apache2/template/etc/mod_sec3_
CRS/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "294"] [id "932115"] [msg "Remote Command Execution: Windows Command
Injection"] [data "Matched Data: & \\x22Sleep found within ARGS:message: After a couple hours of painstakingly editing/previewing to update my
Liner Notes post... I've got it down to a couple of things where I would get the 500 Server Error. The Song Fight! & Nur Ein titles
\\x22Shutdown\\x22 & \\x22Sleep Tight\\x22 would trigger the error in my post, until I renamed them as \\x22Shutdxwn\\x22 & \\x22Slxxp 
Tight\\x22 and then it's possible there is a limit to how \\x22long\\x22 your post can be because once I got to my \\x22Cover Art\\..."] [severity
"CRITICAL"] [ver " [hostname "songfight.net"] [uri "/forums/posting.php"] [unique_id "YmhQ6OWzycW5ndorYRXR5gAAAAE"], referer:
https://songfight.net/forums/viewtopic.php?f=11&t=12284

So apparently the string '& "Sleep' is a problem, because it might cause... Windows remote command injection? Who the fuck cares, Dreamhost isn't running Windows

[fluffy]

And ironically now I can't edit that message because I seem to have exceeded some sort of "anomaly score"

This is getting ridiculous and this is yet another reason why I don't take Dreamhost particularly seriously anymore.

[ujnhunter]

"shutdxwn" and "slxxp" are both common commands that would be bad to allow a hacker to remotely execute by way of an Apache instance running on a server. What's the link to your liner notes post?

My liner notes post is in my signature. It's fixed now as I've renamed Sleep to Slxxp and Shutdown to Shutdxwn as well as made it more compact with less line breaks so hopefully as long as Deep Throat doesn't use any more common Windows Commands in the Song Fight! titles... we should be all set unless someone else tries to reference the songs Slxxp Tight and Shutdxwn in the future.

Note: I had to modify your quote... because otherwise I got the 500 Server Error. LOL!

[Lunkhead]

Hey folks, fluffy put in some seriously heroic work engaging with our host service's support and I think thanks to that they have disabled the specific security rules that were causing this issue. I can now put this in my post without trickery and without errors:

; more or less
; shutdown
; sleep
; rm -rf /

more or less
shutdown
sleep
rm -rf /

Huge thanks to fluffy!

[fluffy]

And now we also know the magic incantation to bother them with if it starts happening again due to a new security rule in the future.

[Sober]

Go fluffy!

[ujnhunter]

Thank you fluffy! I was able to edit my post back with the original titles again. Still going to keep it in the current condensed format though I think, even though it looks like I can post the original format again now as well. Great work.

[fluffy]

Glad to help!