Songwriting competition judging mechanisms

Discuss the many little competitions/projects that spring up amongst the Song Fight community.
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Songwriting competition judging mechanisms

Post by Lunkhead »

There must be a lot of ways to judge songwriting competitions, I bet.

One way that is familiar to many folks around here is the Nur Ein way. The judges each rank the songs in a round individually, most liked song with the highest number ranking, least liked song ranked with a 1. Then the rankings from all the judges are summed together to determine the final ranking.

This has the advantage of being efficient in that the judges do not have to coordinate or communicate at all. They just need to listen to the songs and rank them on their own and submit their rankings. In a situation where that has to be done with 48 hours, that efficiency is important.

Polarized opinions can result in very interesting results, though. Someone can win without any judge ranking them as the winner. And someone can be eliminated without any judge ranking them as being eliminated.

Someone compared this to Olympic judging. You would not be likely to see, for example, a panel of judges there be split the way the Nur Ein judges often are, giving the same gymnastics routine a 10, a 10, and a 1. (Apparently that style of scoring gymnastics got thrown out 10+ years ago but the point isn't about the actual numerical values of the scores but about their relative consistency.)

Sometimes I've thought it would be great if in a tournament like Nur Ein the judges had to collectively pick the winner and the eliminated participants in each round. It would take more time because folks would have to confer to the point of agreement, and it would be more difficult because there'd have to be some compromising and cooperation involved to reach consensus. It's likely not practical to do in the 48-hour time span with judges spread around the country or world.

I'm mostly interested in this right now in the context of an idea I had of trying to figure out a way to bring some of the intensity and participants from Nur Ein over into Song Fight, by coming up with some kind of unofficial "meta" tournament layer on top of Song Fight.
User avatar
Geech
Somebody Get Me A Doctor
Posts: 133
Joined: Mon Mar 18, 2019 9:08 am
Instruments: Guitar, Vox, Drums, Keys, Bass, Handclaps
Submitting as: Geech Sorensen
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Geech »

Seems like you're asking two different things - one is how to arrive at more consistency for judging in general and another to bring over more enthusiasm to SF.

I think a conference for judging is unrealistic for the reasons you state, but I also think it could be bad because some judges will be better advocates for their choices simply because they are better at arguing. For something like Nur Ein, the rankings are irrelevant for the purposes of the competition. All you need is a winner and a list of those eliminated. The rankings help arrive at that, true, but for those who do not win and are not eliminated it doesn’t matter. (Now, I actually like the rankings regardless as a listener, as they are generally accurate, with better songs ranked higher.)

I think any scored judging should consider throwing out the top and bottom scores for each submission. This would reduce some of the crazy disparity seen in Nur Ein. I would then keep the rankings. As an artist, it’s good to understand what the judges are looking for. I would require the judges to submit reviews for each submission, either in written or podcast/oral form.

For SF, in comparison to NE, I think that because SF is on-going vs NE being once per year and that it is elimination-oriented over weeks just make it more intense. I also think that having judges makes it seem more serious than whatever SF members and randos over the internet that can vote on in SF.

For SF, maybe there should be two tiers of voting
1. As it is done today, anyone on the internet could vote for any two+.
2. Any SF member could rank them best to worst (either precise rankings or, perhaps more simply, in three tiers), provided she includes written or podcast reviews to explain.

And the voting in the second tier would be worth some amount more than the first. This would have the additional benefit of weighting the voting towards SF members instead of people's random friends and family on the internet.
/// --- geech.com --- ///
User avatar
Niveous
Beat It
Posts: 7177
Joined: Mon Sep 27, 2004 6:45 am
Instruments: vocals, songwriting, guitar
Submitting as: Lucky Witch and the Righteous Ghost
Pronouns: He/him
Location: Staten Island, NY
Contact:

Re: Songwriting competition judging mechanisms

Post by Niveous »

Geech wrote:
Thu Jun 06, 2019 10:47 am

For SF, maybe there should be two tiers of voting
1. As it is done today, anyone on the internet could vote for any two+.
2. Any SF member could rank them best to worst (either precise rankings or, perhaps more simply, in three tiers), provided she includes written or podcast reviews to explain.

And the voting in the second tier would be worth some amount more than the first. This would have the additional benefit of weighting the voting towards SF members instead of people's random friends and family on the internet.
Interesting. It's the Rotten Tomatoes way. 1 is the Audience Score, 2 is the Critical Consensus.
"I'd like to see 1984 redubbed with this in the soundtrack."- Furrypedro.
NUR EIN!
X-Tokyo
Lucky Witch and the Righteous Ghost
User avatar
ken
Hot for Teacher
Posts: 3870
Joined: Sat Sep 25, 2004 6:10 pm
Instruments: Guitar, bass, drums, keys
Recording Method: MOTU 828x, Cubase 10
Submitting as: Ken's Super Duper Band 'n Stuff
Pronouns: he/him
Location: oakland, ca
Contact:

Re: Songwriting competition judging mechanisms

Post by ken »

I think it would be interesting to create some kind of rubric for judging and see where that gets you. I'm not sure what exactly that would look like, but maybe something like a Creative Writing course rubric you can find online.
Ken's Super Duper Band 'n Stuff - Berkeley Social Scene - Tiny Robots - Seamus Collective - Semolina Pilchards - Cutie Pies - Explino! - Bravo Bros. - 2 from 14 - and more!

i would just like to remind everyone that Ken eats kittens - blue lang
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

Yeah, I sort of broached two subjects but I only wanted to mention the topic of "[how] to bring over more enthusiasm to SF" to provide context for why I'm curious about "how to arrive at more consistency for judging". I don't want to actually tweak Song Fight itself in any way. And I think discussing what motivates people more to participate in Nur Ein than Song Fight is a whole other topic to try to delve into in a separate thread.

Tossing out the top/bottom rankings is an interesting idea for sure. And we can look at the Nur Ein results and apply that retroactively to see how that would have changed the outcomes of rounds. Maybe one of stats/spreadsheet loving folks would enjoy diving into that analysis. Glenn? Glennny? Niveous? ;)

Audience vs critical evaluation is also interesting and definitely has precedents in not just Rotten Tomatoes but many shows of the reality competition type that if I recall correctly originally inspired Nur Ein. I think unfortunately the Song Fight votes are way too easily gamed to be used in a contest where anybody actually cared about winning, personally. But maybe we could have open voting by a larger group than a judges panel (like folks on the boards) could provide a more trustworthy signal.
User avatar
Pigfarmer Jr
Jump
Posts: 2293
Joined: Sat Mar 21, 2009 6:13 am
Instruments: Guitar
Recording Method: Br-900CD and Reaper to mix
Submitting as: Pigfarmer Jr, Evil Grin, Pork Producer, Gilmore Lynette Tootle, T.C. Elliott
Pronouns: he/him
Location: Columbia, Missouri
Contact:

Re: Songwriting competition judging mechanisms

Post by Pigfarmer Jr »

Ranked voting by board members is a great idea. How it would be weighted with a public vote or implemented code wise are all things I have a great amount of ignorance about.
Evil Grin bandcamp - Evil Grin spotify
T.C. Elliott bandcamp - T.C. Elliott spotify

"PigFramer: Guy and guitar OF MY NIGHTMARES." - Blue Lang
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

ken wrote:
Thu Jun 06, 2019 12:43 pm
I think it would be interesting to create some kind of rubric for judging and see where that gets you. I'm not sure what exactly that would look like, but maybe something like a Creative Writing course rubric you can find online.
Yeah, I was imagining trying to figure something like that out too. Something like:

https://academics.lmu.edu/spee/officeof ... plerubric/

Where you provide folks some individual dimensions along which to score aspects of each song, give them possible scores, with descriptions for each score, and have folks pick the ones that apply the best. The current gymnastics scoring system apparently now is actually the sum of two scores, one for difficulty and one for "execution" (that one starts at 10 and execution issues result in reductions to that score).

https://www.bustle.com/articles/177091- ... han-before
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

"rubric for assessing songwriting" returns some potentially interesting material:

https://www.rcampus.com/rubricshowc.cfm ... ode=P95A79&

https://www.slideshare.net/EmilynMapalo ... ent-rubric
owl
Panama
Posts: 974
Joined: Mon May 15, 2017 1:29 pm
Instruments: vox keys uke guitar
Recording Method: REAPER
Submitting as: Vowl Sounds, miscellaneous owl
Pronouns: she/her
Location: Madison, WI
Contact:

Re: Songwriting competition judging mechanisms

Post by owl »

This is an interesting discussion. On the one hand, I like the idea of a rubric leading to more consistent grading, but trying to come up with what constitutes "good" in this context seems really tricky and arbitrary.

e.g. one of those rubrics has this requirement for song structure: "There must be at least TWO separate verses to it, the chorus must have at least four lines to it." Extra points for three or more verses!
I don't even know how many great songs there have to be out there that don't fit this boring metric. (Plenty of songs do, of course. But plenty don't, too.)

It might make more sense in the context of the Nur Ein challenges, which are definitely already arbitrary and kind of mysterious. But for general judging it just seems like you might end up with a lot of formulaic or box-checking songs floating to the top. Whereas a song that's great but breaks rules or where it's not easy to articulate what you like about it might end up getting thrown out.

I'd like the throwing out the top and bottom scores idea better if there were more judges involved. It seems like as a judge maybe you'd then rank your actual favorites high-but-not-highest in an attempt to protect them? And mediocrity might be rewarded? Or a judge whose tastes don't align with the others' would have their vote voided out more often than not? My husband has a theory that the most polarizing movies on Rotten Tomatoes are really the best ones, he's always way more interested in movies with a bunch of high and low scores than ones with a consistent fairly high score.

The difficulty + execution thing is interesting, maybe scoring on different broad facets of a song would work--not necessarily with strict rules about what constitutes "good" for that category, to give judges some freedom, but individual scores for different aspects to take into consideration plus a "personal ranking" for songs that defy categorization, or are more than the sum of their parts, or particularly speak to a judge for one reason or another?
  • music and structure
  • lyrics and melody [not always applicable?]
  • arrangement and production [how do you rank a G&G song against EDM?]
  • performance
  • etc.
but what is important? Originality? Catchiness? Emotion? Listenability? There are so many good songs that would break any of those metrics too.

On the other hand I guess people may be unconsciously scoring on these types of metrics anyway and this might just make it more transparent.
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

e.g. one of those rubrics has this requirement for song structure: "There must be at least TWO separate verses to it, the chorus must have at least four lines to it." Extra points for three or more verses!
I don't even know how many great songs there have to be out there that don't fit this boring metric. (Plenty of songs do, of course. But plenty don't, too.)
Hm, it's possible I misread or too quickly glossed over those two rubrics. Doh! Checking more closely now I guess that first one is more specifically prescriptive than I thought. That's not entirely what I had in mind. I was more thinking about something like the second one where the descriptions for the scores are more qualitative than quantitative.
On the other hand I guess people may be unconsciously scoring on these types of metrics anyway and this might just make it more transparent.
Yeah, that to me seems like a big potential benefit. That and possibly some consistency. I was thinking a rubric could be helpful in two ways, particularly for judges who maybe aren't trained in music or aren't otherwise musically very technical, or maybe are just music lovers and not musicians. They could help to get judges to think about dimensions like performance and arrangement more explicitly than they might have otherwise. And they could help to give some suggested general/non-technical descriptions for scores in categories to help judges match up their feelings about a song with scores. In a sense they'd provide some guide rails and sign posts to attempt to get judges to consider songs in more ways and in more depth.
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

I'd be curious about some specific examples folks have too of great songs that break metrics of "Originality Catchiness Emotion Listenability" or otherwise "break the rules". But that's probably a whole other thread's worth of a topic. I have definitely thought of a lot of songs that I love and think are great and been like "That would be eliminated in Round 1 of Nur Ein", lol. I'll have to think of some examples later.
User avatar
grumpymike
Push Comes to Shove
Posts: 418
Joined: Sun Apr 17, 2016 3:12 pm
Instruments: Rage and curmudgeonry
Recording Method: Zero-turn lawnmower
Submitting as: Grumpy Mike

Re: Songwriting competition judging mechanisms

Post by grumpymike »

Not to be a downer, but what’s different about this time we all think of ideas that take effort to implement?

Honestly, if the goal is to have somewhat understandable Nur Ein rankings, the easiest thing to do is ban mastering and normalize all of the tracks before handing them to the judges.

If the goal is to have better rankings in general, everyone should be required to judge every X Nur Eins. Perhaps extend the round duration such that judges can put more thought in. How many times have you heard “I would have ranked this one higher if I could go back...”

To make Song Fight more involved, IMO the fights need to have consequences, the titles need to change on time, and the web site needs to join the 90s to foster engagement. On the first note: I think a decaying scoreboard would be great. Something where a good solid 3 entries in a month will put you on the leaderboard.
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

grumpymike wrote:
Thu Jun 06, 2019 4:41 pm
the easiest thing to do is ban mastering and normalize all of the tracks before handing them to the judges.
Ha! I was just saying at BSS practice that there everybody should have to use the free YouLean loudness meter plugin and master their tracks to a specification. I don't think that would really changes things much, it would just make things nicer for everyone involved. Judges wouldn't have to reach for the volume knob between songs, participants wouldn't have to read/hear judges complain about that, etc. ;) It might be helpful in terms of maybe setting the specific loudness range a bit lower than the most extreme loudness some folks used, to give everybody's ears a break.
User avatar
grumpymike
Push Comes to Shove
Posts: 418
Joined: Sun Apr 17, 2016 3:12 pm
Instruments: Rage and curmudgeonry
Recording Method: Zero-turn lawnmower
Submitting as: Grumpy Mike

Re: Songwriting competition judging mechanisms

Post by grumpymike »

Lunkhead wrote:
Thu Jun 06, 2019 4:55 pm
I don't think that would really changes things much,
From what I’ve seen in judging, yes, certain judges are affected differently and this year I think a lot of songs rode past-or got cut- on mastering. I think that’s part of the beautiful game that makes everything a little more random. But if you want to reduce randomness, easy win.
User avatar
Chumpy
Twilight Sparkle
Posts: 691
Joined: Sat Apr 18, 2015 2:06 pm
Instruments: Vocals, guitar, bass
Recording Method: Logic
Submitting as: Jerkatorium, Chumpy
Pronouns: he/him
Location: Seattle, WA
Contact:

Re: Songwriting competition judging mechanisms

Post by Chumpy »

grumpymike wrote:
Thu Jun 06, 2019 4:41 pm
Honestly, if the goal is to have somewhat understandable Nur Ein rankings, the easiest thing to do is ban mastering and normalize all of the tracks before handing them to the judges.
I think "easiest" may not be the right word here. You might want to consider tackling low hanging fruit like id3 tag normalization before you set your sights on loudness normalization. I do think it would be cool though. You know what else would be cool? Artist accounts and mp3 uploads.
"I don't recommend ending on a bad joke." --ken
User avatar
grumpymike
Push Comes to Shove
Posts: 418
Joined: Sun Apr 17, 2016 3:12 pm
Instruments: Rage and curmudgeonry
Recording Method: Zero-turn lawnmower
Submitting as: Grumpy Mike

Re: Songwriting competition judging mechanisms

Post by grumpymike »

Chumpy wrote:
Thu Jun 06, 2019 5:21 pm
I think "easiest" may not be the right word here. You might want to consider tackling low hanging fruit like id3 tag normalization before you set your sights on loudness normalization. I do think it would be cool though. You know what else would be cool? Artist accounts and mp3 uploads.
I wasn't even talking loudness so much as banning contestants from using a whole mastering FX chain. I agree all of that would be nice to have, though I don't think that solves Sam's perceived "randomness" in the judging. The randomness is a factor of different people noticing different things and weighting them differently - even if unconsciously. So just start eliminating factors! No mastering, no samples, only 2 instruments, only English languages, under two minutes, only an approved subset of words, only 8/8... Hey even the freestyle olympic events have restrictions. I'm being pretty tongue-in-cheek here.

If we want to talk collaborating on code in general, which would be helpful for progress, I think most of the interested parties have discussed it before and turned up their noses at the programming language/priorities/etc. :)
User avatar
MicahSommer
Push Comes to Shove
Posts: 411
Joined: Tue Dec 13, 2011 3:09 pm
Instruments: Voice, accordion, keyboards, guitar, euphonium
Recording Method: StudioOne Artist / AudioBox USB
Submitting as: Micah Sommersmith, All The Robots
Pronouns: they/them or he/him
Location: Wisconsin, USA
Contact:

Re: Songwriting competition judging mechanisms

Post by MicahSommer »

grumpymike wrote:
Thu Jun 06, 2019 4:41 pm
If the goal is to have better rankings in general, everyone should be required to judge every X Nur Eins. Perhaps extend the round duration such that judges can put more thought in. How many times have you heard “I would have ranked this one higher if I could go back...”
To me, the short turnaround for judges' rankings is the craziest thing about Nur Ein. SpinTunes gives a week to submit entries and then the judges have a week to rank and review before the next prompt is even posted. Not that Nur Ein has to turn into SpinTunes, but it lets the judges be more thoughtful in their rankings and the participants usually end up with more extensive feedback.
"you did a skillful job pulling off the sexy" - RangerDenni
User avatar
Chumpy
Twilight Sparkle
Posts: 691
Joined: Sat Apr 18, 2015 2:06 pm
Instruments: Vocals, guitar, bass
Recording Method: Logic
Submitting as: Jerkatorium, Chumpy
Pronouns: he/him
Location: Seattle, WA
Contact:

Re: Songwriting competition judging mechanisms

Post by Chumpy »

grumpymike wrote:
Thu Jun 06, 2019 6:31 pm
If we want to talk collaborating on code in general, which would be helpful for progress, I think most of the interested parties have discussed it before and turned up their noses at the programming language/priorities/etc. :)
I had envisioned a system where you upload your mp3 to to whatever music contest site, and it runs some LUFS analysis software on it and based on what it discovers either accepts, rejects, or offers to "fix" it. This hypothetical system would know when the deadline for submission is and would let your submit/replace your song as many times as you like prior to the deadline.

I don't know if you've ever used ffmpeg, but it's an amazing Open Source audio and video manipulation library. I use it to automatically generate little mp3 chapter files based on the chapter metadata in our finished podcast mp3 so I can make links to individual chapters. It also has a module called "loudnorm" which can both collect detailed loudness stats on a given audio file, and can also normalize your audio to specific parameters. All the hard code is written, it just needs to be glued together by whatever scripting language you like.
"I don't recommend ending on a bad joke." --ken
User avatar
grumpymike
Push Comes to Shove
Posts: 418
Joined: Sun Apr 17, 2016 3:12 pm
Instruments: Rage and curmudgeonry
Recording Method: Zero-turn lawnmower
Submitting as: Grumpy Mike

Re: Songwriting competition judging mechanisms

Post by grumpymike »

Chumpy wrote:
Thu Jun 06, 2019 7:13 pm
I had envisioned a system where you upload your mp3 to to whatever music contest site, and it runs some LUFS analysis software on it and based on what it discovers either accepts, rejects, or offers to "fix" it.

Yeah, I'm looking at ffmpeg for exactly that and some other related purposes at this very moment. That's the easy part. The annoying part that I'm not interested in tackling - but have frequently thought about - is prioritizing building out a user system.
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

Maybe a new thread would be a good place for discussing code and ffmpeg and stuff that's not about how to judge songs ... ? I also don't really think any of that stuff would/should matter that much in judging, I would hope. If any judge were like "Great song, but, this other one that I liked just as much had mp3 art and yours didn't, so, I ranked you lower" they should be immediately fired, imho. :P
User avatar
Chumpy
Twilight Sparkle
Posts: 691
Joined: Sat Apr 18, 2015 2:06 pm
Instruments: Vocals, guitar, bass
Recording Method: Logic
Submitting as: Jerkatorium, Chumpy
Pronouns: he/him
Location: Seattle, WA
Contact:

Re: Songwriting competition judging mechanisms

Post by Chumpy »

Lunkhead wrote:
Thu Jun 06, 2019 7:42 pm
If any judge were like "Great song, but, this other one that I liked just as much had mp3 art and yours didn't, so, I ranked you lower" they should be immediately fired, imho. :P
Heh, stranger things have happened. It's a pretty well established notion that things that are louder sound better. That's why it's so import to level match plugins when you're A/B testing mix changes. Having contest songs adhere to loudness standards can help level the playing field. I wouldn't be surprised if BYD lost points in some judge's estimation due to their quiet mixes.
"I don't recommend ending on a bad joke." --ken
User avatar
Lunkhead
You're No Good
Posts: 8106
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

I wasn't contesting that part, more the stuff about ID3 tags.
Post Reply