[OOC] Feedback for the Outplay Organizers

[Wynamoinen]

Let me suggest the following: we're bending over backwards to protect round-0 voting anonymity (which I support!), but the judges get none.

Maybe it would make sense to make JUDGES anonymous while making all their comments public. We should know who the judges are in some broad sense (just like this year, posting them in the Rules thread), but the actual comments could be posted for players (for example, by the GM) by saying they're from "judge 1" "judge 2", etc. This is exactly how quality-control happens in academic publishing, for example. It doesn't solve problems of "bias", but it does foster healthy constructive honest criticism. I wonder if judges are worried about creating feelings of bad blood by giving honest constructive feedback. It's a tough, thankless job, and no one wants to make that job more painful by pissing off some player who juts got snubbed.

I agree with others that I'd prefer all judge's comments to be fully open to all readers.

I gotta say, I'm a little frustrated that I've gotten no specific commentary from anyone but my GM (thanks Bhelogan!) As a relative newcomer to the site who is still learning the ropes, I didn't join this contest because I thought I'd win. I joined to become a better player. That was a big selling point to me. I had tons of fun playing, and I hope to play again. But I don't feel like I'm leaving with any more information than I came with, about what my areas for improvement are. I'd like to figure out how to ensure everyone gets detailed, healthy commentary. I'm pathetically jealous of the people who got Aethera's and Moozuba's detailed and thoughtful comments.

[GhostlyDaemon]

Originally Posted by Wynamoinen

I gotta say, I'm a little frustrated that I've gotten no specific commentary from anyone but my GM (thanks Bhelogan!) As a relative newcomer to the site who is still learning the ropes, I didn't join this contest because I thought I'd win. I joined to become a better player. That was a big selling point to me. I had tons of fun playing, and I hope to play again. But I don't feel like I'm leaving with any more information than I came with, about what my areas for improvement are. I'd like to figure out how to ensure everyone gets detailed, healthy commentary. I'm pathetically jealous of the people who got Aethera's and Moozuba's detailed and thoughtful comments.

I have to say as an aside I agree with this entirely - Bhelogan's comments were fantastic and really helpful but I haven't really recieved any other feedback.

[Aethera]

I know at least one of the 5e judges has been ill, but I'll check in with them to see what comments you might be able to get.

At some point in the past we did just as

Originally Posted by Wynamoinen

Maybe it would make sense to make JUDGES anonymous while making all their comments public. We should know who the judges are in some broad sense (just like this year, posting them in the Rules thread), but the actual comments could be posted for players (for example, by the GM) by saying they're from "judge 1" "judge 2", etc. This is exactly how quality-control happens in academic publishing, for example. It doesn't solve problems of "bias", but it does foster healthy constructive honest criticism. I wonder if judges are worried about creating feelings of bad blood by giving honest constructive feedback. It's a tough, thankless job, and no one wants to make that job more painful by pissing off some player who juts got snubbed.

Wynamoinen suggested, with public comments labeled only "Judge 1" not "Aethera". Pretty sure some of the comments were obvious who they came from, but just as people spent time trying to figure out who the anonymous entrants were, it would be much the same, and for the judges, it would be without confirmation. I will discuss this with the team for next year, but for this year, since it was not laid out ahead of time, we'll have to stick to making those comments public that players are comfortable having made public.

[Squeak]

Here is a suggestion -- one that I know will not be easy because it is a lot of work for the DM's

Have an optional consolation round for those who were not able to advance. Since everyone put a lot of effort into building their characters, it would be nice to use them for more than one week. It would allow those than didn't advance a chance to learn more about the competition and how to best improve for next year.

[Aethera]

With no promises offered, only curiosity and a desire for clarity, I'd like to know more about this "consolation round" of yours, Squeak. What would it entail? Roleplay only? Backtracking through the same skill challenges they had on the way in? A whole new set of "secret exit" encounters? It absolutely would be more work for the DMs, but depending on how much work you're suggesting, it might be something we could implement. Maybe. Not necessarily this year, unfortunately.

[Squeak]

My only thought is that it would be nice for everyone to play at least two rounds, particularly if one of the combat round. It would be the same adventure and hidden from others, just like the different systems are done now.

Everyone puts so much work into their character, anything we can do to extend the game for them will only make for better competitors next year.

[GeoAvanti]

I've done a great deal of peer-review in the past, and I've had a lot of peer review done to my own work for many years.

I grimace at the thought of someone posting reviews of my own works (good and bad) on an open forum for everyone else to read. I consider it a breach of personal etiquette to openly post criticisms and critiques of another person's work in public without their expressly-given permission. If that individual wishes to share that review, then they are more than welcome to do so.

I, however, strongly oppose the idea that feedback from Judges be made public to sate the curiosity of other people without the author's stated permission to do exactly that.

If a player wants to get an idea of what they could have done better in the competition, then your best bet is going to be to ask your Judges, not reading the feedback of other competitors. Proper Peer-Review (which this essentially is) ought to be a dialogue as much as a checklist, and I will not hesitate to explain myself and my decisions to anyone who asks me for clarification. Maybe I made a mistake of my own and blamed you for something I shouldn't have (which I already did once, sorry again!). Maybe your style never meshed with me, despite the quality of the writing (which also happened). There are countless reasons why the scores fell the way they did, and its very unlikely that any of the Judges gave the same scores or even the same ranks.

That's my thought on that particular matter.

[Squeak]

If it means anything, feedback was posted in secret text for the last few years -- and pretty much universally people made in public.

I definitely do not mean to detract or disagree with your opinion -- just wanted to share how it is has been done in the past.

[wodine]

Originally Posted by GeoAvanti

I grimace at the thought of someone posting reviews of my own works (good and bad) on an open forum for everyone else to read. I consider it a breach of personal etiquette to openly post criticisms and critiques of another person's work in public without their expressly-given permission. If that individual wishes to share that review, then they are more than welcome to do so.

I, however, strongly oppose the idea that feedback from Judges be made public to sate the curiosity of other people without the author's stated permission to do exactly that.

I completely disagree. Every fine arts, theater, and creative writing class I've ever taken has had open critiques for everyone to take advantage. As long as what people are saying is constructive feedback, it helps everyone grow. I think it opens up pathways for discussion that would otherwise never have been raised, because as we all know, there is no accounting for taste.

That is why I suggested adding it to the rules for next year so that participation means consent to open judging.

 

Perspective: The pros and cons of open peer review
Nature (2006) | doi:10.1038/nature04991

The main disadvantage of open review is the likely accumulation of 'enemies' who may later try to torpedo one's own manuscripts or grant applications. And this occurs, it seems, regardless of the outcome of peer review. On the other hand, anonymous review does not always stop authors harbouring resentment if their paper is rejected. In a small field, there may be only a handful of genuine experts who have sufficient experience with the techniques, pitfalls and potential artefacts of the system, and authors are often likely to know who the reviewer is.

But I do think that there are several advantages to an open peer-review system. First, reviewers would be more tactful and constructive.

Second, reviewers with a vested interest in suppressing the publication of a manuscript could be more easily unmasked by authors.

Third, a completely open review system would have reviewers' names published in a footnote to each paper to further encourage reviewers to do a thorough job.

- Dr. Tom DeCoursey professor of molecular biophysics and physiology at Rush University Medical Center.

Granted this only loosely applies but I think it would put more pressure on the judges to be impartial if everyone else was also looking over their shoulders, in addition to the advantages of seeing what has worked well for others in the past.

[Atalla Wanderer]

Originally Posted by wodine

I completely disagree. Every fine arts, theater, and creative writing class I've ever taken has had open critiques for everyone to take advantage. As long as what people are saying is constructive feedback, it helps everyone grow. I think it opens up pathways for discussion that would otherwise never have been raised, because as we all know, there is no accounting for taste.

That open-sharing is why I decided to drop those public slaughterhouse creative writing classes that forced me to share instead of allowing me to share. To some it'll be a small difference, but to me, it's important. Maybe I'm just selfish, but I'd rather other people not "take advantage" of my own errors to grow unless I shared them myself.

[Squeak]

Based on the above, I'd suggest going back to the way input was given in the past.

Each player posts their interest in receiving input. The input is put into secret text under their thread by the GMs. It is up to the players themselves to decide whether or not they make the comments about their writing public.

For those who are not familiar with how this was handled, HERE is a link to how we used to do it.

[Qiksilv]

Personally I liked being able to thank the specific judge who wrote my critiques this year and I always let the judges make my evaluations public; it doesn't bother me. That said, in previous years when the option of sharing feedback has arisen there have been people who choose not to share and I think that's important to note.

I think the default for sharing should be on private and that having a thread to allow permission should be standard procedure. It could even be set up from the very start next year so as to streamline things a little.

Allowing the judges the opportunity to be anonymous also isn't a bad idea either. As much as I love for everything to be open and available I believe for some having their "grades" public may hinder their enjoyment and even keep them from participating. Several systems barely scraped enough people together to run this year and if we discourage those who are reserved about sharing it may be even closer next year.

Tl;dr - Make it all private unless permission is given by the person it concerns to keep from making players second-guess becoming involved.

[JonnyGulliver]

I simply say that it should be an option given as there are those who would prefer to be private versus public, right in the applications for next year:

Name/Race/Class/Public?/etc...

Little more work for the admins admittedly, but it seems to be a real issue for some to warrant a response, and people seem varied in their opinions.

[driftwood]

While I would agree with some mechanism to allow players to keep their critiques private, I personally don't have a problem with mine being made public.

[TeufelHeunden]

Originally Posted by driftwood

While I would agree with some mechanism to allow players to keep their critiques private, I personally don't have a problem with mine being made public.

Agreed ^