Feedback for the 2005 Game Chef Challenge!!

[Andy K]

Hi all- Just like the title says, I want to hear what you thought, as a participant, about this year's Game Chef challenge experience. I want to gather feedback so that we can put things into motion for next year that will improve the project overall. I can't promise that all your ideas will be incorporated (many will conflict, etc), but all of your feedback will get an ear.

If you had something you really liked, let us know. Something you didn't like, let us know, and why. We're looking for the bad mostly (so we can improve things), of course, but the good has its place too.

Also, feel free to comment on others' feedback here, too. If you think Joe Poster's feedback idea is great, chime in. If not, tell us why.

Thanks again for participating. I hope you submit feedback so that we can make things better next year!

[Andy K]

I'll begin:

1) Timeliness of picking winners. I was really disappointed and frustrated that we didn't have the winners picked on time as we had scheduled it. Next year, there needs to be a failsafe or two in place on the back end (however judging works next year) so that we can get the scores in, and the winners picked, on time.

2) Peer feedback Only. I was really impressed by the amount of feedback that the participants gave to each other. Incredible. In fact, the feedback was in almost all cases more solid than the feedback we provided as judges.

Yet, a lot of people were still getting this great feedback from their peers, and still looking up to feedback from the judges, as if it were "more real" or something like that.

Hoenstly, next year I'd love to see judge feedback removed altogether, and have a requirement that the participants have to leave each other feedback. I see something to the extent of "One Week to Design a Game, then one week of Break, then a time period of 3-4 weeks where you must read and leave feedback for 5-6 of your peers' games." Honestly, it's what a lot of you took upon yourselves anyway, and it really kicked ass.

3) Peer voting? Not so positive about this one, but if we make you submit feedback, why not have you vote for your peers as well? Have the Top Ten chosen from an aggregate list made up from peer reviews.

There's the feeling of "Well, they would just vote low for their peers so that their own game would get higher scores". But I think that, with so many systems and game mechanic designers laying around here, that we'll be able to put together a system of peer voting that takes that into consideration. I have a few ideas myself.

Let me know your ideas, your criticisms, etc.

-Andy

[hamsterprophet]

Awesome.

The Good: Everything was good. I'm serious! The ingredients chosen were fun and challenging, the peer response (and baiting...) was helpful and entertaining, and the comments from both peers and judges was just great. I really, really liked the rules restriction part of the ingredients. And, I made a great game, and I got to read a bunch of great games from other people.

The Bad: The wait was a real drag *shrugs*. It's true. Also, this isn't really a bad, but there were so many games! I still haven't read the bulk of them (sorry, sorry...) and I'm sure that was overwhelming for the judges. Again, this isn't bad per se, it just makes the processes surrounding the competition more of a bitch.

For Next Year: While I think the Peer Review requirement would be cool, I reiterate - so many games! Which also makes it hard to get behind a peer voting system. I mean, personally, I wouldn't want to vote until I read them all, so I know which I really should be voting for - which, again, is a time committment that not everyone can make.

Here's a thought on a 2-stage process. While we're all kibbitzing and informally peer reviewing, the judges are just reading, not scoring, the games. After 6 weeks we get a shortlist of 10 games, and then everyone's responsable for "officially" peer reviewing one or two, and also voting for a top 5, or something like that. The judges get votes too? Probably. Anyway, doing a positive rating system (as in, choose 5 games from the 10 and rate them 1 to 5, 1 being lowest and 5 being highest) eliminates the problem of voting everyone else's low - by voting at all, you're giving the game credit. Add all the points for each game in the top 10, highest is the winner.

Also, something I'd really like to see, but is totally up to how complex you want things to be, is recognition for different arenas. That is, maybe each game gets an Overall score, which determines the Iron Game Chef, but they also get a score for Integration of Ingredients, Playability, and (maybe) Layout. Then you would have an overall winner, but also recognition for the best Integration, most Playable, etc.

Of course, the latter could purely be a function of peer voting, while the overall winner is determined by the judges.

Whew! Lots of thoughts there, and not all of them good, I'm sure. But there ya go.

-N

[Anomaly]

[PlotDevice]

[Doug Ruff]

Great thread, Andy.

Good stuff: I'm going to agree that the extra restrictions and ingredients were a great idea, and I think this reflects in the quality of the entries this year. The overall "vibe" between contestants was good. Nikolai Volkoff was great too, even if he did pull the wool over my eyes!

But I think the most awesome thing was Tobias' Iron Game Destroyer thread. A whole thread devoted to tearing apart the entries before the judges posted their comments, and everyone got into the spirit of it. Props to Tobias for doing it, and to everyone else for getting into the spirit of it.

Bad stuff: Personally, I'm disappointed that I only reviewed 6 games, thus failing miserably in my Iron Game Reviewer boast. I think it helps me to understand what it would feel like to be a judge - it is tough to write a full game review, tougher than I thought.

I'm also a bit disappointed that we lost momentum on the forums a bit after the entries had been submitted. There's a lot more we could have said to each other, but it feels like we were just waiting for the judgement before saying anything more. I know that I stopped reviewing because it was nearly time for the results and I didn't want to to be seen to be influencing the judging. Crazy, I know, but there you go.

And I don't think the judging delay was as bad as not knowing why the judging was delayed.

What to do about it?

I'm going to say now that I think that mandatory peer reviewing as part of the contest is a bad thing. I think it's a barrier to newer entrants, or to anyone who is going to feel nervous about passing judgement on their peers.

Also, I think we'll run into issues over what constitutes a minimum standard for the review. For example, what if my review consists of "I read QuestQuest and thought it was a lot of fun! But the author spelled something wrong on page 2."

That's an extreme example, but I hope you can see where I'm coming from: if you don't set a minimum review standard, some people will be pushed for time and submit short or ill-thought-out reviews; but if you do set a standard, not only is that another barrier for contestants, someone has to judge the reviews. I think peer reviewing is still possible, but it has to be backed up by volunteer judges/reviewers.

I think that a preliminary sift is a better way of reducing the judging workload. If you can get the entries down to a shortlist of 8 games, then you can have these, and these only, judged in depth. You could even judge them head-to-head, like a playoff series!

Big question is whether the peers or the judges, or both, do the sifting. Whoever does it, I like the idea of each voter picking 5 games and awarding 5-4-3-2-1 points as the sift method.

But, and this is a big but, I only think that this works if the games have been publically reviewd before the voting starts, and the designers have had a chance to reply to the reviews.

So, here's my proposed recipe for next year's GameChef, including lots of stuff stolen from previous posters:

1) Entries are submitted as usual.
2) Every single game is reviewed in depth, either by peers or judges. Some guidelines for reviewing should be in place (ie reviews must cover X,Y,Z) and a hard core of volunteers must be in place to ensure that all games get coverage, without requiring mandatory comments from each entrant. I'd set a 3 week deadline for this bit.
3) Each review serves as a point of further discussion, also allowing the designer to respond to the review. I'd set a 1 week deadline for this bit. Judges should be watching this processs closely and starting to draw their own conclusions about the games
4) 1 week for shortlist voting, using 5-4-3-2-1 system. At the end of the week, top 8 are placed in a hat and drawn against each other, or "seeded" according to points total.
5) 2 weeks for judging the final 8: games are paired off, and each judge votes for one or other of the pair, with a brief explanation of their decision. Odd number of judges ensures no ties. Winners advance to the next round until there are only two games left for the IRON GAME CHEF FINAL. This would actually take 7 pairings, 2 days for each pairing.

So, the whole thing would take 7 weeks, but there would be constant activity, instead of just waiting for the judges. If we can get enough peer reviewers for the in-depth reviews, then all the judges have to do is watch and vote. The biggest job would actually be to oversee the peer reviewing and make sure each game gets the attention it deserves. But I think we can do this, and I'll volunteer to help out with reviewing if you'll have me.

[matthijs]

I liked

- the whole concept of working within a set time limit, with specific ingredients.
- the sideline interviewer & peer review threads.

I didn't like

- being judged without being played. Judge and reviewer feedback indicated that Charles wasn't judged on what it focuses on - game play, rules interaction. To be honest, peer review didn't do me much good this year. I get the strong feeling nobody got what it was about; next year I'll go heavy on the examples.
- not playing other people's games.
- not providing as much feedback as I'd liked for other games.

Suggestion:

- What if each contestant is teamed up with one other contestant for follow-up? That means posting in eachothers' design threads, and doing at least one playtest after your teammates' game has been submitted.

[Tobias]

This contest is supposed to be about helping each other, yes?

Other than participating in the contest format, which in its nature can be boost to creativity - I find the contribution of those judges who only read and gave a numeric result without any review (or a really short one), help or explanation, wanting.

And yes, I know how much work just reading and giving a number is. Iron game destroying is a strange thing - you've got to mentally play the game as much as you can to find the flaws - not just read the text.

The amount of intra-participant help was also low. Don't know how to fix that, though.

Also, knowing the specific guidelines by which the judges judge in advance may also be helpful.

[JudgeDelta]

I have to say I didn't enjoy judging this year and won't be doing it next. I was also disappointed with some of my fellow judges failure to get their marks in on time; I felt taht reflected badly on all of us. Particularly as the first few Judges to get their results in appeared to have put more care into their comments anyway.

Next time I think deadlines should be set with far more margin of error (three weeks for general judge comments, perhaps).

I think the decision not to require comments from the Judges was a big mistake and should be rectified if we still have Judges next year. It would have been helpful to me, as a Judge, to have known that the scores would not be made public before writing the comments - more than once I refered to the score given in the text. I also think the actual scores should have been made public.

I think contestants should be required to say which ingredients they are using in the text and where they have picked more than the minimum to state which ones of them they wish to be judged on. This would help with Judging, as at the moment one has to start out by figuring out what ingredients have been used. A case in point was Clinton's criticism of me for picking out 'Invincible' as an ingredient in City of Brass.

I do prefer an official judgement structure; however I think the decision to keep Judge identities a secret is something of a mistake. The stature of Mike Holmes is one of the things that lent the contest credibility in prior years.

While play would be advantageous in judging, I see no way that it could actually be acheived in a short time frame.

[jmstar]

Thanks for opening this feedback thread, Andy.

It's a contest, and a judge is never going to "get it" the way a designer does, so I'm not worried about that at all - I think we all took our lumps and they may not have all been warranted or accurate. That's how it works, no big deal. Look how merciless and idiotic that fortune teller lady is on Iron Chef! And those guys just stand there and suck it up.

I suggested this during the contest, but I'll reiterate - looking at the finalists, I think that, on some level, layout and art make an impact. By all means encourage people to pretty their entries up, but if it is all about the text, hand the judges plain ASCII files without identifying the authors.

The peer review - particularly the destroyer thread - was invaluable. Way more useful than the judges comments. I wish I could have gotten - and given - more. That said, I think there should be a separation. Why not have a Chef's Choice award, voted on by every participant who submitted 3+ reviews? If there is no layout/presentation restriction, add an "Iron Chef Design" to the awards.

I'm sorry Judge Delta felt bad about the way things turned out. Delta's reviews were substantial, and he/she took the time to reply to specific concerns in the forums. That was cool, and Delta should be proud of his/her efforts, at least. Thanks, dude!

--Jason