[Wervyn]

We've been kicking this around for a couple days, and last night came to a pretty good revision of the rules governing scoring. Given that, I've been working on a more formal document detailing exactly what will be changing (or at least, what I think should be changed). Here's what I've got so far:

Proposal for New DsDoZ Scoreset

It has become apparent since the last major revision that the current scoring system has some issues. Most glaring is that the theme score, which was at one time the major portion of the score, has somehow dwindled to a nearly insignificant factor. Attempts to assign validity to the category through a disqualification rule have proven counter-productive and largely ineffective. Clearly what we need to do is return to the notion that theme is not just a way to prevent people from cheating, but is in fact the major driving force behind the competition. I no longer accept that DoZs should just be about making the best game possible in 24 hours. They should be about making the best game possible based on the given themes.

Another issue is confusion over the exact purpose the "Innovation" category is supposed to serve. Originally this had something to do with the old "Originality" category, but lately it has been used more often to award points to particularly well-pulled-off engines and other coding. I feel that given the state of the community right now, we don't need engine originality so much as we need engine effectiveness; most of what can be done in MZX has been done. Therefore I propose renaming this category to "Technique" and having it explicitly focus on the programming skill displayed in the game. Scoring for originality should be dispersed among all of the categories as a reward for its effectiveness in graphics, programming, gameplay, etc.

By the same token, I propose removing Discretion as a separate category and dispersing it in much the same way as a negative penalty in the appropriate categories. I feel that the remaining categories adequately cover any aspects that might warrant point reduction. Obscene or generally idiotic dialogue should come out of Plot (or Story, as I'd rename it; more on this in a moment), while remarkably buggy code should entail penalties in Technique or Gameplay, depending on the nature of the problem.

Finally, while the nature of the Plot category should not be changed from the way it's generally interpreted now, I feel that renaming it "Story" would more accurately capture what the category is really about. To the point, the Story category judges how well the game uses dramatic elements to emotionally engage the player, as a contrast to Gameplay which is more a measure of the effective use of formal elements (this Video Game Design course is actually paying off).

Carrying that dichotomy further, these categories lend themselves well to a conceptual split, with Gameplay and Technique judging the game's technical merits, while Graphics, Story, and Sound relate to its artistic or atmospheric merits. This allows for a more natural division of the point weight between them. Theme should account for a quarter of the points by itself, leaving the Technical and Artistic categories with three-eighths each. Gameplay and Technique I judge to be about a 3:2 ratio to each other in terms of importance. Sound is the least important of the Artistic elements and should be worth perhaps a fifth of those points, while Graphics I still judge to be more important than Story in terms of atmospheric effectiveness, again by about 3:2, especially in the current MZX climate. This suggests the following breakdown:



There was some minor adjustment to Graphics and Story to make the numbers come out evenly (50 and 70 as opposed to 48 and 72). The total number of points was also increased to 400 to facilitate this. Notice that apart from the greatly increased Theme score and the lack of a Discretion category, the relative weight of the categories is fairly well preserved; the old system had most of that right.

As for the Rank Weight, there has been some discussion that while a good concept, it currently carries too much weight. Thus, it will remain at 100 points, but percentage-wise this represents a 20% drop, due to the increased total. With the rank weight, the total score from each judge will be a nice, round 500 points.

-------------------

I'll work on writing new text for the categories, particularly for the ones that have changed, in a bit. For now, I'd appreciate any discussion or input anyone else might have on this.

"It's my way or the...well, I guess you could do it your way, too."

[Dr Lancer-X]

The new system is good.

[KKairos]

Shoot, yeah. I'd agree that innovation doesn't have to be a category, given that it probably did arise mostly out of audience expectations.

As for discretion, I've probably driven Exophase nearly insane with the number of times I've asked him about what it means and presented some paranoid theory.

[Quasar84]

Phew, hardly feels like the last major revision was four years ago. Anyway, I'll add my support for the new system too. I don't know that reducing the importance of weight rank is inherently necessary, but it does yield a much nicer total number and I don't think it'll hurt anything either.

[Seven]

I hate to be the long poster, but I will. It is the will of Godwin! Also, nazis will be involved somehow.

Reading through the proposal, I was actually quite impressed, Wervyn. As always, you did your homework and put a lot of thought into it. Now, I cannot say I think this is "good" or "bad." I think it's a great proposal, but still needs some soft tweaking and perhaps some analysis.

One point I want to bring up is the fact that Story/Plot is a category at all. As much of a games-as-a-form-of-storytelling advocate I can be, not all games are stories. Given the topic Zombies, do you need a story? And yes, it could be argued, to take this analogy further, that the story is that you want to get back to your wife and two and a half children, but it can also be argued that tetris has story in itself, to express one's inner desire for order and strength, furthering that ability. But in the end, the topic Zombies, you're just killing zombies. Story is moot. I think thought needs to be put into this, though, I think.

Second on my list is the weights of each. Staring at the graph, I feel that the weights are off somehow, and I'm sitting here trying to think of a better weighting for them.

Failing this, I'm calling Wervyn a statistics nazi for no reason, to distract from my logic (or mislogic, as it may be...) to bring this point up: Perhaps we should cut it down to three or four categories, instead of having seven or nine categories. Theme, Look and Feel, Technical Detail, seems to be the strongest sorting in my mind in reducing the scoring sections. Of course there will be disagreement with this; I'm only suggesting it as long as we're analyzing and proposing.

... nazis.

[Dr Lancer-X]

Cutting it down to fewer categories will only cause even more inconsistency in the judging than we have already!

[weasel]

I think Graphics and Sound should be merged into one category, Aesthetics - since many times the two are taken together. Aside from that, the revisions to the scoring system are very well thought out, comprehensive, and logical.

[Wervyn]

As the thinking on Plot has gone in the past, games that by the nature of their genre don't really involve a plot get judged more leniently in the category, while genres that require the game to be very plot-heavy may get a break in the gameplay category (e.g. Honor Quest type games). I see no reason to shake this up, and allow judges to apply some discretion to the case. I wanted to shift away from just calling it plot, though, because that implies some sort of narrative action going on. I guess "story" isn't really much better in that regard, but what I'm really trying to capture in the category is the use of dramatic elements within the game. One of my favorite ZZT games is myth's "winter". I would rate it highly in the story category, though it has no dialogue or even words at all. Because in the end, I think that a game that a game that both has excellent gameplay and good attention to story is necessarily better than a game that has excellent gameplay but no story.

Regarding the weights, you'll really have to be more specific. What I was really going for was to keep things mostly the same with tweaks, except for giving theme a lion's share, and that's basically how it ended up. But if you can think of a reason to change them numbers...

Also, regarding the number of categories, I really think having six fairly discrete categories better serves the judges than three general ones. Too broad, and the judging criteria for each become rather vague; I'd be worried about judges not really being sure what the categories are supposed to mean. But of course, loads of specific categories makes it hard for the judge to really get across what he feels about the game itself. I think these categories work best then because they divide the game into fairly intuitive areas that can be individually judged on their own merits. The fact that a judge going in has a fairly immediate idea of what he's supposed to be scoring on is a strength of the old system that I don't think we should abandon. The subdivisions are there to help further conceptualize the purpose of each category; trying to judge a single score for Artisitic and Technical categories alone seems like it would be too vague. Six feels like a pretty good number to me.

"Art makes Maeby feel RAINBOW, English makes Maeby feel CROCODILE, Math makes Maeby feel C-."

[asgromo]

Seven, on Nov 6 2006, 11:52 PM, said:

One point I want to bring up is the fact that Story/Plot is a category at all. As much of a games-as-a-form-of-storytelling advocate I can be, not all games are stories.

Lots of games are almost exclusively stories, and MegaZeux more than fits the bill as an effective medium in which to tell them. It's been done before, actually, and each time it got big notice, even if the two primary examples (one ancient, one relatively recent) were not my favorite games (or stories).

What bugs me is the suggestion that judgment of the quality of a game should be weighted in favor of gameplay over how generally entertaining it is. The problem is one of definition, I suppose- you get caught up in the "game" implication and you expect table tennis, not hours of technobabble and pseudo-philosophy dressed up with postmodern giant robots

Of course, I can think of movies at least as entertaining as video games. Why, hell, I can think of cutscenes in video games that I basically played the video game for. Metal Gear is one fine example. Final Fantasy is another. Maybe I'm in the minority for not playing Final Fantasy games for their battle system... I'm just saying that interactivity may be a more important element of future entertainment than the current strict dichotomy of game versus movie suggests.

Then again, when was the last DoZ title that focused on storytelling and got away with it? (Certainly none of mine =D) Past experience obviously does not warrant granting greater weight to those elements.

Clearly, if it were all up to me, there wouldn't be a categorical scoring system. I'd trust the judges to be objective because they are judges, which is because they ostensibly have used MegaZeux in the past and understand the inherent limitations and circumstances very well, even if they can't all agree on them. Someone told me that was the point of having more than one judge anyway.

I guess my point is that no matter how you apply specified systems of measurement to a medium as surprisingly versatile as this one, you're going to wind up with unusual entries that force you to fudge the results. All we can do is adjust the systems of measurement to follow the trends.

[Skylark]

wildweasel, on Nov 7 2006, 02:11 PM, said:

I think Graphics and Sound should be merged into one category, Aesthetics - since many times the two are taken together. Aside from that, the revisions to the scoring system are very well thought out, comprehensive, and logical.

I definitely don't agree with this.

Sound and graphics definitely need to be separate - obscuring this further not only makes it a pain for the judge to get through the point he is making about the visual and aural appeal in a numeric form, but also makes finding a game worthy of playing more difficult. I don't know about you, but when I'm selecting DoZ games to play, I want to be able to quickly see the scores and know what is good and bad about a game. What if a game has fantastic sound but absolutely awful graphics? Or vice-versa? You'd have to look up the review to find out. Sure this might only be a minor annoyance, but it is only one of many that I would have if these two sections were amalgamated.

Wervyn: Nice job. I understand your reasoning, and no matter what happens about this, I definitely approve of the "Technical" section over innovation. It is such a pain to try and score people for innovation on its literal meaning nowadays; last time I judged I just did what you have said the technical section is for, so why not change it?

I also agree with lowering the rank weight. I don't think 1/4 of the game should be just the judges opinion - especially not if the anonimity rule is no longer enforced. Everyone is biased no matter what they say - and even a subtle movement in rank weight can affect the outcome of the DoZ since it is worth that many points.

In summary, I like this new system. Shall we try it in the upcoming Winter DoZ? I'm interested to hear RoSS's thoughts.

[Dr Lancer-X]

asgromo, on Nov 7 2006, 06:49 PM, said:

Clearly, if it were all up to me, there wouldn't be a categorical scoring system. I'd trust the judges to be objective because they are judges, which is because they ostensibly have used MegaZeux in the past and understand the inherent limitations and circumstances very well, even if they can't all agree on them. Someone told me that was the point of having more than one judge anyway.

Being totally objective is hard. In fact, I'd say impossible. Throwing away the categories would make it even harder for them to award games appropriate scores, because then they have nothing to work against.

[Nash]

i agree with asgromo. you should just have the judges rank the games from best to worst and the games would get scores accordingly (like if there's 15 games first place gets 15 points, second place gets 14 points, 15th place gets 1 point, etc.) the results would probably come out almost the same if done with the categorical style, and that way the judges don't have to write 7 or whatever sections of reviews for like 10+ games.

as for the theme part the judges or whoever should just vote directly after the doz ends on whether or not certain games should be disqualified for theme things or not. it doesn't really have to be a separate score, let alone like 100 whole points. remember the main point of having themes in the first place is just to make sure people don't cheat and make games beforehand, it shouldn't be the primary thing the games get scored on.

[Skylark]

Nash, on Nov 7 2006, 10:09 PM, said:

i agree with asgromo. you should just have the judges rank the games from best to worst and the games would get scores accordingly (like if there's 15 games first place gets 15 points, second place gets 14 points, 15th place gets 1 point, etc.) the results would probably come out almost the same if done with the categorical style, and that way the judges don't have to write 7 or whatever sections of reviews for like 10+ games.

as for the theme part the judges or whoever should just vote directly after the doz ends on whether or not certain games should be disqualified for theme things or not. it doesn't really have to be a separate score, let alone like 100 whole points. remember the main point of having themes in the first place is just to make sure people don't cheat and make games beforehand, it shouldn't be the primary thing the games get scored on.

The point is that games shouldn't just have the title "PIRATES" and pass the pirate theme score if NONE of the game is attributable to the theme. Wervyn is right; DoZ games really need to tie the theme into every other facet for it to have correctly followed the theme. It is essentially the only way to separate good games from the bad - by how well they non only adhered to, but utilised the theme. I think more emphasis on theme is not only a good thing, but necessary.

[Wervyn]

Well you're right about one thing Nash, your games' scores wouldn't change that much. However, just taking one DoZ as a sample (2004 Christmas):

                                        Galladin     Lancer       Weasel       Astral       Total        Original
#6168 - Judgement of the Immortal       2            1            2            2            7  (1)       2 (1240)
#9532 - Run! Run! Run!                  3            5            1            1            10 (2)       1 (1330)
#4865 - Scottish adventure              1            4            7            3            15 (3)       3 (1179)
#5349 - Unearthin' Egypt                5            3            3            5            16 (4)       5  (893)
#84865 - Fear nothing                   7            2            4            7            20 (5)       4  (984)
#323 - Voyage of Fear                   4            6            6            6            22 (6)       9  (694)
#35974 - Are you afraid?                6            7            12           8            33 (7)       6  (833)
#812 - Jeff Kenny: secret agent         13           8            5            10           36 (8)       10 (650)
#7486 - Zeeb's interplanar adventure    10           9            11           9            39 (9)       7  (760)
#2004 - Afraid to forget                8            14           14           4            40 (10)      14 (512)
#17915 - They hide in the dark          11           13           8            11           43 (11)      8  (709)
#962 - Zombies                          12           11           9            13           45 (12)      11 (606)
#1799 - Dungeons Down Under             9            10           13           15           47 (13)      13 (543)
#723 - Temple o Ketera                  14           12           10           12           48 (14)      12 (554)
#548 - The infernal fear                15           15           16           14           60 (15)      15 (172)
#22947 - A game about fear              16           16           15           16           63 (16)      16 (126)
#96234 - Tha Mindkilla                  17           17           17           17           68 (17)      17  (95)

As you can see, scoring by rank weight alone would have significantly altered the outcome of the DoZ, especially in the mid-range. Your system also makes it difficult to decide on appropriate penalties for late games (which is one factor that doesn't really come across in those score differences). Finally, no one said judges have to provide detailed comments for each category, that's just something many of us have done on our own. Look at the times I didn't leave comments because I didn't feel like it. Dividing the score into discrete units still helps competitors understand and connect with where the scores came from; switching to pure rank weight would make the final decision feel extremely arbitrary. In the end, too, if we didn't provide judges with an explicit rubric, I think they would create their own out of necessity. Trying to rank as many as 20 games just based on "feel" is extremely difficult.
Also, I have to entirely disagree regarding the purpose of theme, now. Somehow, what you've described is what happened, and theme just became a way to prevent people from cheating. But if you really think about it, we aren't really concerned with people cheating much at all. And besides, it doesn't help that much: if someone really wanted to make a game before the competition, it would be fairly simple to rework the game elements and the plot enough to incorporate the theme, given 24 hours to do so. Much less work than creating the game itself. I think that in relegating theme to a pittance of the score, we've really lost touch with the idea of DoZs as THEMED competitions. I'd really like to get that feeling back.

"Sou...datoomou."

[Exophase]

Wervyn said:

Most glaring is that the theme score, which was at one time the major portion of the score, has somehow dwindled to a nearly insignificant factor.  Attempts to assign validity to the category through a disqualification rule have proven counter-productive and largely ineffective.  Clearly what we need to do is return to the notion that theme is not just a way to prevent people from cheating, but is in fact the major driving force behind the competition.  I no longer accept that DoZs should just be about making the best game possible in 24 hours.  They should be about making the best game possible based on the given themes.

Completely disagree. The theme category has always been there for anti-cheating, there has never been any disagreement regarding that. For one thing, it puts a lot more weight on the host's ability to come up with good themes. All too often have we seen people either complain that they were being held back by poor themes or dropped out altogether because of it. And of course, no one will ever agree on what makes a good theme. Having the theme category be worth so much implicitely overweights the "story" category because effective theme is much more passively influenced by such elements.

When you get down to it, the DoZ is an MZXing competition and should not be this impacted by how creatively one can adhere to a theme, something that may have nothing to do with their MZXing abilities in general (as opposed to more general writing, which does).

No, the theme category doesn't prevent cheating. But it greatly limits it. It's a psychological thing.

All in all, we'd probably have BETTER DoZ entries if we got rid of the theme category all together, but I think the temptation to cheat would be too great.

When DoZers receive a theme, the usual reaction isn't "Wow, I have a great idea for this theme! I can't wait to see how this turns out", it's "*groan* I have to make a game about THAT?" and the fact that so many games so poorly adhere to the theme reflects this. In fact, RARELY do we have DoZ games that really make exceptional usage of the theme, perhaps one or two a DoZ. I'd hate to see the next DoZ game win with a mediocre game that happened to be really creative with the theme.

When someone says the DoZ is a themed competition, I think this assertion is there for the sake of the person playing the DoZ games. To make a more interesting cohesive experience out of DoZ game playing. But the DoZ isn't for players, it's for authors. At least, that's how I feel about it.

Wervyn said:

Another issue is confusion over the exact purpose the "Innovation" category is supposed to serve.  Originally this had something to do with the old "Originality" category, but lately it has been used more often to award points to particularly well-pulled-off engines and other coding.  I feel that given the state of the community right now, we don't need engine originality so much as we need engine effectiveness; most of what can be done in MZX has been done.

A number of things either haven't been done or haven't been done effectively. Fighting games, sports, several kinds of puzzle games, to name a few. Strategy games haven't been done particularly well either (especially turn based strategy). The point of the category was to try to encourage people to try something more than yet another side scroller.

Wervyn said:

By the same token, I propose removing Discretion as a separate category and dispersing it in much the same way as a negative penalty in the appropriate categories.  I feel that the remaining categories adequately cover any aspects that might warrant point reduction.  Obscene or generally idiotic dialogue should come out of Plot (or Story, as I'd rename it; more on this in a moment), while remarkably buggy code should entail penalties in Technique or Gameplay, depending on the nature of the problem.

That's fine, although this doesn't make much of a difference overall.

Quote

As for the Rank Weight, there has been some discussion that while a good concept, it currently carries too much weight.  Thus, it will remain at 100 points, but percentage-wise this represents a 20% drop, due to the increased total.  With the rank weight, the total score from each judge will be a nice, round 500 points.

Why do you think it carries too much weight? It has almost never actually changed scores. I think it actually has represented significantly too little. At 20% you'd may as well just drop it.

The only reason the rank weight was affecting that particular DoZ as much as it did is because Lancer was outright abusing it. I don't recall how much he changed his scores after I called him on this though.

[Skylark]

Exophase, on Nov 8 2006, 01:25 AM, said:

Completely disagree. The theme category has always been there for anti-cheating, there has never been any disagreement regarding that. For one thing, it puts a lot more weight on the host's ability to come up with good themes. All too often have we seen people either complain that they were being held back by poor themes or dropped out altogether because of it. And of course, no one will ever agree on what makes a good theme. Having the theme category be worth so much implicitely overweights the "story" category because effective theme is much more passively influenced by such elements.

When you get down to it, the DoZ is an MZXing competition and should not be this  impacted by how creatively one can adhere to a theme, something that may have nothing to do with their MZXing abilities in general (as opposed to more general writing, which does).

Although I agree with you that the theme was originally about anti-cheating, I also agree that it certainly isn't now; Wervyn already pointed out that anybody can cheat in a DoZ. But we still use theme. And I agree with Wervyn that theme should be worth more than it is. Yes, the DoZ is just a way to get people to actually produce games, but the challenge is to make a game that fits the theme well - and too often many games fall far short of this. I think it has merit to increase the amount of points awarded for theme (although in hindsight I still think gameplay should have more prescence score-wise than theme) however I think good adherence to the theme should be a must - I also think that a solid theme present in a good game should be awarded points than a better game with very little thematic relevance. Otherwise, why have a theme at all?

Exophase said:

A number of things either haven't been done or haven't been done effectively. Fighting games, sports, several kinds of puzzle games, to name a few. Strategy games haven't been done particularly well either (especially turn based strategy). The point of the category was to try to encourage people to try something more than yet another side scroller.

The point I think is similar to discretion - innovation can occur in any/all sections, and I think that innovation itself can be incorporated for the most part into those sections. However, I think an actual section for technical ability would be good. An engine that is similar to its predecessors but much smoother, easier to handle, natural etc is just as good as a new idea, and there are few new ideas out there that couldn't be covered in another section. But if you decide to keep an innovation section, I think another section for technical ability should be added.

Exophase said:

Why do you think it carries too much weight? It has almost never actually changed scores. I think it actually has represented significantly too little. At 20% you'd may as well just drop it.

The only reason the rank weight was affecting that particular DoZ as much as it did is because Lancer was outright abusing it. I don't recall how much he changed his scores after I called him on this though.

I don't think that the judges personal preference (which will be biased on his favoured type of game, etc, whether they try to be objective or not) should be worth 25%. Maybe that's just my opinion. In any case, 20% is still a relatively large chunk of the marks. But more of it would be based on what is actually in the game, and less on a personal opinion.

EDIT:

Exophase said:

All in all, we'd probably have BETTER DoZ entries if we got rid of the theme category all together, but I think the temptation to cheat would be too great.

When DoZers receive a theme, the usual reaction isn't "Wow, I have a great idea for this theme! I can't wait to see how this turns out", it's "*groan* I have to make a game about THAT?" and the fact that so many games so poorly adhere to the theme reflects this. In fact, RARELY do we have DoZ games that really make exceptional usage of the theme, perhaps one or two a DoZ. I'd hate to see the next DoZ game win with a mediocre game that happened to be really creative with the theme.

When someone says the DoZ is a themed competition, I think this assertion is there for the sake of the person playing the DoZ games. To make a more interesting cohesive experience out of DoZ game playing. But the DoZ isn't for players, it's for authors. At least, that's how I feel about it.

Keep in mind that judges would find it nearly IMPOSSIBLE to judge games that did not have a relative theme. How do you compare apples and oranges? At least they have some idea of the varying ideas authors had when given the theme when multiple games use the same theme. Aside from judging, without theme the DoZ would not be the same. I don't think anyone would have the drive to do it if they weren't told what to make the game about; how would it be different from mzxing any other day? Well, that's my opinion, anyway.

I personally think (from a DoZer's point of view) that theme is what makes the DoZ fun. Trying to come up with a game idea that would suit the theme is the hardest part of the DoZ - frantically thinking of how you can possibly make your game idea suit the theme is FUN for me. But maybe like I said it's just me...

This post has been edited by Skylark: 07 November 2006 - 03:53 PM

[Koji]

I suggest 4 categories evenly weighted!

THEME
how well it follows the theme presented, and how much they made of the theme. did they build UP on the theme or build up TO the theme?

PRESENTATION
The graphics and sound involved with the game and how they are

CONCEPT
basicly the plot if there was one, and the overall idea behind the game. The quality and originality of the concept behind the game, how fleshed out it was, did it work for you etc.

GAMEPLAY
Was the game enjoyable? Did it feel new or just the same old same old?

There use that.

also there should be a props category for "extra cred" if they went above and beyond in some aspect to make up for a failing in another like if their gameplay is spectacular but their presentation sucked balls, I think there should be some way a judge could say "kudos!" to the doz team and it actually count for the scoring.

This post has been edited by Koji: 07 November 2006 - 04:06 PM

[Wervyn]

Exophase said:

Completely disagree. The theme category has always been there for anti-cheating, there has never been any disagreement regarding that. For one thing, it puts a lot more weight on the host's ability to come up with good themes. All too often have we seen people either complain that they were being held back by poor themes or dropped out altogether because of it. And of course, no one will ever agree on what makes a good theme. Having the theme category be worth so much implicitely overweights the "story" category because effective theme is much more passively influenced by such elements.

When you get down to it, the DoZ is an MZXing competition and should not be this impacted by how creatively one can adhere to a theme, something that may have nothing to do with their MZXing abilities in general (as opposed to more general writing, which does).

You're right, we completely disagree. Theme's effectiveness as an anti-cheating measure is basically nil. I think that the general creativity level of the competition could be improved by driving the importance of theme. You're absolutely right that this will change the nature of the competition--I think it should be changed. And damn straight this puts pressure on the host to come up with better themes, I also see that as a good thing. I expect some personal responsibility from the host in that department, rather than just throwing up our hands in despair at the idea that no good topics exist. I think especially in light of the Dualstream concept, theme should be important. Otherwise why bother with it at all?
Dualstream, too, alleviates the issues with theme being a primarily story-based category (I don't think it has to be at all, look at RUN! for example). Set up properly, the general topic would lend itself more to dramatic-based theme adherence, while the specific topic would allow for more conceptual, gameplay-based experimentation. Not that it always has to be that way, of course--the old favorite Gravity, for example, could be a very effective gameplay theme, while being a general topic. What the host should be aiming for with the topics then is to provide concepts that lend themselves to this sort of artistic vs. technical dichotomy.

Exophase said:

A number of things either haven't been done or haven't been done effectively. Fighting games, sports, several kinds of puzzle games, to name a few. Strategy games haven't been done particularly well either (especially turn based strategy). The point of the category was to try to encourage people to try something more than yet another side scroller.

The point of the revision is that many judges, including myself, do not like the category, and feel that it is not accomplishing that stated goal. As I described, points for originality would be better served as a bonus targeted in the specific categories where originality is displayed. A well done sports game would probably get an originality bonus from many judges in Gameplay, while a particularly innovative engine would add points to technique. This allows points to be awarded to unique stories or graphics as well. Basically, I think originality works better as an overall judging paradigm than it does a category. Discretion then becomes a sort of counterpart to it, in the new system.

Exophase said:

Why do you think [rank weight] carries too much weight? It has almost never actually changed scores. I think it actually has represented significantly too little. At 20% you'd may as well just drop it.

The purpose of rank weight to me is to help the judge differentiate for himself between games that come out close together by the other scoring methods, particularly when the relative compositions of the scores is different. It doesn't need to do more than that, and it doesn't need to be massively huge to do it. If it gets too large, I think the lack of fine-tuning in the gradiation becomes a real negative factor, and the final score begins to feel very arbitrary. I actually don't think 25% is too high (though I wouldn't support making it higher), but I don't think 20% significantly weakens its impact, since as you claim (and I agree) it really has no impact on a large scale. Since 20% makes all the numbers nice and even, I like it this way instead.

"Devils in the details."

[Exophase]

Skylark, on Nov 8 2006, 01:25 AM, said:

Although I agree with you that the theme was originally about anti-cheating, I also agree that it certainly isn't now; Wervyn already pointed out that anybody can cheat in a DoZ.

It isn't a black and white issue of whether someone somewhere cheats or no one does ever.

Skylark, on Nov 8 2006, 01:25 AM, said:

But we still use theme.

For novelty's sake. Personally I think we'd be just as well off getting rid of it.

Skylark, on Nov 8 2006, 01:25 AM, said:

And I agree with Wervyn that theme should be worth more than it is. Yes, the DoZ is just a way to get people to actually produce games, but the challenge is to make a game that fits the theme well

There's already plenty of challenge in making a good game, period. Why add to that something which is arguably separate?

Skylark, on Nov 8 2006, 01:25 AM, said:

- and too often many games fall far short of this. I think it has merit to increase the amount of points awarded for theme (although in hindsight I still think gameplay should have more prescence score-wise than theme) however I think good adherence to the theme should be a must - I also think that a solid theme present in a good game should be awarded points than a better game with very little thematic relevance. Otherwise, why have a theme at all?

It's not a strong argument to insist that theme be more important simply because it's there. Like I said, I wouldn't be completely opposed to getting rid of theme entirely, but it has novelty value (and tradition).

Skylark, on Nov 8 2006, 01:25 AM, said:

The point I think is similar to discretion - innovation can occur in any/all sections, and I think that innovation itself can be incorporated for the most part into those sections.

I don't disagree with you, I was just saying that I don't think innovation is dead. I do think that it is not necessarily important enough to merit having its own category.

Skylark, on Nov 8 2006, 01:25 AM, said:

However, I think an actual section for technical ability would be good.

But then it becomes more of a matter of judging the game on effort rather than more exact merit. Is this the route we want to go? Also, I don't think you mean technical ability, I think you mean what is actually exerted in the creation of the game.

I really don't see why a game should be judged at all separately for its implementation, which is what this sounds like.

Skylark, on Nov 8 2006, 01:25 AM, said:

An engine that is similar to its predecessors but much smoother, easier to handle, natural etc is just as good as a new idea, and there are few new ideas out there that couldn't be covered in another section.

All of these things ALSO naturally fit under the "gameplay" section. More naturally than "innovation" does, in fact.

Skylark, on Nov 8 2006, 01:25 AM, said:

But if you decide to keep an innovation section, I think another section for technical ability should be added.

Could just have neither...

Skylark, on Nov 8 2006, 01:25 AM, said:

I don't think that the judges personal preference (which will be biased on his favoured type of game, etc, whether they try to be objective or not) should be worth 25%. Maybe that's just my opinion. In any case, 20% is still a relatively large chunk of the marks. But more of it would be based on what is actually in the game, and less on a personal opinion.

The entire scoresheet is subject to the judge's personal preferences, and the entire thing is their opinion too, only along different guiding lines. The actual scoring partitions are also subject to a single person's preferences (the host's), which is why the rank weight was introduced, to allow for more voices that are independant of the scoring weight decisions.

Skylark, on Nov 8 2006, 01:25 AM, said:

Keep in mind that judges would find it nearly IMPOSSIBLE to judge games that did not have a relative theme. How do you compare apples and oranges?

So you're saying that two games of completely different genre are only comparable if they have the same theme? That doesn't make sense to me at all.

Skylark, on Nov 8 2006, 01:25 AM, said:

At least they have some idea of the varying ideas authors had when given the theme when multiple games use the same theme.

Why does that matter?

Skylark, on Nov 8 2006, 01:25 AM, said:

Aside from judging, without theme the DoZ would not be the same. I don't think anyone would have the drive to do it if they weren't told what to make the game about; how would it be different from mzxing any other day? Well, that's my opinion, anyway.

It'd be different, but you could be right. Of course, you're listing a rationale for theme that is completely independent of scoring.

So you could keep theme, drop the category, keep the disqualification vote, and you'd still end up with the same thing you're saying.

Skylark, on Nov 8 2006, 01:25 AM, said:

I personally think (from a DoZer's point of view) that theme is what makes the DoZ fun. Trying to come up with a game idea that would suit the theme is the hardest part of the DoZ - frantically thinking of how you can possibly make your game idea suit the theme is FUN for me. But maybe like I said it's just me...

I think most DoZers agree that coming up with a good idea to fit the theme is not the hardest part, especially not in terms of actual work hours required dedicating to.

[Exophase]

Wervyn said:

You're right, we completely disagree.  Theme's effectiveness as an anti-cheating measure is basically nil.

Again, its effectiveness isn't measured on whether or not it prevents all people, but the overall degree to which it deters others. You can't really prove it one way or the other, of course.

Wervyn said:

I think that the general creativity level of the competition could be improved by driving the importance of theme.

I doubt it. I don't think people are going to suddenly buckle down and crank out games that adhere better to the theme just because it's worth more. I think that what we will get is more people dropping out and just lower scores in general. That's possibly what you want, of course.

Wervyn said:

You're absolutely right that this will change the nature of the competition--I think it should be changed.

Because you want to emphasize the theme more than MZX games, and IMO the first is independent of the second. This is a personal bias on your part.

Wervyn said:

And damn straight this puts pressure on the host to come up with better themes, I also see that as a good thing.  I expect some personal responsibility from the host in that department, rather than just throwing up our hands in despair at the idea that no good topics exist.

And yet basically no one is going to agree that a theme is universally good. I know that a lot of people thought your themes were simply horrible, same with mine, and I at least liked them (mine, not yours, I thought yours were bad ). That's just what is going to happen.

Wervyn said:

I think especially in light of the Dualstream concept, theme should be important.  Otherwise why bother with it at all?

Well, "why bother at all" seems to be the argument of the day, but it's certainly not a convincing one for making it worth more. Skylark gave a a good reason for this, to give people some direction, a starting point for creating their games. There are other reasons too, such as making the DoZs themselves more unique amongst each other. Theme is valid without making it an integral part of what you are judged on.

Wervyn said:

Dualstream, too, alleviates the issues with theme being a primarily story-based category (I don't think it has to be at  all, look at RUN! for example).  Set up properly, the general topic would lend itself more to dramatic-based theme adherence, while the specific topic would allow for more conceptual, gameplay-based experimentation.  Not that it always has to be that way, of course--the old favorite Gravity, for example, could be a very effective gameplay theme, while being a general topic.  What the host should be aiming for with the topics then is to provide concepts that lend themselves to this sort of artistic vs. technical dichotomy.

Sure, but again, requiring theme adherence doesn't really mean that so many points have to be awarded.

The fact is, almost all DoZ judges have a very hard time judging theme as it is. I'd say that next to innovation it has been the most troubling one, and they tend to give very high scores if it matches at all. If it's going to turn into that again then it'll obviously be wasted.

But maybe this is something that you'll have to see succeed or fail in practice. Personally I'm not interested in making such bold experiments with the DoZ.

Exophase said:

The point of the revision is that many judges, including myself, do not like the category, and feel that it is not accomplishing that stated goal.

Agreed, but I could basically make the same argument about the theme category. That is, that many judges don't like it and that it is not accomplishing driving more innovation in theme. Of course, your answer is to make theme worth more, but obviously not make innovation worth more...

Exophase said:

As I described, points for originality would be better served as a bonus targeted in the specific categories where originality is displayed. A well done sports game would probably get an originality bonus from many judges in Gameplay, while a particularly innovative engine would add points to technique.  This allows points to be awarded to unique stories or graphics as well.  Basically, I think originality works better as an overall judging paradigm than it does a category.  Discretion then becomes a sort of counterpart to it, in the new system.

Problem with "bonuses" is that they're not such if there's no room for "extra credit." Judges are not going to artifically partition the categories themselves and not abstain from giving high gameplay scores to unoriginal games. Nor should they.

But I suppose innovation isn't really something MZXers are going to push for either way, and no one likes giving everyone 0's, so may as well get rid of it.

Exophase said:

The purpose of rank weight to me is to help the judge differentiate for himself between games that come out close together by the other scoring methods, particularly when the relative compositions of the scores is different.

We created rank weight to be a "second opinion" to outweigh the opinions in score distribution (which various people didn't like). Clearly some people think that the rank weight should make up the entire scoring (Madbrain has said so, Nash said so here), so it's a comprimise.

At 20% it doesn't really contribute anything though.

Exophase said:

It doesn't need to do more than that, and it doesn't need to be massively huge to do it.  If it gets too large, I think the lack of fine-tuning in the gradiation becomes a real negative factor, and the final score begins to feel very arbitrary.  I actually don't think 25% is too high (though I wouldn't support making it higher), but I don't think 20% significantly weakens its impact, since as you claim (and I agree) it really has no impact on a large scale.  Since 20% makes all the numbers nice and even, I like it this way instead.

With no impact just get rid of it. At 20% it isn't even worth everyone's time. If it were up to me I'd change it to 50%. That is, of course, under the assumption that good judges are picked who don't have extremely bizarre tastes and aren't going to use the weight rank to try to overcorrect the main scores.

Let me elaborate on that last part, since I don't think what I'm saying is that clear, and I don't think some people understood why it was unfair.

The rank weight should be done independenlty of the scores. So it shouldn't follow the scores blindly. But you also shouldn't rank a game HIGHER than you would have otherwise because you don't think theme deserves as many points as Wervyn gave it and are trying to boost Logicow's game instead.

Does that make sense? I yelled at Lancer once for this and he changed his ranking because of it.

[MadBrain]

Honestly, I have to say that personally, I don't like the Theme category. I think it tends to favor boring good-plot-by-no-gameplay or at best cinematics-plus-minigames games. Stuff with a theme like... let's say, DISGRACE. This makes a game where you have to endure endless fading text about some Emo anime guy and how the world of Alemath is slowly being taken over by demonic spirits or whatever. Honestly, I'm sorry but I just don't have the patience to play through those games. Do SNES games do that to you? No. They cut the crap and actually have gameplay.

Of course, you can still make an action game, I guess. The thing is, to make a good action game you must cut the time spent making up plot and writing boring fading text nobody reads. Every half hour you spend on plot is a half hour not spent on gameplay. Puzzle games ? la tetris suffer even more from this emphasis on Plot and Theme, because they are by definition abstract. Another example would be fighting games, which often simply cannot have this sort of unified RPG-like plot, because they has many characters with very divergent interests, which means there is almost 1 plot per character. Your proposal basically advantages two genres, RPG and Adventure, and disadvantages all the others.

In other words, I wanna make games about blowing stuff up, and not crying emo anime guys with dead parents. Your proposal of increasing the weight of Theme to some ridiculous amount runs directly counter to that.

[asgromo]

Exophase, on Nov 7 2006, 02:32 PM, said:

Skylark, on Nov 8 2006, 01:25 AM, said:

Keep in mind that judges would find it nearly IMPOSSIBLE to judge games that did not have a relative theme. How do you compare apples and oranges?

So you're saying that two games of completely different genre are only comparable if they have the same theme? That doesn't make sense to me at all.

He's also saying that two games that are of a completely different theme, which is the entire point of a dualstream Day of Zeux, are impossible to rank against each other, which is what happens every dualstream Day of Zeux. I guess he has a point?

edit: coherence

[Pyro1588]

well, i was going to say i liked the new system, but then people just had to go and start posting, and now there's another 3 chapters of thread to read through so i will have to get back to this when i'm not running late for work...

[Nash]

Wervyn, on Nov 7 2006, 09:54 AM, said:

Well you're right about one thing Nash, your games' scores wouldn't change that much. However, just taking one DoZ as a sample (2004 Christmas):

                                        Galladin     Lancer       Weasel       Astral       Total        Original
#6168 - Judgement of the Immortal       2            1            2            2            7  (1)       2 (1240)
#9532 - Run! Run! Run!                  3            5            1            1            10 (2)       1 (1330)
#4865 - Scottish adventure              1            4            7            3            15 (3)       3 (1179)
#5349 - Unearthin' Egypt                5            3            3            5            16 (4)       5  (893)
#84865 - Fear nothing                   7            2            4            7            20 (5)       4  (984)
#323 - Voyage of Fear                   4            6            6            6            22 (6)       9  (694)
#35974 - Are you afraid?                6            7            12           8            33 (7)       6  (833)
#812 - Jeff Kenny: secret agent         13           8            5            10           36 (8)       10 (650)
#7486 - Zeeb's interplanar adventure    10           9            11           9            39 (9)       7  (760)
#2004 - Afraid to forget                8            14           14           4            40 (10)      14 (512)
#17915 - They hide in the dark          11           13           8            11           43 (11)      8  (709)
#962 - Zombies                          12           11           9            13           45 (12)      11 (606)
#1799 - Dungeons Down Under             9            10           13           15           47 (13)      13 (543)
#723 - Temple o Ketera                  14           12           10           12           48 (14)      12 (554)
#548 - The infernal fear                15           15           16           14           60 (15)      15 (172)
#22947 - A game about fear              16           16           15           16           63 (16)      16 (126)
#96234 - Tha Mindkilla                  17           17           17           17           68 (17)      17  (95)

As you can see, scoring by rank weight alone would have significantly altered the outcome of the DoZ, especially in the mid-range. Your system also makes it difficult to decide on appropriate penalties for late games (which is one factor that doesn't really come across in those score differences). Finally, no one said judges have to provide detailed comments for each category, that's just something many of us have done on our own. Look at the times I didn't leave comments because I didn't feel like it. Dividing the score into discrete units still helps competitors understand and connect with where the scores came from; switching to pure rank weight would make the final decision feel extremely arbitrary. In the end, too, if we didn't provide judges with an explicit rubric, I think they would create their own out of necessity. Trying to rank as many as 20 games just based on "feel" is extremely difficult.
Also, I have to entirely disagree regarding the purpose of theme, now. Somehow, what you've described is what happened, and theme just became a way to prevent people from cheating. But if you really think about it, we aren't really concerned with people cheating much at all. And besides, it doesn't help that much: if someone really wanted to make a game before the competition, it would be fairly simple to rework the game elements and the plot enough to incorporate the theme, given 24 hours to do so. Much less work than creating the game itself. I think that in relegating theme to a pittance of the score, we've really lost touch with the idea of DoZs as THEMED competitions. I'd really like to get that feeling back.

"Sou...datoomou."

wow wervyn who gives a shit. how about you post the scores from the doz that me and luke got second place in rather than the one we spent like 15 minutes on and just made some random joke game.

[Wervyn]

Point number one: Theme is not the same as plot. Anyone who thinks it is should not be judging the DoZ under this set. As Theme stands, a game that can't successfully incorporate the theme into all elements of the game should not be getting full marks. The fact that a lot of people don't seem to be able to make this distinction in their minds is actually a concern when it comes to finding judges, and this system may not work after all if that's the case.

But here's one thing to consider: themes that only lend themselves to expression through the plot are BAD THEMES. Or at the very least, at least one of the themes should lend itself to being better expressed through a gameplay concept. And hey, if you look at the history of the Dualstream competition, pretty much ALL of the specific topics are like this. If people are having real trouble coming up with games to fit the themes (and I haven't generally seen that they are), then they need to be thinking a little harder, because even the BAD themes we've had (Swords, blech) are entirely amenable to creating games with little or no plot.

General topics I'll grant tend to require more focus on plot to really bring out a full exploration of the theme (though I'll say again that I really liked what we did with Fear in Run!...even if it may have been more suited to a topic like "Panic"). With the existence of a specific theme to balance this, I think this is not only acceptable but much better than not having it at all, because we DO want to cater to people who want to make games with plots, just as much as we want to provide options for those people who don't.

Other responses, particularly to Exophase: Of course I'm biased, that was never really in question. This is clearly a major ideological split between us. I am interested in having a competition that places importance on theme, primarily because I think it would make the competition more interesting and fun, generate more creative results, and especially because change itself will attract interest and possibly improve participation. You would prefer not to change a system that for the most part works quite well. Of course this is an experiment, I thought that was obvious. And if it fails horribly we can always roll it back. But I'm optimistic about it.

Also, you may be right about the rank weight, I was trying to leave that in for legacy, but it may be better to just remove it altogether, since I certainly don't approve of making 50% of the score completely arbitrary. This goes out to Nash, too, and thanks by the way for completely ignoring the main content of my post.

"As a rule we perceive what we expect to perceive. The unexpected is usually not received at all."

[Dr Lancer-X]

All of my suggestions for scoring have already been incorporated into the new scoreset, so I really don't have anything to add. However Theme will never ever work as an anti-cheating measure. There is no possible way to stop people from cheating at a DoZ. Theme is about focusing people on a particular concept when making their game. You can go out and make a kickass game in 24 hours any time you like.

The only problem with Theme is that hosts will actually have to think up good themes now.

Rank weight I would not mind seeing removed, but my request to have it toned down has gone through so it does not bother me. And, if anything, I've only benefited from rank weight (to my memory nearly ever DoZ I've entered my team's score/place has increased relatively through rank weight).

[Risu2112]

Break the DoZ up.

I suggest 1 hybrid DoZ like we have now produces less work than 1 segregated DoZ.
I suggest a DoZ with one specific theme topic And one Genre topic.
I suggest each topic have a separate score table that reflects the topic.
I suggest at the end games of both topics be ranked as one group.

Pick your theme, OR pick your genre. This would greatly increase appeal to more robotic oriented DoZ'ers without compromising more story/plot related authors.

Quick example: DOZ TOPICS: 1) Mermaids. 2) Real time strategy game (rts)

FAQ
Q) I don't want to make a game about mermaids! A) Make an RTS!
Q) I don't want to make a real time strategy game! A) Make a game about mermaids!
Q) I don't want to make an RTS or a mermaid game! A) Submit Joke entry to be DQ'ed

Theme oriented people can chose to take the given theme, if it does not appeal to them they could make their own theme as long as it conformed to the genre. I further suspect this would reduce the number of all flash no substance entries that people keep complaining about, WITHOUT hindering the authors. The author selects his/her topic. The topic determines the scoring table, generaly speaking, the genre topic would weigh heavy on gameplay/robotic, and the theme topic would weigh heavy on plot/story. Every body gets what they want, DoZ attendance increases, quality of entries increases.

Discuss!

Edit: specific theme/general theme which ever one is more desirable, I now see no reason to discriminate one way or the other.

This post has been edited by KenOhki2112: 08 November 2006 - 12:13 AM

[Terryn]

Sectioning off the DoZ seems like a bad idea to me. That's probably the biggest factor in how the 24HoZZT died considerably before ZZTers stopped using ZZT.

[Risu2112]

Terryn, on Nov 7 2006, 05:23 PM, said:

Sectioning off the DoZ seems like a bad idea to me. That's probably the biggest factor in how the 24HoZZT died considerably before ZZTers stopped using ZZT.

Keep in mind every one is still in direct competition with each other. With multiple rulesets though we can virtualy eliminate bias in the ruleset favoring one type of game over another, say a themey game or a pro robotic game.

[RoSS]

Ok, first. I know this is going to be an unpopular statement but please calm down. I know the DoZ's are important to everybody, but the main purpose of them is to have fun. Participants can (theoretically) have fun with no scoring at all, but the scoring does add fun, I agree.

Something I decided a while back when I offered to host after Insidious's DoZ was that if I hosted one, I would definitely meet with my judges, far in advance, brainstorm together (but have several ideas beforehand) and make sure the topics were solid. I agree: two good topics can really produce a lot of good games and fun times making them. So I'm going to make sure we have some good ones this time.

I have not yet made a final decision on the score sheet, but I'm leaning towards trying Wervyn's out and seeing what happens.