Where are all the RPGs in the IGF?
Another year, another Independent Games Festival; another Independent Games Festival, another group of finalists; another group of finalists, another army of aggrieved developers who didn’t make the cut. There has arisen something of a tradition among indie developers of complaining about the Independent Games Festival. There are reasons for that. For one thing, the stakes are high. With thousands of dollars and widespread publicity on the line, a strong showing in the IGF can make all the difference for an indie developer. More than that, it’s very expensive to enter, so people want to feel like their entries have been given a fair shake. Oftentimes they don’t feel that way, which leads to grousing and drama.
I have never felt the urge to submit anything to the IGF myself, and as such, I’ve ever had any personal stake in the fairness of IGF proceedings. However, it certainly hasn’t escaped my notice that, year after year, the IGF conspicuously passes over RPG entrants–and now that I run this site, I feel that my role as an advocate for indie RPGs requires me to explore the issue.
The problem is simple: indie RPGs do not win in the IGF. In fact, with few exceptions, they don’t even get selected as finalists. I use the present tense here, but this issue stretches back over the IGF’s entire 13-year history.
The only game among the finalists this year that could even arguably count as an RPG is the frantic, skin-deep action-shooter-that-has-loot-and-leveling Realm of the Mad God. Last year, Bastion and the short-form, dungeon-delving, mostly-just-a-puzzle-game Desktop Dungeons made it into the finals. The year before that? Nothing. (I’m not going to pretend that Heroes of Newerth is an RPG; it isn’t.) The year before that? Nothing. The year before that? Nothing.
The year before that? Well, here is where we reach something unique in the IGF’s history. For only two years, 2006 and 2007, the IGF had a separate Mod Competition. In 2007, an RPG mod won in the category…Best RPG Mod. Something similar happened in 2006, where the mods were grouped by which game they ran on. A category called “Best Mod – Neverwinter Nights” (which might as well have been “Best RPG Mod”) contained RPG finalists. A single action RPG snuck into the “Best Mod – Unreal Tournament 2004” category, but that was it. No RPGs won anything in the main competition in either year. (The indie MMORPG Dofus received a Visual Art nomination in 2006, but it did not win.)
No RPGs were finalists in 2005. If we go all the way back to 2004, we find a hybrid dungeon delver/spelling game that was chosen as a finalist, also in Visual Art. However, it’s just a linear sequence of timed word puzzles with RPG scènes à faire. It is not an RPG. The first four years of the IGF, 1999 through 2003, aren’t much better.
To sum up: strategy games, shooters, puzzlers and DoTA-alikes with fantasy settings have each been recognized in the IGF. Actual RPGs, with very few exceptions, have not. But why?
Breaking Down the Numbers
One popular explanation for this has been that RPGs are relatively rare among indie games due to the extreme demands of creating them. Fewer RPGs created means fewer RPG entries, which in turn means a lower statistical chance that RPGs will be chosen for anything in the competition. There is also a related defeatism argument: I have heard repeatedly from talented RPG developers that they have no desire to enter their games into the IGF because RPGs are never selected to win anything. And yet RPGs cannot win in the IGF if no one is entering RPGs into the competition, right?
These explanations sound plausible enough, but a look at the numbers shows that they do not adequately explain why RPGs have not received more recognition than they have.
This year’s IGF has 567 entrants (I’m ignoring student entrants). Based on a quick control-F search for “RPG” on the entrant pages, I’ve determined that the following self-described RPGs* were submitted this year: A Closed World, Alcarys Complex, Crystalides, Dangerous, Dark Scavenger, Dragon Fantasy, Defender’s Quest: Valley of the Forgotten, Dungeon Defenders, Dust: An Elysian Tail, Faith of the Guardians, Fortune Summoners: Secret of the Elemental Stone, Frayed Knights: The Skull of S’makh-Daon, Kali9, Lair of the Evildoer, Legendary Wars, Loot Pursuit, NEStalgia, RaonDefenders, Serious Sam: The Random Encounter, Slide RPG, Story Universe, The Savage Garden, Wilfred the Hero, and Zack’s Adventure: The Lost IRIS.
Last year’s IGF had 391 entrants. Self-described RPG entrants were: Aphelion, Crusade of Destiny, Din’s Curse, Dungeon Defenders, Fantasy University, Le Petit Chat, Leelh, Legend of Fae, Mark Leung: Revenge of the Bitch, Papercraft, Rainblood: Town of Death, Shadow Rising, and Tumblestone.
* NOTE: I chose not to second-guess the “RPG” designations on entrants due to the demands of manually combing through hundreds of entries looking for data. However, if a game was merely described as having “RPG elements,” I did not include it. If you account for the entries that are really tower defense games, puzzle games or shmups with RPG-style leveling yet mislabel themselves as true RPGs, this list should be shorter; if you account for RPGs that fail to describe themselves as such–Bastion is a good example of this–it should be a little longer. I think these lists are pretty close to right, numerically speaking.
I surveyed the lists of IGF entrants in this way going back to 2008, the earliest year for which the IGF provides the total number of entrants. This provides us with comparison data for the five most recent years of the competition (2012, 2011, 2010, 2009 and 2008). From my brief survey, I can draw some conclusions about these last five years:
1.) The 2012 IGF Main Competition has 38 spots for finalists; so did 2011. 2010 had 30 spots, and 2009 had 34. 2008, 2007 and 2006 each had 28. That’s 224 potential finalist spots in the main competition.
The 2011 IGF Main Competition has 9 spots for category winners; 2010 has 8, as do 2009 and 2008. That’s 33 winner spots. (There are no winners selected for 2012 yet, so we ignore it for purposes of determining the number of winner spots.)
2.) Realm of the Mad God does not describe itself as an RPG, and I tend to agree, so we ignore it in the count. Neither does Desktop Dungeons; it calls itself “a single-screen puzzle adventure,” a designation I also agree with. Bastion doesn’t describe itself as an RPG, but I think that one pretty clearly counts. That brings us to a grand total of one (1) finalist slot occupied by an RPG in the last 5 years (accounting for 0.045% of all finalist spots), and zero (0) winner slots. If we were to count Realm of the Mad God and Desktop Dungeons as RPGs, however, that would mean that the last five years’ worth of RPG entrants ended up filling four finalist spots out of 224, or 1.8% of the total, and one winner spot out of 33, or 3% of the total.
3.) For the 2012 IGF, there were 24 self-described RPGs entered out of 567 submissions. This means that RPGs accounted for 4.2% of all games submitted.
4.) In 2011, there were 13 self-described RPGs entered out of 391 submissions. RPGs accounted for 3.3% of all games submitted.
5.) In 2010, 8 self-described RPGs were submitted out of 301 submissions. RPGs accounted for 2.7% of all games submitted.
6.) In 2009, 7 self-described RPGS were submitted out of 224 submissions. RPGs accounted for 3.1% of all games submitted.
7.) In 2008, 6 self-described RPGS were submitted out of 173 submissions. RPGs accounted for 3.5% of all games submitted.
8.) Adding up all the RPGs submitted (58) and dividing by the total number of all games submitted (1656) over the past five years, we end up with an average of 3.5% of submissions that are RPGs.
Looking at these numbers, it is apparent that the “RPGs aren’t chosen because so few are submitted” argument doesn’t hold up. For the past 5 years, RPGs have consistently accounted for roughly 3-4% of games submitted, and on average account for 3.5% of the total. Even considering the relatively low proportion of entries to the IGF that are RPGs, RPGs are still disproportionately passed over for finalist nominations. Even if we count Desktop Dungeons and Realm of the Mad God as RPGs (which I consider a real stretch), they give us an RPG nomination rate of only 1.8% (roughly half the average proportion of RPGs submitted over this five-year period) and a win rate of 3% (still below the average). Clearly, something else is going on here beyond mere low submission numbers.
How the IGF is Judged
Before we get to other possible explanations for this phenomenon, let’s review some basics about the IGF judging process. In previous IGFs (by which I mean 2010 and earlier), this is how judging worked:
When a game is entered into the IGF, it is given to a a number of different people to judge. Before 2010, it was typically 4 judges per game. As of 2010, each game started receiving roughly 8 judges. IGF judges consist of “representatives from the mainstream game industry, notable previous IGF winners and finalists, other independent game developers, and indie-friendly game journalists.” Who gets to judge what is limited by who has the hardware to run which games, but otherwise the distribution is randomized. Judges have one month to play and rate roughly 14 games, with an average of two days to devote to each (these figures were estimated in 2010; it may have been different in years prior). For each game he/she judges, the judge assigns a score of 1-100 in five separate categories: design, visual art, audio, technical excellence, and overall impression. In 2010, the IGF started to require that judges leave significant written feedback along with their scores, presumably in response to feedback like Anna Anthropy’s.
Starting with the 2011 IGF, a jury process was introduced and the role of judging changed dramatically:
Rather than grading games on a numerical scale to quantitatively determine the “best games”, the wider body of judges will be asked instead to nominate the games allotted to them for any of the IGF’s categories like design, visual art and audio.
The top-nominated games will be then passed to a smaller jury who will together determine the finalists and winner after rounds of debate and conversation. Our hope is that this will make the process more engaging for the jurists and more fair for developers, who will now be directly appraised by a jury of industry peers: visual artists for the visual art category, engineers for technical excellence and so on.
In an email to me, IGF Chairman Brandon Boyer put it this way: “there are no more numerical scores per game in the judging round. Each judge is free to nominate each game for any category, or choose not to nominate it at all. The jury receives this tallied list of nominations to use as a guideline, though they are entirely free to suggest their own picks or ignore highly nominated games before doing their own finalist voting. We generally ask that any game that received over 2-3 nominations from the body of judges be investigated further.”
One of the judges, Ben Ruiz, has publicly stated that there are now really just two determinative factors in which games get nominated: “IGF submissions are nominated because they are a compelling combination of fun and remarkable to the majority of the player constituents.” He expressed confidence that there is sufficient variety in the judge pool that biases about game design philosophy should be eliminated.
For this year’s Festival, Boyer tells me that the number of judges has been increased to 250, with each judge getting approximately 20 games to judge. This seems designed to keep the judge/game ratio at or slightly above the level it was in 2010: with 250 total judges each judging 20 games out of the 567 submitted, an average of 8.8 judges ends up weighing in on each entrant. The judging period is currently 4-6 weeks (28-42 days), which means that judges currently have approximately 1.4 to 2.1 days on average to devote to each game. The period for jury deliberation is also 4-6 weeks, though it’s less clear how that breaks down in terms of time available to try each game.
Possible Explanations for Lack of RPG Representation
With all that in mind, a number of potential explanations jump out at me as to why RPGs keep getting passed over.
Judges face severe time limitations
One of the issues may be a function of the way judging occurs. In 2010, judges had an average of 2 days per game over the course of a month to play and judge each. Now, with each judge responsible for 20 games, 2 days per game becomes a best-case scenario. Further, the reality is that IGF judges are volunteers. Outside of the IGF, they have full-time jobs and schedules that limit the amount of time they can realistically spend playing games each day. They are also human beings, which means that they tend to procrastinate. According to Alex May, “many judges, like me, left it quite late before starting” in 2010. In short: two days per game is a fantasy. It seems far more likely that most games get no more than a few hours of consideration from any given judge.
This is really bad for RPGs. You can get the measure of a shooter or a platformer in a few hours; not so a serious RPG. RPGs tend to be slow-burners. It takes a significant time investment to really appreciate what an RPG has to offer. Worse, because RPGs are built upon the making of irreversible choices, many of them cannot even be fully appreciated without multiple play-throughs! Unless a judge is unusually committed and organized, allotting ample time in advance for play-throughs of his/her RPG submissions, the judge is unlikely to have enough time to get a good sense for the RPGs he or she is assigned to judge.
Entrants lack audiovisual polish
For some reason, the majority of the RPGs I’ve noticed among recent entrants into the IGF have a noticeable lack of polish compared to entrants from other genres. This is purely a subjective judgment, of course, but if you browse through the entries, it’s hard not to see how much nicer many of the games from other genres look. It’s easy to see why this might be the case: RPGs simply demand a lot more content. This leaves less time in the development cycle for polishing, while simultaneously making polishing far more time-consuming. It’s analogous to the difference between painting a dozen portraits and painting two hundred portraits within the same time frame: the more you have the make, the more quickly you have to make them, and the less polished they are all going to turn out.
There’s also a question of expense. From what I’ve seen, RPG developers tend to engage in solo development and usually do not come from visual art backgrounds. This means that they have to pay contractors to produce their art. In a game with a small amount of assets, it’s relatively quick and cheap to make them look really good. But doing that in an RPG means making hundreds of items and dozens of monster types and dozens of spells all look really excellent. That gets very expensive, very fast.
Still, this explanation doesn’t sit entirely right with me. If you look at the entrants from the last few years, you’ll notice games like Bastion, Dust: An Elysian Tail, Eschalon Book II and Fortune Summoners. All of these are pretty darn polished. If nothing else, it’s hard to imagine why Bastion would not end up a finalist for Audio, or Dust not end up a finalist for Visual Art.
Subset of judge / jury has an anti-narrative bias
When I look at RPG entries to the IGF, I see many of them trying to stand out by referencing their detailed narratives and setting instead of highlighting what (if anything) is unique about their game mechanics. And if my experience in the indie scene tells me anything, it’s that this is a little like trying to become the President of the United States by getting fired out of a circus cannon into the reflecting pool on the national mall.
Here’s the thing. Narrative focus is actually a pretty divisive thing in the indie community. Some indie developers, to their credit, appreciate good writing. Other indies think text in games is indicative of bad design, and some go so far as to say that it’s actually manipulative. If you get one of these second types of people as a judge, they are not going to receive your story-heavy game favorably.
Now that games proceed to jury based on judge nominations rather than scores, this doesn’t have quite the disastrous impact it once did–you no longer have to worry about an anti-narrative judge dragging down your game’s average score because of his preconceptions about text in games. (It could still have an impact, however, particularly during the jury process.)
Perhaps a bigger problem is that the IGF judging system simply doesn’t encourage judges to consider a game’s narrative merits. A judge is only able to nominate a game in the IGF’s award categories: there’s one for good visual art, one for audio, one for having an impressive engine, one for good design, and a second design award for experimental work. What’s that? You say your game is well-written? Well! In that case, you get to take home the Boo-Hoo Award, consisting entirely of your own, bitter tears.
RPGs employ traditional game mechanics
Mechanics-wise, the RPG genre is very bound-up in tradition, and that’s reflected in many of the entries. This dovetails with the point about narrative focus above. People who get into developing RPGs are oftentimes more into the story and the setting than into doing something innovative with the game’s mechanics. Necessarily, if a developer pours most of her energy into the narrative and setting, that is not going to leave her much time to focus on doing something truly interesting with the guts of the game.
To be fair, not all indie RPG developers are interested in story. Many are primarily motivated by a desire to return to mechanics that were once standard in the genre, but have since all but disappeared. The one thing you don’t generally see, though, are indie RPG developers who are interested in pioneering radical new twists in RPG mechanics.
One way or another, the result is the same: these games end up having very familiar systems. Most of the RPGs I see among the entrants to the IGF are either using some variation on the old Final Fantasy battle system, doing a variation on Puzzle Quest, doing a variation on Protector, or going for a side-scrolling beat-em-up approach. I don’t mean to suggest that that is as a bad thing. However, it does have major consequences for a game’s chances at winning the IGF’s grand prize and design-focused awards.
IGF judges tend to favor games with innovative or unusual core mechanics. Given the time limitations involved in judging IGF submissions, games which innovate in striking, immediately noticeable ways are going to have a big advantage. In practice, the games that have been most successful in recent IGFs have been those which introduced one or two big, obvious changes to the core mechanics of their genre: things like realistic physics, dimensional manipulation, real-time creation / destruction of in-game environments, or procedural generation. (Not that their innovative use of procedural generation helped Din’s Curse or Depths of Peril, mind you, but that’s just two games–not a big enough sample size to conclude that this sort of approach wouldn’t make a difference for future RPG entrants.)
RPGs are jacks of all trades
This is a big one, so bear with me. By and large, the creators of the most successful IGF entrants picked one or two things that they wanted to do and poured all their efforts into doing those things really, really well. This is generally thought of as a good design approach, but in practice it runs counter to what we generally consider desirable in the RPG sphere. Among RPG developers, open-endedness and player freedom are highly valued. This has its roots in the traditions of the pen-and-paper RPGs that spawned the genre: we want the player to be able to go anywhere, make consequential choices, and play the game in a wide variety of different ways (a.k.a. “role playing”).
This means that your typical RPG features at least half a dozen distinct gameplay systems to suit a variety of different gameplay styles: item collection and management; crafting; dialog; stealth; combat; magic (typically with a number of different sub-specialties); character building (both in terms of creation and point allocation/skill tree navigation); and oftentimes, gambling (simulated with stat rolls, or via a full-fledged card or tile-based mini-games). That is a lot of stuff to have to do and do well; for a solo developer with limited time and budget, depth and polish can easily end up getting sacrificed on the altar of breadth.
This same issue crops up with RPG narrative. A few narrative-heavy games are now starting to make it through the IGF with some success, notables being To the Moon, Trauma and Dear Esther. Oddly, these games have almost no mechanics to speak of! They feature no combat; no puzzles worth mentioning; no sense of danger. They offer no challenge, essentially–they are single-mindedly focused on story and exploration, so much so that some believe they simply are not games at all. These games follow the rule we set forth above: they pick one thing and do it really, really well. In this case, it just happens that the thing they choose to do well is narrative.
The open-ended nature of RPGs makes this difficult to do. RPG players will tell you that there is something uniquely satisfying about having a direct role in the outcome of the game’s story, with elements shifting based on the things you’ve said and done. While that is true, non-linear narratives of this sort are extremely difficult to pull off well. A simple interaction with three or four choices takes longer to create than a linear interaction three or four times its length. A legitimate storyline branch can tremendously increase the amount of work involved in finishing the story. What ends up happening is the RPG developer inevitably spends a huge amount of time providing choices and consequences–player freedom buttressed by reactivity from the game world–while developers creating linear narratives get to focus all their energy on honing theme, establishing tone, and deepening character relationships.
If you play To the Moon or Dear Esther, you will see that these games feature very affecting and personal stories. The conflict in these stories is mostly emotional (as opposed to physical): they are not about war or killing, but about relationships and feelings. The characters are carefully crafted, with loads of work put into mood, tone and pacing. They are moving and reflective experiences–but at the same time, they are experiences largely divorced from player input. Imagine what would happen if these narratives weren’t linear. Imagine if you could abandon the house in To the Moon, or irreparably damage the friendship between the doctors, or do any number of other things that would involve big consequences for the storyline. How on earth are you going to manage a consistent tone? How are you going to enforce a satisfying story arc? How are you going to ensure that the theme survives intact?
It’s possible, but it’s incredibly difficult. The characters have to remember all of the things the player has done; they have to react convincingly; and the story itself has to account for them. Is each variation going to be just as moving? Will the characters all be as fully developed in each variation? If so, we’re looking at exponentially more work to create that same sense of progressing through a deep, moving storyline.
This is why you don’t see serious RPGs about character relationships. It’s much, much easier to have the world react to variables like faction allegiances than it is to try to capture the subtleties of how characters feel about you on a personal level. “Hostile” versus “friendly” is a nice, easy boolean value; “will talk to the player” versus “will never speak to him again” is doable. This sort of approach lends itself to systems that can handle it emergently–which is to say, combat. Not so the subtleties of human emotion. In order to write a deep, moving storyline that revolves principally around the complex relationships of various individual characters with player freedom and dialog variations, realistically, you need a whole team of writers. Indie RPG developers don’t have teams of writers. So they write war epics instead.
Judges have genre bias
I really don’t think that this is the most likely explanation, but I’d be remiss not to at least give it cursory consideration. At the outset, I have a very hard time imagining that any statistically significant portion of IGF judges actually dislikes RPGs. However, it is certainly a possibility that most of them prefer other genres. The truth is, we can’t really know without polling the judge and jury members, or at the very least seeing a breakdown of the judging pool by genre of expertise. The only thing really worth noting here is that the judging pool is made up in part of past winners–which, as we established above, contains non-RPG developers pretty much exclusively.
So, Who Is to Blame?
Realistically, I’m not sure that there’s really any one party we can blame for all of this. The IGF could certainly do more to accommodate long-form games like RPGs in terms of the judge/jury process. It might be smart for them to have entrants submit an “approximate time to complete the game” so judges can budget their time better; giving the judges and juries more time with the games could be helpful as well. Also, it would probably be a good idea to make sure that the judge pool adequately represents the RPG developer community, as a precautionary measure if nothing else.
Indie RPG developers bear some responsibility here as well, however. No one is making us hew to hide-bound traditions in our game mechanics. No one is forcing us to half-ass our characters and rehash the same old hero’s journey plots. (As much as I went on about the difficulty of making an RPG about deep personal relationships, I’m honestly a little bemused that no one has even tried yet.) And of course, at the most fundamental level, no one told us to pick a genre that is very, very difficult to make games in. We chose that. We picked RPGs because we wanted to make them. Now we just have to choose to do something interesting with them.