Last night at the reboot of the Melbourne IGDA chapter, I gave a short talk on the state of things locally and options for indie developers. The full presentation is available below, but I thought I’d make the first half – the data on metacritic scores – a bit more accessible.
For more information on the second half – opportunities for independent developers – check out Simon Carless’ and David Edery’s presentations from Film Victoria‘s Digital Distribution Summit:
Simon Carless (from here)
Indie Game Metrics – October 2009
Digital Distribution Summit Video
David Edery
Digital Distribution Summit keynote
A more detailed breakdown of the numbers is below the fold…
The data was gathered from Metacritic for companies based in Melbourne. There were 107 games released between 2000 & 2009, with an average rating of 62.82%, a minimum of 27% and a maximum of 93%
Since 2000, the number of SKUs released has increased from 3 in 2000 & 2001 to 18 in 2009, with a peak of 20 in 2007.
The nature of these titles has changed too, with most of the growth coming in the area of porting games across multiple platforms. This has jumped from 2 titles in 2003 – 2005 to 13 in 2007 and 11 in 2009.
Broken down by platform, most of the titles shipped have been for the GBA (18), the Playstation 2 (20),and the Wii (16). Next-gen consoles are poorly represented, with only 6 titles available on Xbox 360 and 2 on Playstation 3.
Looking at the breakdown by rating shows an almost bell-curve skewing slightly to the right.
Breaking down the ratings by year, reveals a downward trend of average ratings. In 2000, the average rating was 81% with a minimum of 68%, a max of 93%, and a spread of 25. In 2009, the average rating is 63%, with a minimum of 46%, a maximum of 76%, and a spread of 30.
The minimum rating is the one that has dropped most dramatically – from 68% in 2000 to 46% in 2009, with a minimum of 27% in 2007. The maximum rating overall has dropped far less, but still tends slightly downwards – from 93% in 2000 to 76% in 2009.
Taking a look at just licensed titles, the overall trend is mirrored by the global average. In 2000, the average rating for a licensed title was 81% and in 2009 is 58%.
Splitting this into minimum and maximum ratings, both are trending downwards. In 2000, the minimum rating for a licensed title was 68% and in 2009 is 50%, with a minumum value of 35% in 2008. in 2000, the maximum rating was 93% – also the global maximum – and is 70% in 2009.
Original titles, by comparison, have fared better. The overall trend is fairly flat, although there are far fewer original titles – with none being released at all in 2000, 2001, 2002, and 2006.
Looking at the minimum and maximum, the lower values trend downwards slightly, but are also more erratic than for licensed titles. In 2003, the minimum for an original title was 75% and in 2009 is 69, with a minimum of 38 in 2008. The maximum rating for original titles has trended slightly upwards, with a maximum of 78% in 2003 and 76% in 2009, but with an overall maximum of 84% in 2007.
Looking at these values in aggregate, it can be seen that in years when original titles are released, they rate the same or higher than licensed titles.
So, what does all of this mean?
In the absence of other data (unit sales, profit, chart positions), it shows that in the core console & PC market, local developers have been producing more titles, but those titles have overall seen a drop in their critical reception. There are any number of factors that are contributing to this and I’m not in a position to speculate on those. What it does suggest though is that, as developers, we need to start thinking about alternatives to the current development model across all of the areas covered in my presentation – team structure, financial models, audience engagement, and creativity and innovation.
Great talk, Paul, and very energising while being sobering. One very important proviso is that increasingly, the types of games that are made in Australia (ports / sequels / Wii / racing / sports) are reviewed lower and lower by a game review industry (as opposed to the fallacious ‘game journalism’) that often pre-reviews titles in these categories. While working at Gamespot and IGN dealing with these games, both editorial teams explicitly told me that I was free to go all the way down the scale with these games because they needed to legitimate the higher marks given to A games. This is the very definition of a false economy. There have been -amazing- game designs in these liscensed categories made in Melbourne + Australia as you well know – but I would absolutely not assume that game critics for Metacritic rated sites would have played them for more than a minute, if at all.
So an adjunct point might be that Australia could use a ‘tasteforming’ blog like Offworld or Rock Paper Shotgun with a super strong dedicated team that pushes Australian issues and drives traffic (and therefore ratings) for games these big sites will ignore no matter how good they are.
Hey,
Thanks for putting these up. You mentioned that you had the raw data available too, and I was wondering how you collated that data, as there seem to be some anomolies coming from my knowledge at Ir gurus/TG.
In titles by platform you have only 2 titles on PS3, and both of those are ports. Transmission released both Ashes Cricket 2009 (a license) and Heroes over Europe (original) in 2009, for 360, PS3, and PC. Where these left out somehow? Or are they considered “ports” of the PC version (can’t be the 360 version since you don’t have any originals in there either)?
In the title breakdown by year, is that by SKU or by Title? The comments indicate it is by SKU. This doesn’t quite add up for 2005 either as Heroes of the Pacific was released on Xbox, PS2, PC that year, and you only have 2 originals there (unless once again the PS2/Xbox are considered ports of the PC version?). Not to mention that the numbers themselves would be completely wrong if you add that AFL 2005 was also released over 3 SKUs and GG:F on PS2 in 2005 (7 SKU’s that year from IRGurus, you only have 6 total), but i’ll forgive that somewhat since AFL 2005 and GG:F don’t even have an entry on metacritic.
That said, AFL 2005 (3 SKUs 2005) Gaelic Games Football (PS2 2005), AFL 2007 (PS2 2007) GG:F 2 (PS2 2007) GG:Hurling (PS2 2007) are probably all missing from your stats since they’re not listed on metacritic that i can see. How did you handle titles such as AFL 2006 and Equestrian Challenge which have a metacritic entry but not enough ratings to have an average score?
Cheers
@Christian
I agree that metacritic isn’t the best source of data, but sadly it is all I have easy access to. Ideally, if metacritic had an api I could do some deeper data mining. It would be interesting to see overall trends across years, along with whether or not certain outlets rated some types of games – those that fall into the categories you point out – more harshly than they deserve. Removing outliers in my data (or at least knowing where they were) would be a useful addition.
As a start – what licensed games released by Melbourne developers on PC / Console would you consider to have been rated harshly? If there are one or two I’d be interested in taking a deeper look at the reviews overall to see if removing the outliers significantly changed the overall trend.
Like I said during my talk though – everyone in the room will have their own reason for the overall downward trend. I don’t want to draw conclusions on why because there isn’t enough data to support a conclusion other than ‘based on metacritic aggregate scores, games developed in Melbourne have trended downwards since 2000.’
@Blitz
I’m looking at releasing the spreadsheet at some point.
To gather the numbers, I pretty much just typed the name of the developer into the metacritic advanced search and copied the results straight into a spreadsheet. Some games, as you’ve pointed out, weren’t represented – AFL being a big example – or didn’t have a score. In those cases, I left them out or searched gamerankings.com for a representative sample in cases where there was more than one SKU (as there was for Lucinda Green).
In cases where there was more than one version (both Heroes of the Pacific and Heroes over Europe are good examples) I treated the highest rated version as the ‘original’ and the other versions as ‘ports’ of that. All versions of Heroes of the Pacific rated 76% so it’s just a function of a sorting algorithm that I took the Xbox version as the original. In the case of Heroes over Europe, the PC version rated 66% compared to 64% on PS3 and 62% on Xbox 360 so that’s treated as the lead platform and the others as ports.
If you have ratings data for the AFL titles and the Gaelic Football titles, I’m happy to add them in.
@Paul
Hey,
Thanks for clearing that up. I don’t think I have any better ratings data for you though 🙂 I don’t think any of the gaelic games ever got reviewed in any official online media, unless it was in gaelic? I only ever remember seeing one review online for hurling and i think that was just some guys blog 😛
Cheers