Michael
Let me give you a generalized overview of why I disagree with your statement ("the only other way...")
Today, there's plenty of feedback additional feedback venues for game developers. Forums are no longer the only major place we can go to get information. They are still relevant, but only a part of the big picture.
The industry has moved bounds beyond what it was even a few years ago with a heavy focus not only to proper market research and feedback solicitation but also metrics and behavioral analytics (e.g. here is something I presented at last year's GDC on that topic - the industry leaders have moved well beyond that already: http://gdc.gulbsoft.org ).
To give you an idea of what kind of feedback venues exist for most major players at this point (not specific to any actual game. Projects may use one, all, or many of these venues):
- Passive metrics (very important since all other venues actually require active participation, something a lot of people don't do. Chat log analysis, etc.).
- Testing forums.
- In active game feedback solicitation (e.g. popup surveys)
- Out of game feedback solicitation, sometimes based on in game behaviors or events (we noticed you did this...)
- In game user submitted feedback (with contextual information)
- Out of game active feedback solicitation (surveys, mail, etc.)
- Market Research focus groups and face to face discussions groups (you know, the stuff with the 1 way mirror)
- External Market Research / Trend Analysis
- Chat/Voice/Video Chat between developers and testers
- Social Media Feedback or Engagement (twitter, g+ like right here, facebook, etc.)
- Customer service interaction data
- Social Media Analysis or direct user engagement.
- Community Solicitation (General Forums) and Analytics (e.g. how many unique people actually get engaged about a topic / loundness vs. quantity filters. Hell, modern forum software has some pretty cool built in trend analysis functionality if you look around in the open source area)
- Personal engagement of players and fans at industry events and shows.
The key is being willing to listen and being able to filter, analyze and contextualize. Metrics are very powerful but notoriously easy to misread - did so many players abandon a quest because it sucked? or because it was too hard? or because they couldn't find the objective? It takes time and effort to identify such things. Solicited feedback is prone to creating a false impression of demographics, it takes effort to analyze and understand as well. If you get 1 feedback item every 5 seconds, you also need the ability to filter, aggregate and prioritize.
I think even a few years ago, most of the industry wasn't very professional about this . Metrics were used to support gut feelings or agendas or often just misread because of simple mistakes when it comes to statistics and understanding them. I remember dealing with publisher issues early in my games career (almost a decade ago) where I had to explain over and over that player death is not always a bad thing and that just because there were 70 death in one spot, that wasn't a problem because it was the same one guy generating all of them.
Unprofessional use of metrics in the early years have created animosity and fears both with developers (where will the creativity go?) and fans (They used metrics to justify a decision that broke my favorite game), which is sad. Today, major players have their own analytics groups to support business operations. Just look at the schedules for industry events like GDC to get an idea just how much of a part of the industry this has become. Quite similar to working in a large bank actually (from experience) - you have professional departments to supply your needs for decisions.
Maybe 5 years ago the primary way of getting feedback was forums. Today, the major players in this industry are a lot closer to the pulse of the (prospective) consumer, generally equipped with better and more professional market research (half a decade ago, you were lucky if your publisher was willing to share sales numbers on comparable titles) and the strongly connected world of social media has created a much more transparent consumer, which, in this case, isn't all that bad. Mistakes are still made, of course, because after all, this is a creative industry and taking risks means risking mistakes and not making only sequels.
I would argue that this change in the industry has been mostly invisible to the general forum population and the less sophisticated troll. The going assumption is still that game developers create things on paper, in a vacuum and are jolted by great feature ideas from various sources.The thing is, games have become as expensive as large scale movie productions and the garage developer mentality can't work for such setups - there are long term plans, projections, budgets, etc. all bound to those features and decisions that are made and that means there's a lot at stake with most of them - so when a major decision is made, due diligence, research and testing are a must.
If you create a really cool feature for a service based game, but it increases customer service interactions by 5 per 100 player, you need to think about how you control that. If you can't, the feature is much more expensive than the few days of implementation cost. This data was hard to measure before putting these things live, but large scale professional testing efforts (not glorified 'try for free beta') allow you to do just that. And then make business decisions based on it.
Summary: When major decisions are made, they are supported by research and validated by testing. Forums are an important way of gathering feedback and still the primary way of interfacing and understanding the sentiment and desires of your most loyal potential customers, but there are many more additional venues available to developers to get data for decision making.
Still, even with all market research and data, there are no certainties. Creative projects need to take risks and risks mean mistakes will happen. If you're lucky, you can catch these mistakes before they affect your service, but there is no guarantee. The marketplace can change underneath you while you work, public interest can change, or you may have been simply wrong all along and your target audience wasn't there. Look at movies like 'Cat Woman' and ask yourself ... did they plan for it to be a flop? Not sure what the answer is to this one though :)
No comments:
Post a Comment