Showing posts with label AppEngine. Show all posts
Showing posts with label AppEngine. Show all posts

Sunday, August 1, 2010

Watch Your Users Use

Time and time again, I re-learn this lesson: Nothing makes your system more usable than watching your users use it.

As I mentioned in my last post, I'm building a telemetry system on top of Google AppEngine so that my studio can understand how our games will be played. As I've demoed pieces of it, I've put together web pages and interactive charts and other nice little visualizations. I've made JavaScript libraries to make it easy to build those sorts of things, and I've demoed some of the prettier ones to the stable.

But I also put in a way to export data as tab-delimited fields. I figured this would be a good last resort if the JavaScript version wasn't quite there yet, or if we hadn't written a specialized view on the data, but I figured we'd favor those.

Naturally, then, my principal users have all just used the tab-delimited export and not pushed beyond that. In particular, they grab a bunch of tab-delimited data, import it into Excel, and then do manipulations on it there. Though there are macros and various other things, it still seems to take a senior engineer a couple of hours to compile all the data for his weekly reports.

I know this because I sit with him a fair amount as he explains his process.

The other day, I was thinking of his process and realized that Python can create Excel spreadsheets. So I could remove one minor barrier by just letting him get the data in Excel format instead of tab-delimited text. Easy.

But that doesn't buy him much. For his weekly reports, he trims some of the unneccesary columns (my exports grab all the fields and dump them out, including some that are system-level fields), runs some macros to insert formulas to give him percentages, sorts the data, and then copies and pastes that into an email. He does that for about 10-15 reports, in one form or another.

What if, I thought, I could let him define the format of the spreadsheet the system generates? Then he could say, "give me this data, but put it in this layout." That would shave a big chunk of time from his flow.

One thing about working with him is that he's not an online engineer. This turns out to be very good, because it makes me think in terms of letting him modify config files instead of modifying server code. (A medium-term goal is building a real interface on top of my system so that anyone in the studio can interact with it, but for now, config files are how we do things.) One of our other online engineers used the system by writing a custom request handler that interacted with the AppEngine datastore directly to generate a customized HTML page. Very neat, but if he were my only user, my system wouldn't be very evolved. I'd just say to someone new, "write a request handler."

I mentally sketched out what the config file would need to contain, what abilities it would need to give him, and within 45 minutes had a system in place that will let him generate something much closer to the final spreadsheets he needs for his reports.

A key component of the new feature is the Django templating language that comes with AppEngine. I actually dislike the language for HTML generation, but for letting users construct simple templates, it's pretty good. One important feature: It can render a template contained in a string, not just one contained in a file. And you can give the renderer a context, filling in some variables that are accessible within the template. In my case, I create a context that contains the current piece of aggregation data (which is a summary object of a large number of events) and the current row number.

From his perspective, to generate a report about how much damage a creature is doing in our testing, he creates a YAML config file that looks like this:

header:
- value: Name
- value: Avg Damage
- value: Total Damage
row:
- value: "{{aggregation.groupByValue}}"
- value: "{{aggregation.average}}"
- value: "{{aggregation.sum}}"


Then, when he requests data in Excel format, he can tell it to go through the layout he specifies. Rather than getting a dump of every piece of data in each object, he gets a spreadsheet with just the information he needs. The {{}} is part of the templating system and translates as "spit out the result of this expression."

Let's say he also wanted to capture the number of times that creature did damage. That's actually available to him in another field in the aggregation, but let's say it wasn't. My system also supports formulas. So he could do this:


header:
- value: Name
- value: Count of Attacks
- value: Avg Damage
- value: Total Damage
row:
- value: "{{aggregation.groupByValue}}"
- formula:"ROUND(D{{row_num}} / C{{row_num}},0)"
- value: "{{aggregation.average}}"
- value: "{{aggregation.sum}}"


That formula will become, in the output for the first row, ROUND(D1/C1,0).

This is still a long way from user-friendly, but I think it's going to give him back an hour and a half of time each week. And building the system this way means that a user-friendly version just needs to write the format into this config file, and away it will go.

Tuesday, July 20, 2010

Google AppEngine

Three months ago or so, I had an interesting idea for Google's AppEngine, an environment for building web applications. I wanted to use it for telemetry.

Telemetry, literally, means measuring something from afar. (Merriam-Webster's definition is somewhat circular if not outright tautological.) In the gaming industry and perhaps others, it means capturing information about how users interact with your system. Think of stats packages for web pages; they're the same thing. You also hear the terms business intelligence and plain old data analysis.

I want to be clear: This isn't about watching you, in particular, play a game. If you die in a particular spot on a particular level when our newly announced Darkspore ships, I don't really care. But if lots of people die in that same spot, it tells you something about the difficulty of that spot, doesn't it?

I needed something that could take in a large stream of data without hiccuping and allow me to process it and report on it. I went with AppEngine.*

A big selling point of AppEngine is scalability. Google obviously knows something about running big systems, and they take control of scaling your app. At any given point, I don't know how many servers are running my code: AppEngine starts up new instances as traffic grows and shuts them down when traffic slows. It's not like Amazon.com's EC2, where you manually say, "I want this many servers. No! Now I want this many!"

Closely tied to its scalability is the datastore behind AppEngine, which is a proprietary NoSQL technology called BigTable. When you run queries against BigTable, you're using a MapReduce algorithm — the same algorithm that allows Google to index and search the web so quickly. I had a potentially huge set of relatively unconnected data: AppEngine seemed like a good fit.**

And, for the most part, it has been.

But the datastore that makes AppEngine so powerful has some quirks that I've had to work around. Keep in mind that it's not a relational database, so it doesn't have the same behaviors. Most notably, it's not very good at aggregating data: sums, averages, whatever. And remember where I said that telemetry is all about aggregation? In the long run, I'm hoping BigQuery will make this easy, but in the mean time, I had to build my own aggregation system on top of AppEngine's cron jobs and task queues. The system is pretty neat, including support for a "group by" concept so that I can get views on subsets of the data in a given aggregation, but I'll be happy to let someone else handle that heavy lifting.

Another issue I've run into is that AppEngine sets hard caps on certain aspects of the system. For instance, any given request can't run too long or it will be terminated. On my own servers, if a webpage for an admin tool takes thirty seconds to load, I'm not too upset. It's an admin tool; I can wait. Google isn't so forgiving, presumably to prevent some process of yours from running amok on their machines. This has led to some early performance tweaking that's probably, in the long run, better to get out of the way now.

But these have been interesting problems to solve, so I haven't minded them so much as been annoyed by them. We're currently pulling in 100,000 events or so on any given day (as testers play the game and developers work on it), and I've only had to do optimizations on the reporting side and aggregation system. There's a whole other round of work I can do on optimizing writes, but I haven't bothered yet. Granted, 100,000 events isn't that significant, but it's nice to defer some of the deeper tuning until we're a bit closer to shipping. Meanwhile, we've got graphs and charts and spreadsheets with our telemetry data served up in a useful way to different people in the studio.

It's always hard for me to gauge the ease-of-use of a web system for someone who's not a long-time server engineer, but AppEngine feels like it is very good for either the people who know nothing about web stuff or people who know a lot about it. For middle ground types — maybe a bit of PHP or other dynamic site system — who won't be facing scalability issues, I wonder if Rails might be a better choice.

But I do know that one of our non-online engineers was able to install a new index on AppEngine for a query he wanted without even asking me. I didn't know about it until I happened to see his check-in note. That's pretty powerful. AppEngine supports both Python and Java, but I deliberately went the Python route first, despite not knowing the language at all, because there's more Python knowledge in general in my studio. And that language I didn't know? The code I wrote to take in telemetry events is virtually unchanged from the first day I wrote it. So it's easy to get started on AppEngine, and the cost is free well into the needs of most websites.

If you're interested in developing on AppEngine, I recommend the yet-to-be-released Code in the Cloud.The writing in the early drafts is a bit too exuberant, but there's solid information from the get-go, and Pragmatic Programmers delivers polished final products.

* Yes, yes. I don't trust Google, either, though, as always in blanket statements, the AppEngine team seems to have their hearts in the right place. What's to prevent them from pilfering through our data and gleaning their own information about our users? Well, there is a clause about that in the Terms of Service — they won't do it — but I'm not naive. We won't be storing information that anyone else could use to trace back to our users. An ID that makes sense inside EA? Yes, sometimes. Anything that makes any sense anywhere else? Nope. Our legal department was pretty firm on that.

** What if AppEngine goes down on our big launch day? Am I nervous about tying my company's game to a service that may not be there when I need it? In this case, not really. Let's say AppEngine goes down for a full 24 hours. The whole thing, not just pieces. Such an event would be unprecedented in my experience, but still. So we lose a day of telemetry data. Big deal. People will play the game one day the same way they played it the day before and the day after. Since I'm dealing with aggregates, I can cheerfully ignore missing data.