Hiring Online Engineers

(By the way, Maxis is hiring).

I've interviewed a lot of candidates over the years. At my last job, we seemed to have an unending stream of candidates because we had learned to be picky. And I didn't even do the phone screens that kept the numbers down. But I did do what we affectionately called the "make them cry" part of the interview, drilling down on Java knowledge as much as I could.

At Maxis, we have less candidate churn, but I now have to sift through the resumés and do the phone screens. And I'd like to pass along some advice about getting your resumé to the top of my pile. None of these are absolutes, but the more flaws in your resumé, the greater your strengths need to be.

First, have someone proofread your resumé. I don't care if English is your first language or not; get a good editor to give you feedback. I'll wince at one typo but let it slide. Much more than that, and I begin to wonder if your code will be as unprofessional as your text. A recent candidate said in the header of his resumé that he's a "self-mortification" person. I'm guessing he meant self-motivated, but clearly not self-motivated enough to have someone sanity check his text.

In a similar vein, make sure you use the correct terms when discussing things you know. You do not "program in" AJAX, HTML, XML, or CSS. You may understand them, but it doesn't look like you do if you call them programming languages.

Next, keep your resumé brief. I've always liked the saying, "the only thing on the second page of a resume should be the Nobel Prize you won," but I am in the minority. The style du jour is to make your resume as long as possible.

I'll grudgingly accept a two-page resumé even without the Nobel Prize, but a recent applicant I saw had a many-page resumé with this line item: "Coded Perl functions, invoking subroutines and functions calling functions." In other words, you did some programming? And that was simply the most ridiculous in a long list. "Set up Object-Relational Mapping with Hibernate" is another common line item. For those who don't know, this involves writing a config file. If you have items such as these on your resumé, you're padding, and I'll think you need to pad because you don't have any real skills.

If you're a mid-level Java engineer with some enterprise software experience, I'm afraid your resumé looks like approximately 100,000 others. Spring, Hibernate, JUnit, Struts, MySQL, Oracle. I'm yawning already. So make your projects sound interesting. We don't all get to work on famous video games, but before I did, I worked on projects you've never heard of. And I made them sound neat. I focused on the compelling problems and described those. If you can't find interesting problems in your work, you're not a programmer I want to work with: Every problem is interesting in its own way.

I also look for personal projects on a resumé. Yes, people have families and whatnot. But think of great writers. They don't just write because they are required to: They write because they need to.

If you're working on personal projects, no matter how esoteric, you're telling me that you're so in love with programming that you can't simply kick it aside when you clock out. You're telling me you love the craft, the problem-solving, the tinkering. You're telling me that you're a programmer I want to work with.

Visualizing My Bike Rides

My three main interests are programming, writing, and food and wine.

You'll notice exercise isn't on that list. In fact, quite the opposite: My primary interests are all anti-exercise. But I grudgingly acknowledge its value, and I've decided to "trick" myself into exercising more by turning it into a programming project.

I live close enough to work that I can commute on my bike, and so I've started a project to gather data on my rides and do interesting things with that data, inspired in part by Cooper Smith's visualization of Nike+ data from New York City. Since I need a lot of data to make useful visualizations, I'm riding more consistently to get it. And hopefully by the time I have enough data, the bike riding will seem routine.

I picked up a couple of bike tracking apps for the iPhone, but have settled on Abvio's Cyclemeter, based on the recommendation of a co-worker who is both a data geek like me and an avid cyclist. You press Start on your phone, ride your bike, and press Stop when you're done. It gives graphs, maps, and all sorts of other goodies.

Getting to the data is then just a minor step: All of these apps seems to support exports in KML and GPX. Since these are actually just my rides, that data isn't all that interesting by itself. I know how I get to work.

But with that data, I can create meta-analyses. For instance, how does my speed look across a given ride? Here's a ride I took from the Saturday Berkeley farmers market to Berkeley Bowl, our preferred grocery store.

Green lines indicate places where I was faster than my average speed for that ride. Red lines indicate places where I was slower. I add in the Start and Stop pins, and also provide meta information about the ride off of the extra data in Cyclemeter's GPX file: total distance, average speed, and so forth.

Ruby made this work pretty straightforward. I use Nokogiri to parse a GPX file and calculate the velocity between subsequent points. Each velocity item has the coordinates and timestamps of the two points as well as the calculated velocity. I then use an ERB template for the KML I want to create. That ERB template sets up the styles and other items, and then uses the state variables to construct the line segments, the start/stop pins, and other items.

Eventually, I want to add arrows indicating the direction (which is more useful when you're looking at lots of overlapping routes), pins for the slowest point and the fastest point, and other items.

That will do for individual routes, but I also plan to start aggregating my rides to show even more data.

Visualizing A Week Of 99.7

When we're driving, Melissa and I often listen to Bay Area radio station 99.7 FM. It specializes in dance-focused pop, club, and hip-hop songs. In other words: boppy, brainless music.

But at any given point in time, it feels like they just replay the hits du jour. How much diversity do they really have? I wanted to know.

The answer? Not much. A mere nine songs made up half the station's rotation in the week I measured. During that week, about half the songs you would have heard would have been one of those nine songs.

I also made an interactive version that requires a Java-enabled browser. It shows the song titles as you hover over them.

Here are the top 9 songs:

Song Title	Artist	Times Played
The Edge of Glory	Lady Gaga	100
I Wanna Go	Britney Spears	75
Till The World Ends	Britney Spears	69
How To Love	Lil Wayne	59
Stereo Love	Gym Class Heroes	54
Rolling In The Deep	Adele	53
Written In The Stars	Tinie Tempah	52
The Lazy Song	Bruno Mars	46
Cheers (Drink to That)	Rihanna	39

To gather the data for this chart, I used the station's published playlist, which lists 25 songs at a time. I set up a micro-instance on Amazon.com's EC2 to run a Ruby script every 45 minutes that fetched that playlist page, extracted the information I wanted (via regexes and the excellent Nokogiri library) and appended it to a file.

Then I set up a node.js server* that returned a cleaned-up version of the raw playlist data I amassed. It removed duplicates caused not only by my data fetching script, whose 45-minute interval meant that there was always some degree of overlap, but also by a peculiarity in the data. Remixes on the site get two or more entries with the same timestamp, one for the original song and one for the remix title, and I collapsed those down into the original song. A remix of Britney Spears' "Till the World Ends" might be different in some ways than the original, but to me it counts as playing the same song. I've published the final dataset I used for this chart.

Along the way, I discovered two lacunae in the website's playlist — there are probably more — which affect the numbers a bit. As Nathan Yau says in his new book Visualize This, "Just because it’s data doesn’t make it fact."

Katy Perry's "Last Friday Night" and Pitbull's "Give Me Everything" got plenty of rotation on the station during this week but never showed up in the published playlist. I checked my raw data, the cleaned-up form, and did spot checks on the site whenever I heard one of those songs. They're just skipped. I assume this is some discrepancy in the database, since other songs from the same Katy Perry album show up in the list.

I don't know that it would change the numbers very much. There might be a more gradual drop-off from "Edge of Glory" to "I Wanna Go," but, if anything, the halfway mark would be closer in. The gap doesn't change the premise: 99.7 replays a lot of the same music.

For visualizing the data, I put it into a couple of tools -- a custom tool I'm writing as well as R -- but ultimately decided on Processing, the big gun in any data visualizer's arsenal. Processing is a full programming language aimed specifically at making digital images, with an emphasis on visualization. I could both fully churn through and munge the semi-raw data and quickly visualize it, all with the same tool. And since Processing is basically Java with some handy utility methods, I'm already very comfortable with the language.

Inspired by Yau's book, which encourages a storytelling mindset, I decided to add visual cues and callouts for "points of interest" to my graphic: the most popular song, the cutoff line for the songs that made up fifty percent of the total, and the cutoff line for the the songs that made up eighty percent of the total.

Because Processing is a programming language, I drove everything off the data itself. While I obviously had to program the callouts I wanted on the chart, I don't have a line that says, "Draw 'Edge of Glory, Lady Gaga' at these coordinates." Instead I have a line that says, "draw the name of the song that got played the most next to the leftmost bar." I used the same mindset for all the callouts. Change the dataset, and the callouts change with it.

Once you have all this shiny, pretty data, you start looking at other ways to explore it. For instance, what's the average number of songs played in each hour? A little bit of modification to my Processing program, and I had a new chart ready to go from the same data.

Then Melissa and I wondered if Bruno Mars' "The Lazy Song" gets played a lot more on weekends, since it's, well, about being lazy and deciding to do nothing with your day. Not really. As a rule, expect to hear it six to eight times a day at the moment, and not more on weekends.

I have more ideas for this, but they're going to take a bit more data collection, so stay tuned for more in the coming weeks.

*There was no need to use node.js here. It just gave me a chance to play with it for something deeper than the "Hello, World!" example.

More With Google's WebGL Globe: Legen - Wait For It - Dary

In my last post, I talked about Google's WebGL data visualization globe and briefly mentioned the "legend" format for the data array.

I finally got a chance to do something with it, and it is both simple and powerful. If you're in legend mode, your big data array is four pieces of data at a time, not three. The first two pieces are latitude and longitude. The third is the magnitude of the line that will be drawn (divided by 200 to compensate for their multiplying by 200). So what is the fourth value?

Anything you want. When you specify legend mode, the color function you pass in to the globe's constructor gets that fourth value for each point on the globe. That in turn allows you to define the color of the line based on something other than magnitude (the default).

What that really means is that you get an extra dimension in your data. Height is always height, but legend mode allows you to add non-height information based on the color of the line.

Google's initial example — search language by volume — is a great example. The color of the line comes from the dominant language of the area. Looking at their globe, you can immediately figure out where the high search volume comes from (big cities, mainly) just by looking for the tall lines. You can also see which languages dominate search in a given area. English (blue) covers the United States and the United Kingdom. French (light green) covers France and also Quebec. Portugese (dark green) dominates Portugal and Brazil, and also Madeira. Likewise, the Canary Islands are yellow, because they're a part of Spain.

For a new visualization at work I did with the globe (as usual, NDAs make me cautious about giving more exact details), people at my studio wanted to see the height of the line represent the number of events at a given location. But they wanted the color to represent whether the average data from that spot represented a "good" or "bad" user experience. Little red lines would be unfortunate, but might not get flagged as high priority. Big red lines would be a problem, because it would mean that a lot of players were having a bad experience. Fortunately, there weren't any of those, but there were some long yellow lines, which suggests an area where we could improve the player's interaction with our game.

To get that view, I set the globe to legend mode and set the magnitude field of each point to the sample size. Then I set the fourth data point to a very heterogeneous number that represented the user experience. My color function looks at that number and puts it into a bucket. Good returns green, bad returns red, and a middle ground returns yellow.

That means each line on my globe gives three pieces of information: location on the planet, sample size from that location, and quality of user experience from that location. It's easy to spin the globe and look for red hot spots. It's also just a pleasure to play with the visual, even though you can find the worst spots pretty quickly.

One person saw my work and suggested adding a calculation that would make the line more or less red, for instance, depending on how far over the bad threshold it went. Bright red would mean a really bad experience; dim red would be right at the line. I guess that would give us three and a half data points. Before, we knew it was bad. Now, we'll know how bad it is.

Legend mode makes this all possible.

Working With Google's WebGL Globe

Today, Google released a data visualization that shows search volume by language across the globe. It uses WebGL, so you'll need a recent version of Chrome and decent video drivers to see it.

It's different than the normal "things on a globe" visualizations you see in — for instance — Google Earth, because it incorporates height as an additional dimension. Google Maps and Google Earth give flat perspectives: You can only guess magnitude based on clusters of the familiar red, upside-down teardrops.

By itself, the visualization would be a five-minute folderol to enjoy on a lunch break. But Google's Data Arts team released all the source. Take your own data, muck with it a bit, and you too can have an interactive globe dripping in an oh-so-modern, shades-of-black aesthetic.

I took the bait.

Over the course of today, I took some of our game analytics data and built a local WebGL globe visualization. We have an established workflow for creating latitude/longitude tables from our data — this isn't the first map visualization I've done — and I built on top of that. Once I extracted the information I wanted, I wrote a quick Ruby script that converts the exported data into the format their code looks for.

I can't share a link with you — it's on our internal network — and I can't tell you what I mapped. But I can tell you that it got lots of oohs and aahs in the office.

I can also tell you that it wasn't a simple "replace their data with our data" exercise. If you're planning on doing something with the tool, here are a few things I figured out by trial and error.

1. Note: This has since been fixed. Don't rely on their README: It's wrong. At least, the JSON format is. It's much better to read the globe.js file to see what it wants, though that requires you to know JavaScript. Rather than a complex set of nested arrays, the code prefers one long array with latitude, longitude, and magnitude strung together like beads on a string. (There's an optional "legend" mode that requires a fourth point, but I haven't played with it. I assume it lets you define different data series. Their addData method takes the data block and a set of options, and one of those options is format: The default is magnitude — three data points — and you can specify legend — four data points.)
2. A portion of their code multiplies the magnitude by 200 to ensure the small numbers in their data — percentages, I suppose — become big enough for bars that reach high into the sky. For our particular data set, I had to divide the number by 200 in order to get the bars to be correct once they multiply it by 200.
3. The default coloring of the lines relates to the height, but it looks like you can pass a custom "get the color" function for different logic.
4. Their source code relies on a top-level directory on your website named /globe for the location of the map image that wraps onto the sphere. You can change that easily enough. Search for "imgDir" in the source.

More Readable Hash Member Access In Ruby

I've been doing a lot of work with the amazon-ec2 Ruby library for working with Amazon Web Services. I've also been using YAML a lot for config files. But both those systems return big nested hashes with string keys, so my code ends up looking like this: config['database']['security_group']

That gets tiresome to read. To fix that within these scripts, I extended the Hash object by adding an implementation of method_missing?, which Ruby calls when you try to execute a method on an object and it doesn't exist. My implementation says to try and use the method name as a key into the hash.

class Hash
   def method_missing(method_name, *args, &block)
      fetch(method_name.to_s,nil)
   end
end

Now the same code from before looks like this: config.database.security_group. Ruby tries to call the "database" method of the config object (a Hash), fails, and then my extension to the class looks up a key named 'database' in the hash and returns that.

I find this much easier to read, particularly when it's repeated throughout the file. Of course, it assumes the keys in the hash are strings, but in this script they are (so my extension is only added for that script).

One-Button Loadtesting With EC2

I recently realized a minor ambition.

As I've said before, EA's loadtesting group doesn't meet all my needs: They come in near the end of a project and test the nearly-finished system, which means that it's a big headache all around. Fixing performance issues means rummaging through months — or even years — of code to try and suss out issues they're seeing. Also, the mere act of getting them up and running is usually a big disruption for one or more members of your team.

I had loadtesting scripts running already, but they suffered in a few ways. JMeter is good at what it does, but I want easier programmatic control of the scripts (especially for measuring things it's not so good at, such as the lag time on asynchronous processes under load). Also, I didn't have a loadtesting environment. I was running the scripts on a dev server against that same server. I could have put together an environment of multiple servers, but that would incur expense to the studio for machines that were largely idle.

Then I read an article somewhere about a company that uses EC2 as a dynamic loadtest environment. Yes, I thought, that's what I want.

Off and on over the last few weeks, I put together a Ruby script that can, with a single command, set up a full environment in Amazon's cloud-based servers. That one command instantiates an RDS database, an EC2 server that will act as our application server, and another EC2 server that acts as an instance for running The Grinder, a loadtesting tool that another team in the studio is using and that I liked for its extensibility. That one command also builds a custom version of our war file (to handle all the dynamic addresses you get from EC2/RDS), sets up access permissions between all the servers, and runs checks to make sure the whole environment is ready to go.

A second command fires off the loadtest itself, running the test script through The Grinder and collecting the results back to the local machine.

Finally, a third command shuts the whole system down. I do have one command that runs all three steps, but at the moment I tend to run each one manually.

Each run takes less than an hour for now, so the cost to the studio is on the order of $.28 per loadtesting run. Not something you'd want to run continually, but certainly something you could run once a day. As we move along in development, we'll probably need to spend more to set up bigger, more realistic environments, but that will also be when the studio has more budget for my project. The environments we need will scale up alongside the money we have.

One of the key pieces that I set up was the concept of a profile. A config file lets you specify which script to run and how many of each item to set up. So, for my first profile, I just set up one each of the different servers. But you could imagine some profile later down the road that sets up 2 of each machine with some sort of autoscaling system. And you could imagine one much further down the road that sets up something approximating our production environment and runs tests against it. All that is mostly supported.

The advantages to this system are huge. First, it makes real loadtesting feasible even early on in our project. But it also empowers developers to do rapid iteration on performance fixes without having to push them live and see if they make a difference. See a problem in the results, figure out a fix, run the script again to test. Repeat and then release when you can prove that your fix makes a difference. Because each script fires up its own environment, I'll be able to distribute performance fix work to my team.

This ad hoc loadtesting tool is already proving its worth, and I've only just started employing it. I can't wait to see how effective it is going forward.