An Obsession With Programming

Books For Systems Geeks

2011-12-19T20:55:00.000-08:00

I've been involved in lots of system releases over the years. Not only major versions and upgrades for the various companies I've worked for but the monthly or weekly minor (or, "minor") releases that are the norm in startups everywhere.

And like everyone else who's gone through those circuits, I've picked up a bunch of ad hoc, empirically proven knowledge about how to do these things. I know what's worked, and I know what hasn't.

But as the person helming the online systems for my studio's next major game, I thought it was a good idea to see what other people have to say about the complex art of launching a big system. Not just making it scale but the processes and practices that make it a smooth, worry-free (or as close as one can hope) launch. No sense in avoiding the wisdom of others, after all.

Here are my thoughts on a few of the books I've read recently.

The Art of Scalability - This is one of my favorites, and it's a book I'll come back to again and again. While it's light on actual technical details, it does a great job of explaining how scalability decisions are actually business decisions and how you need a culture of scalability, not just a decision to shove it all in at the end.

That sounds obvious, right? That's the other thing the book does really well: Encapsulating ideas you, like me, have probably learned on your own into nice, articulate concepts. I know about horizontal scaling, sharding, and splitting up servers based on functionality. But when they talk about their X-, Y-, and Z-axis scaling cube, it summarizes it quickly. Little phrases like "technology agnostic design" and "swim lanes" become keywords that you can quickly call up when you're thinking, "Something about this doesn't sound right."

Release It! - I was scribbling down notes constantly while reading this book. This is the distilled advice of someone who's seen lots of systems work well and poorly. It's one of the few tech books that is super relevant even five years or so after it was published. (Though it merely hints at the cloud technology that has sprung up since then.)

It documents a slew of patterns and antipatterns that will have you nodding your head constantly. A lot of the topics are things that you probably kind of know if you've done a bunch of launches. But this book takes them out of the realm of intuition and into concrete knowledge, real experience, and practical advice.

Its notion that the time before launch is a comparably short time in your product's life versus the time it spends live won't necessarily apply if, like me, you work in the games industry, where years of work go into a product whose user base drops off, under normal circumstances, rather than grows. But that does not diminish the value of the text. (And I won't always be in games.)

Scalability Rules - This book is the sequel to The Art of Scalability, though perhaps companion is a better word. Whereas the authors were light on tech and heavy on business in the first book, this book is all about the technical concepts you need to launch scalable systems. Again, lots of great advice that I scribbled into my notes. (This book has more to say on cloud systems than the original, earlier book did.)

Best of all, they provide a handy list of the 50 rules in order of "amount of risk management you get" and also provide a rough guess at the amount of work. Sort by most bang for the least buck, and you'll be knocking out a few in no time.

Scalable Internet Architectures - On the other hand, if you really want technical advice, this book gets way down to the nitty-gritty. There's a lot of good info here, but it's going to be most useful if you're on the IT/operations side of the fence versus the development side. That said, his argument that you should focus on scaling down (i.e., cutting costs) as well as scaling up has become a fixture in my architecture.

Continuous Delivery - The notion of continuous delivery, which means that your software is always ready for launch, is a compelling one. The authors are basically trying to convert the seemingly inevitable tension around a production release into a ho-hum experience that most of the team probably doesn't even need to be aware of.

It sounds great, but I'm not sure how much I'll be able to apply to my day-to-day work. The authors set a high bar when they cavalierly suggest that you should have 90 percent coverage just from unit tests. I used to feel fairly good about the fact that I had 40 percent coverage from unit tests and integration tests.

Still, I definitely came out with some ideas that will make our own launches more relaxed once I implement them.

My Radio Broadcast Podcast

2011-12-03T16:30:00.000-08:00

While I have an enthusiasm for boppy, bubble-gum music, particularly as a backdrop to coding, I have also had a passion for opera in the past. I've even had questions used in the Opera Quiz on the Metropolitan Opera radio broadcasts.

Unfortunately, that passion is hard to fit into my life these days. Opera tickets are expensive unless you want to stand, and performances take a long time. This is the nature of opera.

This used to be easier. On Saturday mornings, I'd turn on the radio broadcast and listen to the opera for several hours as I went about my morning. Then I got into food. And farmers markets, my favorite of which are on Saturday mornings. That then became my normal food shopping day, and the radio broadcast went by the wayside. I'm usually coming home from shopping right as the opera ends.

Sure, you can listen to albums, but the Met broadcasts are fun because they provide context. The host describes the costumes, experts provide backstory, and they do the quiz.

When podcasts became popular, I realized that a podcast of the Met's Saturday matinees would be perfect. I sent letters asking the Met to do it. When they called and asked for money, I'd mention it. I'd even tell them that I would pay for such a podcast. Imagine: paying for Internet content!

They've never done it. And so I've fallen behind on opera, enjoying it as much as possible with a ticket or two a year to the San Francisco Opera.

Earlier this year, I was ranting about this yet again when I realized that I could probably craft my own podcast based on the radio broadcasts. What I wanted was the ability to call up a podcast on my iPod and see the latest opera broadcast, already synced. So began a day or two, off and on, of work on a podcast-creation script that would use radio stations as its source material.

Rube Goldberg would like this one.

First, I needed to figure out how to capture the music. A friend suggested FStream, Mac OS X software that had two valuable features: I could open a wide range of streaming URLs with it, and it was scriptable. I like scriptable apps. And these days, one can even use Ruby to do that scripting.

What I ultimately wanted was to not even think about this. That meant that my script would need to know a schedule. It reads in a config file with YAML entries that contain the name of the item, the start time, end time, and streaming URL. When the script runs, it parses the file and checks once a minute to see if it should be recording. If it should (and it isn't), it starts up FStream, points it to the appropriate URL, and tells it to start recording. When it reaches the end time, it tells FStream to stop recording.

Once the file is closed, the script uploads it to S3 and creates an XML file that points to all the appropriate links of the files. Voila: a podcast of streaming radio.

Though I did this originally with the Metropolitan Opera broadcasts in mind, it obviously works for any streaming radio. I've set up new entries for CapRadio's Friday Night at the Opera, and KDFC's San Francisco Opera broadcasts.

There are a couple of problems with this script. One is a bug that causes it to throw an exception when uploading the file. I'll fix that at some point. It just means I have to manually upload the files to S3. The other problem is logistical: Neither of our computers is on all the time. To add one more step to my baroque script, I set myself a reminder so that I know to set up my computer during the appropriate time period. I wonder if iCal is scriptable …

In addition to all the stuff this script is supposed to do, the script passes its config file through Ruby's ERB system. That means that I can actually set up my config file so that the start times are programatically driven (e.g., 9:00am on the coming Saturday).

I'd still like the Met to do a podcast of their own. I'd even still pay for it. But until they do, I not only have their broadcasts in a podcast, I have a wealth of others.

Here's the script, with various sensitive bits taken out. One thing I've found useful is to put my S3 connectivity information in a separate, included script so that I can distribute the main script and not accidentally include my S3 credentials. On the off chance you want to use this script, you'll need a file named aws_info.rb in the same directory as this script which defines three variables: S3_BUCKET_NAME, S3_ACCESS_KEY, and S3_SECRET_ACCESS_KEY.

require './awsinfo'
require 'rubygems'

require 'appscript'
include Appscript

require 'fileutils'
require 'time'
require 'yaml'
require 'aws/s3'
require 'erb'
require 'rss/1.0'
require 'rss/2.0'
require 'rss/maker'

class File
 def name
     pieces = self.path.split('/')
     pieces[pieces.length - 1]
 end
end

SLEEP_INTERVAL = 60
PODCAST_FILE = "podcast.xml"
PODCAST_URL = "https://s3.amazonaws.com/#{S3_BUCKET_NAME}"

$schedule_file = 'opera_schedule.yaml'
$is_recording = false
$current_schedule = nil

#constants are defined in awsinfo
def start_s3
 AWS::S3::Base.establish_connection!(
    :access_key_id     => S3_ACCESS_KEY,
    :secret_access_key => S3_SECRET_ACCESS_KEY
  ) 
end

def stop_s3
 AWS::S3::Base.disconnect!
end

def parse_time(time_string)
  regex = /^(\d*?)-(\d*?)-(\d*?)\s*?(\d*?):(\d*)/
  year = time_string[regex,1].to_i
  month = time_string[regex,2].to_i
  day = time_string[regex,3].to_i
  hour = time_string[regex,4].to_i
  minute = time_string[regex,5].to_i
  Time.local(year,month,day,hour,minute)
end

def parse_schedule_file(filename = $schedule_file,time=Time.new)
   schedule = File.open(filename,'r') {|file|YAML::load(ERB.new(file.read).result binding)}
end

#side effect: sets $current_schedule if appropriate
def should_be_recording(time=Time.new)
  schedule = parse_schedule_file(filename=$schedule_file,time=time)
  schedule_flag = false
  schedule.each do |schedule_entry|
     start_time = parse_time(schedule_entry['start_time'])
     end_time = parse_time(schedule_entry['end_time'])
     if (start_time..end_time) === time then
         schedule_flag = true
         $current_schedule = schedule_entry
         break
     end
  end
  schedule_flag 
end

def next_scheduled_task(time=Time.new)
    schedule = parse_schedule_file(filename=$schedule_file,time=time)
    sorted_schedules = schedule.sort {|a,b| parse_time(a['start_time']) <=> parse_time(b['start_time'])}
    return_schedule = nil
   
    sorted_schedules.each do |entry|
      if parse_time(entry['start_time']) > time then
        return_schedule = entry
        break
      end
   end
   return_schedule
end

def add_file_to_podcast(file,schedule_info=$current_schedule)
   xml_file = file_from_pieces(fstreams_dir,PODCAST_FILE)
   rss = nil
   if !File.exists?(xml_file) then
      rss = RSS::Maker.make("2.0") do |maker|
         maker.channel.title = "Derrick's Radio Podcast"
         maker.channel.link = PODCAST_URL + PODCAST_FILE
         maker.channel.description = "Radio programs captured by script"
         maker.items.do_sort = true # sort items by date
      end
      File.open(xml_file,"w") {|file| file.write(rss)}
   end
    
   content = ""
   File.open(xml_file,"r") do |existing_file|
       content = existing_file.read
   end
   rss = RSS::Parser.parse(content,false)
  
   item = item = RSS::Rss::Channel::Item.new
   item.title = schedule_info['name']
   item.date = File.mtime(file)
   item.link = "#{PODCAST_URL}/#{File.new(file).name}"
   item.pubDate = File.mtime(file)
   item.enclosure = RSS::Rss::Channel::Item::Enclosure.new(item.link, File.size(file), 'audio/mpeg')    
   rss.items << item
  
   File.open(xml_file,"w") {|file| file.write(rss)}
end

#todo: use fstream.recording flag
def is_recording
 fstream = app('FStream')
 puts fstream.status
 $is_recording && fstream.status == 3
end

def fstreams_dir
  fstream_path = './fstreams'
  FileUtils.mkdir_p(fstream_path)
  Dir.new(fstream_path)
end

def file_from_pieces(dir,file)
  "#{dir.path}/#{file}"
end

def sync_dir
 dir = fstreams_dir
 start_s3
 dir.entries.each do |filename|
    next if filename =~ /^\..*/
    file_path = file_from_pieces(dir,filename)
   
    # if the file doesn't exist (or it's the podcast file), upload it
    if !AWS::S3::S3Object.exists?(S3_BUCKET_NAME,filename) || filename == PODCAST_FILE then
        AWS::S3::S3Object.store(filename,open(file_path),S3_BUCKET_NAME)
    end
 end
 stop_s3
end

def s3_safe_name(english_name)
  english_name.gsub(/\s/,'_').downcase
end

def start_recording
 $is_recording = true
 puts 'Starting to record'
 fstream = app('FStream')
 fstream.openStreamWithURL($current_schedule['from_url'])
 fstream.startRecording
end

def stop_recording
 puts 'Stopping recording'
 fstream = app('FStream')
 fstream.stopRecording
 fstream.stopPlaying

 # find the file most recently created, and rename it
 dir = fstreams_dir
 filepaths = []
 dir.entries.each do |filename|
    next if filename =~ /^\..*/
    file paths << file_from_pieces(dir,filename)
  end
  filepaths.sort {|a,b| File.mtime(b) <=> File.mtime(a)}
  filepath = file_from_pieces(dir,s3_safe_name($current_schedule['name'])+".mp3")

  FileUtils.mv(filepaths[0],filepath)
  
  # file can take a while to close
  while true
     begin
       sleep(10)
       File.mtime(filepath)
       break
     rescue exception
       puts "Waiting for #{filepath} to close"
     end   
  end
  
  # update podcast file and S3
  add_file_to_podcast(filepath)
  sync_dir
  
  $is_recording = false
  $current_schedule = nil
end

def do_start_app
  while(true)
    if !is_recording && should_be_recording then
      start_recording
    elsif is_recording && !should_be_recording then
      stop_recording
    else
      next_sched = next_scheduled_task
      puts "#{Time.new} Next scheduled task is #{next_sched['name']} starting at #{next_sched['start_time']}" if next_sched
    end
    sleep(SLEEP_INTERVAL)
  end
end

$schedule_file = ARGV[0] if ARGV.length >= 1
do_start_app if !$testing_mode

Partitioned Concurrency

2011-11-03T19:00:00.000-07:00

Here is a common block of code in Java. Assume cache is a Map of some form.

if (!cache.containsKey(someKey)) {
  cache.put(someKey,someValue);
}

Or: if a given object is not in the map with a certain key, put it there. Otherwise, proceed along your merry way.

I was writing code like this in a controller the other day, because I wanted to cache Broadcaster objects (from the Atmosphere framework) that would manage separate chat channels. Requests would come in and get attached to one or another Broadcaster based on a channel ID.

But that code snippet, in its current form, isn't thread-safe. You could have two threads get past the if and thus both think that the map needs the object inserted. That isn't always a problem — my reflection-based API marshaling layer caches Field and Method objects that are constant for a given class, making me unconcerned about threads that replace values — but in this case, you could have the second thread replace the Broadcaster that the first one inserted, meaning that some chunk of the chat clients wouldn't see messages because they'd be attached to the wrong object.

The standard way to solve this is to put a mutex around the cache-modification code:

synchronized(this) {
  if (!cache.containsKey(someKey)) {
      cache.put(someKey,someValue);
  }
}

But this is code that is likely to have a lot of requests fired at it, and synchronizing this way introduces a major performance bottleneck. Every single request would have to wait for that lock. Yes, I know: Premature optimization is the root of all evil. Still, it seemed problematic. Synchronization at that high of a level can cause major issues when you're dealing with hundreds of thousands of concurrent users, which we may very well be when we launch.

As I was pondering this, I remembered a technique for increasing concurrency. I had discussed it with friends before, but never implemented it. Still, it's straightforward enough. (In fact, it's how java.util.concurrent.ConcurrentHashMap is implemented, or near enough.) Note that this technique should also work in Objective-C and any other language with similar semantics, though I haven't tried it anywhere other than Java.

Here's the idea: If you've got some sort of cache, you only really need to make sure that two threads aren't working on the same cache entry at the same time. If I have a Broadcaster, and I attach it to some ID for the channel, I only care about isolating activity around that ID. I don't care if someone's working on some other ID. (except when I do: see below) In other words, if I'm working with an ID of 3, I don't care if some other thread is checking about whether or not ID 1 already exists; I only care that someone else asking about ID 3 doesn't cause problems.

So if you could create a sequence of locks, and just ensure that anyone working on the same ID ends up synchronizing on the same lock, you're good to go.

Consider this code:

public class ConcurrentCache {
  private static Object[] locks = new Object[10];
  static {
       for (int lockIndex = 0; lockIndex < locks.length; lockIndex++) {
               locks[lockIndex] = new Object();
        }
    }

    private Map cache = new HashMap();
    public Object getOrInsertIntoCache(Object key) { 
           synchronized(locks[key.hashCode() % locks.length]) { 
                 if (!cache.containsKey(key)) {
                      cache.put(someKey,new Object());
                 } 
               return cache.get(key);
            }
     }
 }

The static initialization code gives you ten locks to work with, and you can get to one by just modding the hash code by the length of the list. Any given ID will always end up with the same lock (assuming a consistent hashCode result, which is true of Long, Integer, String, and other object representations of Java primitives typically used as keys.

To test this, I wrote a simple program that would put 5,000 tasks on an ExecutorService with ten threads. Each thread would generate a random number between one and 100. That would become the key to use on the cache. Depending on some command-line arguments, those threads would either lock on the cache or on a partitioned lock as above. Any given thread captured current system time when it was constructed (put on the queue) and then printed the difference between the new system time and the start time when it eventually ran. I ran the program and eagerly checked the average wait times each thread experienced.

Only to discover that they had almost identical performance. The partitioned code showed threads waiting a mere millisecond less, on average, than the "block on everything" code. And that was averaged over 5,000 jobs, remember. That's a lot of complexity for not a lot of gain.

I figured out what was going on with a bit of poking. The basic code says: See if the key's already there, and, if it's not, insert it. What that really meant was that 4,900 of my jobs called containsKey, saw that the key was there, and exited. I had an extraordinarily high cache hit rate. The lock was acquired and released so quickly that there simply wasn't noticeable lock contention.

Once I realized that, I made a simple change. After doing the cache logic, I simply had each thread sleep for one millisecond. And I ran my program again.

That produced the results I expected! Threads in the "block on everything" version waited almost precisely ten times as long, on average, as their partitioned cousins (29 seconds versus 3).

Real-world caches are messy things. In fact, my simple test case wouldn't really be done this way at all: You'd prefill the cache with the fixed items you wanted and avoid all this nonsense. But real caches need to expire items, cached results can be more or less complex, and so forth. The caching situation I have involves lots of different IDs with corresponding amounts of object churn. So the in situ results will likely be very different than an almost exact division. Still, it obviously made a big difference for time-consuming activity and at least didn't hurt performance in the simple case. You have more code complexity, which means more potential bugs and less maintainability, but in my system this logic is tucked into a class by itself, so no clients need to worry about these details: They just request a Broadcaster and don't worry about how it gets to them.

Still, if your cache handling is no more sophisticated than this, synchronizing on the whole thing was basically equal in speed, and you should therefore avoid the readability/maintainability cost altogether. If you're worrying about a one-millisecond difference across 5,000 tasks in your server code, you're farther along in your optimizations than I am.

There's also the question of what to do when you do care about the overall state of the cache. For instance, what if you want to get the size of this cache? The above technique won't work, because no one lock guarantees a fixed state. (Ignoring the reality that you probably don't actually care about the exact size of the cache: You simply want the approximation, in which case you're fine.)

Actually, there is one lock that will guarantee the state of the overall cache, and that's the cache itself. If you really care about exact counts, you can synchronize on the cache itself for anything that changes its size. For instance:

public class ConcurrentCache {
  private static Object[] locks = new Object[10];
  static {
       for (int lockIndex = 0; lockIndex < locks.length; lockIndex++) {
               locks[lockIndex] = new Object();
        }
    }

    private Map cache = new HashMap();
    public Object getOrInsertIntoCache(Object key) { 
           synchronized(locks[key.hashCode() % locks.length]) { 
                 if (!cache.containsKey(key)) {
                      synchronized(cache) {
                          cache.put(someKey,new Object());
                      }
                 } 
               return cache.get(key);
            }
     }
 }

You re-introduce the global lock, but you minimize when it's acquired. If you have a high cache hit rate, this should still give you better concurrency but allowing for across-the-board thread safety (though, again, do you really care that your size is 99 and not 98?)

This isn't a new technique, but it was worth jotting down so I don't forget it.

Faking Block Programming In Java

2011-10-15T14:27:00.001-07:00

While I was writing some server-side code to talk to a web service, I did the total n00b thing of forgetting to close my connection when I was done with the call. This works fine in test cases, but in any sort of real-world situation, you'll quickly exhaust your connection pool as connections linger, unavailable, until they time out. And as I fixed the problem, I realized I had done the same thing in one other place — I haven't written client code in a while. Then I realized that I had a recurring pattern:


HttpMethod get;
try {
    get = new GetMethod(url);
    get.execute();
    
    // pull the response body out and do stuff with it
} finally {
   if (get != null) {
       get.releaseConnection();
   }
}

This ensures that, even in the case of an Exception, the connection gets released. Fairly straightforward. But who wanted little copies of that code all over the system? And, worse, what if someone forgot to do this, just as I did? Anytime you set up code people have to remember to type, you ensure that someday someone will forget.

This would be an obvious use case for a closure. In fact, it reminded me of the File.open method in Ruby that takes a block of code as an argument. The method creates a File object, calls the block of code you pass in with said File object, and then closes the file even if there was an exception.

The only problem: pure Java doesn't support closures. (Some JVM-compatible languages like Scala do, however.)

But you can mimic the behavior to some degree with anonymous inner classes, and you can use Java Generics to provide type checking. I created a MethodOperator interface that looks like this


public interface MethodOperator<T> {
    public T actOnMethodPostResponse(HttpMethod method) throws Exception;
}

The <T> and public T … bits basically mean that when I have to instantiate one of these, I can also declare it as being of some type, which then gets returned from the one method.

Once I had that code, I added a simple utility method:


    protected <T> T actOnHttpMethod(HttpMethod method, MethodOperator operator) throws Exception {
        try {
            executeMethod(method); // utility method that checks for errors and so forth on the method
            return (T)operator.actOnMethodPostResponse(method);
        } catch (Exception e) {
            log.error("Error talking to http server",e);
            throw e;
        } finally {
            if (method != null) {
                method.releaseConnection();
            }
        }
    }

And can invoke it with something like this:


 IdResponse response = actOnHttpMethod(post,new MethodOperator<IdResponse>() {
    public IdResponse actOnMethodPostResponse(HttpMethod method) {
        // unmarshal the response and create an IdResponse object with it
     }
 });

The actOnHttpMethod will do the request, hand my object the response, and then close the connection for me.

Inner classes definitely suffer from readability problems, but this setup ensures that it's very easy for developers to not even think about connection management. Furthermore, I can add features and have them automatically used by every client. For instance, if I wanted to profile the request/response time or add logging. If I ever want to add support for asynchronous calls, I can write a new utility method that does all the work of enqueuing the method and so forth, invoking the MethodOperator code as needed, and then change specific code to say actOnHttpMethodAsync or something instead of actOnHttpMethod. A minimal change in client code plus a utility method, and I've added a more scalable alternative in situations where I don't care about waiting for the response.

Once I refactored all that away, I then realized that I could refactor even more. At the moment, I handle a response in one of two ways: I either ignore it (for things like DELETEs) or I unmarshal the contents from XML into Java. Once I had my whole framework in place, I realized I could make implementations of the MethodOperator interface that would cover these two cases.

I created the following:


public class IgnoreResponseMethodOperator implements MethodOperator<Object> {
    public Object actOnMethodPostResponse(HttpMethod method) {
        return null;
    }
}

public class XmlUnmarshallingMethodOperator<T> implements MethodOperator {
    public T actOnMethodPostResponse(HttpMethod method) throws Exception {
        JAXBContext context = JAXBContext.newInstance(Constants.JAXB_PACKAGES);
        Unmarshaller unmarshaller = context.createUnmarshaller();
        return (T)unmarshaller.unmarshal(method.getResponseBodyAsStream());
    }
}

Now my client code actually looks like this:


       IdResponse response = actOnHttpMethod(post,new XmlUnmarshallingMethodOperator<IdResponse>());

I still get all the value of the code that manages connections around my code, but now I don't even have to worry about the unreadability of anonymous inner classes.

(You could make the case that this will create a lot of object churn. If it does, I can look at making a thread-safe implementation that will let me re-use the MethodOperator objects. That's very easy to do with the IgnoreResponseMethodOperator, but tougher with the type-safe xml unmarshaller. I imagine I'd have to create instances for each type of object I might get back. Given that there aren't too many, this wouldn't be too bad. But first I'll see if that's actually a problem.)

Protovis And Wine Visualization: California Crush Statistics

2011-09-24T20:05:00.000-07:00

Radio station visualizations are fun and all, but I realized that I should research data visualization by looking at data I actually care about. That way, I can provide context and ask deeper questions about the subject matter at hand.

As an occasional wine writer, data about the wine industry seemed like a good start.

Harvest — "crush" in wine industry jargon — is afoot here in California, and that spurred me to search for data on previous harvests. The National Agricultural Statistics Service publishes a range of interesting data for wine geeks, some of which I've been using for experiments and explorations with Protovis.

The first public one shows harvest statistics over 20 years for the 15 grapes with highest crush numbers in California in 2010. The interactive version gives you a deeper view, with detailed per-year statistics as you mouse over, but here's a static version to give you an idea.

Groovy, eh?

Wine geeks will know many of this visualization's stories well. The California wine industry has grown tremendously over the last 20 years, thanks to increased consumption in the United States. Grape gluts are periodic, but 2005 was a particularly grape-heavy year. Industrial grapes such as French Colombard, Rubired, and Ruby Cabernet are mainstays of the bulk-wine industry led by Gallo. Pinot Noir tonnage surpassed Syrah tonnage in 2008, about 4 years — when vines start producing worthwhile fruit — after Sideways, the movie that told everyone about Pinot Noir. (Though I should note that I prefer the Pinots of Oregon and the Sonoma Coast to those of Santa Barbara, the setting for the movie. But, really, I prefer the Pinots of Burgundy to those from anywhere else.)

But some items in the data surprised me. Merlot, a common Bordeaux variety, went from almost nothing in 1991 to a dominant grape in 2010. Grenache, the popular, fruity darling of the Rhone Rangers, has actually seen lower crush values in the last 20 years. Pinot Gris has gone from a nonexistent grape in California to one of the top 15 in the state in just over a decade. Tonnage of French Colombard has gone down, which makes me wonder how the industrial market is doing overall.

But if you're reading this blog, you're probably more interested in the technical aspects of this data. I used Protovis, and I have repeatedly found that getting a basic visualization up and running with the library is very fast. Getting the fine details right, however, is much slower. It takes a lot of trial and error to get the language to do what you want. I might switch to D3, its successor, for my next projects. It supposedly gives finer control over your visualization.

What I also keep realizing is that visualizing some set of data isn't really an issue. Organizing the data is. I know this isn't news to anyone who works with data, but these projects are good reminders of how much work that can be.

I started with 20 separate spreadsheets from the NASS and wrote a Ruby script to extract out the bits of data I wanted and compile them into a JSON object I could serve to this chart's HTML page. But even then, the page's JavaScript has to do some processing as well to get the data in a format that Protovis can easily work with. The Underscore JavaScript library is a handy tool for doing data transformations.

But I also used that preprocessing to cache certain items such as the pretty-printed numbers, the colors to use for the different areas (which I calculated with the excellent 0to255.com) and other useful items.

Radio Station Playlist Data Visualization, Part 2

2011-09-03T17:01:00.001-07:00

As soon as I did my visualization of 99.7's music selection for a week, I asked the obvious next question: How does 99.7 compare to other "adult contemporary" radio stations?

There's an interactive version that lets you drill down into the graph, but here's a screenshot.

People listen to radio stations for all sorts of reasons, of course, so I don't know that anyone actually cares about this. But it did give me a chance to look at Protovis and compare it to Processing as I learn about data visualization toolkits.

Gathering Data

When I gathered data for my first visualization, I wrote a simple script that grabbed songs from the 99.7 website. I set that up as a cron job on an EC2 instance and let it go.

I did the same thing for the other four radio stations I decided to look at. 97.3 uses the same website tech as 99.7, and KFOG and KBAY share a different website tech, so those got me two stations for the price of one. 101.3 uses yet another system. Once I had my scripts running, I just had to wait until I had the same week's worth of data from all stations. A bit of cleanup on the data, a quick change to JSON from comma-separated values, and I was ready to go.

I decided to use the concept of small multiples to provide a quick comparison between stations, but then showing an enlarged version for deeper exploration. Each small graph in the chart represents one station across the same span of time.

Protovis Vs. Processing

It took me some time to learn Protovis. I feel that only now, after finishing one visualization, do I really have a grasp on how it works. It seeks to be a declarative language, which means that you define the result and let the under-the-hood bits figure out how to get you there, but I found myself struggling against the lack of control.

Processing gives you that control. You have vast amounts of control, but that's because it starts you with a blank slate. You can probably do anything you want, but the flip side is that you have to do everything you want.

But Processing comes with a strong disadvantage: It creates Java applets. Remember those? I barely do, and I was actually writing Java when that's all people did with it. An applet takes a long time to load in a world where website visitors are accustomed to instant gratification from your page. An applet also won't work on your iOS device. So my first visualization was completely unusable by iPad owners.

(Yes, there is Processing.js, but my attempts to use it only frustrated me. It didn't support Java generics, and even when I removed them from my code, it failed with cryptic errors that were impossible to debug.)

As with so many things, deciding on a visualization toolkit means figuring out what's best for your job. If you're doing something complex and custom, you'll probably want Processing. But for a lot of web-based visualizations, I think Protovis will give you what you need once you figure out how to use it. It can certainly do a lot in that space.

I have still more visualizations in mind for this same set of data, and I'm planning on starting with Protovis (or its successor, d3). The Java applet problems are too big.

Hiring Online Engineers

2011-08-24T09:15:00.000-07:00

(By the way, Maxis is hiring).

I've interviewed a lot of candidates over the years. At my last job, we seemed to have an unending stream of candidates because we had learned to be picky. And I didn't even do the phone screens that kept the numbers down. But I did do what we affectionately called the "make them cry" part of the interview, drilling down on Java knowledge as much as I could.

At Maxis, we have less candidate churn, but I now have to sift through the resumés and do the phone screens. And I'd like to pass along some advice about getting your resumé to the top of my pile. None of these are absolutes, but the more flaws in your resumé, the greater your strengths need to be.

First, have someone proofread your resumé. I don't care if English is your first language or not; get a good editor to give you feedback. I'll wince at one typo but let it slide. Much more than that, and I begin to wonder if your code will be as unprofessional as your text. A recent candidate said in the header of his resumé that he's a "self-mortification" person. I'm guessing he meant self-motivated, but clearly not self-motivated enough to have someone sanity check his text.

In a similar vein, make sure you use the correct terms when discussing things you know. You do not "program in" AJAX, HTML, XML, or CSS. You may understand them, but it doesn't look like you do if you call them programming languages.

Next, keep your resumé brief. I've always liked the saying, "the only thing on the second page of a resume should be the Nobel Prize you won," but I am in the minority. The style du jour is to make your resume as long as possible.

I'll grudgingly accept a two-page resumé even without the Nobel Prize, but a recent applicant I saw had a many-page resumé with this line item: "Coded Perl functions, invoking subroutines and functions calling functions." In other words, you did some programming? And that was simply the most ridiculous in a long list. "Set up Object-Relational Mapping with Hibernate" is another common line item. For those who don't know, this involves writing a config file. If you have items such as these on your resumé, you're padding, and I'll think you need to pad because you don't have any real skills.

If you're a mid-level Java engineer with some enterprise software experience, I'm afraid your resumé looks like approximately 100,000 others. Spring, Hibernate, JUnit, Struts, MySQL, Oracle. I'm yawning already. So make your projects sound interesting. We don't all get to work on famous video games, but before I did, I worked on projects you've never heard of. And I made them sound neat. I focused on the compelling problems and described those. If you can't find interesting problems in your work, you're not a programmer I want to work with: Every problem is interesting in its own way.

I also look for personal projects on a resumé. Yes, people have families and whatnot. But think of great writers. They don't just write because they are required to: They write because they need to.

If you're working on personal projects, no matter how esoteric, you're telling me that you're so in love with programming that you can't simply kick it aside when you clock out. You're telling me you love the craft, the problem-solving, the tinkering. You're telling me that you're a programmer I want to work with.

Visualizing My Bike Rides

2011-08-07T08:44:00.000-07:00

My three main interests are programming, writing, and food and wine.

You'll notice exercise isn't on that list. In fact, quite the opposite: My primary interests are all anti-exercise. But I grudgingly acknowledge its value, and I've decided to "trick" myself into exercising more by turning it into a programming project.

I live close enough to work that I can commute on my bike, and so I've started a project to gather data on my rides and do interesting things with that data, inspired in part by Cooper Smith's visualization of Nike+ data from New York City. Since I need a lot of data to make useful visualizations, I'm riding more consistently to get it. And hopefully by the time I have enough data, the bike riding will seem routine.

I picked up a couple of bike tracking apps for the iPhone, but have settled on Abvio's Cyclemeter, based on the recommendation of a co-worker who is both a data geek like me and an avid cyclist. You press Start on your phone, ride your bike, and press Stop when you're done. It gives graphs, maps, and all sorts of other goodies.

Getting to the data is then just a minor step: All of these apps seems to support exports in KML and GPX. Since these are actually just my rides, that data isn't all that interesting by itself. I know how I get to work.

But with that data, I can create meta-analyses. For instance, how does my speed look across a given ride? Here's a ride I took from the Saturday Berkeley farmers market to Berkeley Bowl, our preferred grocery store.

Green lines indicate places where I was faster than my average speed for that ride. Red lines indicate places where I was slower. I add in the Start and Stop pins, and also provide meta information about the ride off of the extra data in Cyclemeter's GPX file: total distance, average speed, and so forth.

Ruby made this work pretty straightforward. I use Nokogiri to parse a GPX file and calculate the velocity between subsequent points. Each velocity item has the coordinates and timestamps of the two points as well as the calculated velocity. I then use an ERB template for the KML I want to create. That ERB template sets up the styles and other items, and then uses the state variables to construct the line segments, the start/stop pins, and other items.

Eventually, I want to add arrows indicating the direction (which is more useful when you're looking at lots of overlapping routes), pins for the slowest point and the fastest point, and other items.

That will do for individual routes, but I also plan to start aggregating my rides to show even more data.

Visualizing A Week Of 99.7

2011-07-24T14:18:00.001-07:00

When we're driving, Melissa and I often listen to Bay Area radio station 99.7 FM. It specializes in dance-focused pop, club, and hip-hop songs. In other words: boppy, brainless music.

But at any given point in time, it feels like they just replay the hits du jour. How much diversity do they really have? I wanted to know.

The answer? Not much. A mere nine songs made up half the station's rotation in the week I measured. During that week, about half the songs you would have heard would have been one of those nine songs.

I also made an interactive version that requires a Java-enabled browser. It shows the song titles as you hover over them.

Here are the top 9 songs:

Song Title	Artist	Times Played
The Edge of Glory	Lady Gaga	100
I Wanna Go	Britney Spears	75
Till The World Ends	Britney Spears	69
How To Love	Lil Wayne	59
Stereo Love	Gym Class Heroes	54
Rolling In The Deep	Adele	53
Written In The Stars	Tinie Tempah	52
The Lazy Song	Bruno Mars	46
Cheers (Drink to That)	Rihanna	39

To gather the data for this chart, I used the station's published playlist, which lists 25 songs at a time. I set up a micro-instance on Amazon.com's EC2 to run a Ruby script every 45 minutes that fetched that playlist page, extracted the information I wanted (via regexes and the excellent Nokogiri library) and appended it to a file.

Then I set up a node.js server* that returned a cleaned-up version of the raw playlist data I amassed. It removed duplicates caused not only by my data fetching script, whose 45-minute interval meant that there was always some degree of overlap, but also by a peculiarity in the data. Remixes on the site get two or more entries with the same timestamp, one for the original song and one for the remix title, and I collapsed those down into the original song. A remix of Britney Spears' "Till the World Ends" might be different in some ways than the original, but to me it counts as playing the same song. I've published the final dataset I used for this chart.

Along the way, I discovered two lacunae in the website's playlist — there are probably more — which affect the numbers a bit. As Nathan Yau says in his new book Visualize This, "Just because it’s data doesn’t make it fact."

Katy Perry's "Last Friday Night" and Pitbull's "Give Me Everything" got plenty of rotation on the station during this week but never showed up in the published playlist. I checked my raw data, the cleaned-up form, and did spot checks on the site whenever I heard one of those songs. They're just skipped. I assume this is some discrepancy in the database, since other songs from the same Katy Perry album show up in the list.

I don't know that it would change the numbers very much. There might be a more gradual drop-off from "Edge of Glory" to "I Wanna Go," but, if anything, the halfway mark would be closer in. The gap doesn't change the premise: 99.7 replays a lot of the same music.

For visualizing the data, I put it into a couple of tools -- a custom tool I'm writing as well as R -- but ultimately decided on Processing, the big gun in any data visualizer's arsenal. Processing is a full programming language aimed specifically at making digital images, with an emphasis on visualization. I could both fully churn through and munge the semi-raw data and quickly visualize it, all with the same tool. And since Processing is basically Java with some handy utility methods, I'm already very comfortable with the language.

Inspired by Yau's book, which encourages a storytelling mindset, I decided to add visual cues and callouts for "points of interest" to my graphic: the most popular song, the cutoff line for the songs that made up fifty percent of the total, and the cutoff line for the the songs that made up eighty percent of the total.

Because Processing is a programming language, I drove everything off the data itself. While I obviously had to program the callouts I wanted on the chart, I don't have a line that says, "Draw 'Edge of Glory, Lady Gaga' at these coordinates." Instead I have a line that says, "draw the name of the song that got played the most next to the leftmost bar." I used the same mindset for all the callouts. Change the dataset, and the callouts change with it.

Once you have all this shiny, pretty data, you start looking at other ways to explore it. For instance, what's the average number of songs played in each hour? A little bit of modification to my Processing program, and I had a new chart ready to go from the same data.

Then Melissa and I wondered if Bruno Mars' "The Lazy Song" gets played a lot more on weekends, since it's, well, about being lazy and deciding to do nothing with your day. Not really. As a rule, expect to hear it six to eight times a day at the moment, and not more on weekends.

I have more ideas for this, but they're going to take a bit more data collection, so stay tuned for more in the coming weeks.

*There was no need to use node.js here. It just gave me a chance to play with it for something deeper than the "Hello, World!" example.

More With Google's WebGL Globe: Legen - Wait For It - Dary

2011-05-09T22:42:00.000-07:00

In my last post, I talked about Google's WebGL data visualization globe and briefly mentioned the "legend" format for the data array.

I finally got a chance to do something with it, and it is both simple and powerful. If you're in legend mode, your big data array is four pieces of data at a time, not three. The first two pieces are latitude and longitude. The third is the magnitude of the line that will be drawn (divided by 200 to compensate for their multiplying by 200). So what is the fourth value?

Anything you want. When you specify legend mode, the color function you pass in to the globe's constructor gets that fourth value for each point on the globe. That in turn allows you to define the color of the line based on something other than magnitude (the default).

What that really means is that you get an extra dimension in your data. Height is always height, but legend mode allows you to add non-height information based on the color of the line.

Google's initial example — search language by volume — is a great example. The color of the line comes from the dominant language of the area. Looking at their globe, you can immediately figure out where the high search volume comes from (big cities, mainly) just by looking for the tall lines. You can also see which languages dominate search in a given area. English (blue) covers the United States and the United Kingdom. French (light green) covers France and also Quebec. Portugese (dark green) dominates Portugal and Brazil, and also Madeira. Likewise, the Canary Islands are yellow, because they're a part of Spain.

For a new visualization at work I did with the globe (as usual, NDAs make me cautious about giving more exact details), people at my studio wanted to see the height of the line represent the number of events at a given location. But they wanted the color to represent whether the average data from that spot represented a "good" or "bad" user experience. Little red lines would be unfortunate, but might not get flagged as high priority. Big red lines would be a problem, because it would mean that a lot of players were having a bad experience. Fortunately, there weren't any of those, but there were some long yellow lines, which suggests an area where we could improve the player's interaction with our game.

To get that view, I set the globe to legend mode and set the magnitude field of each point to the sample size. Then I set the fourth data point to a very heterogeneous number that represented the user experience. My color function looks at that number and puts it into a bucket. Good returns green, bad returns red, and a middle ground returns yellow.

That means each line on my globe gives three pieces of information: location on the planet, sample size from that location, and quality of user experience from that location. It's easy to spin the globe and look for red hot spots. It's also just a pleasure to play with the visual, even though you can find the worst spots pretty quickly.

One person saw my work and suggested adding a calculation that would make the line more or less red, for instance, depending on how far over the bad threshold it went. Bright red would mean a really bad experience; dim red would be right at the line. I guess that would give us three and a half data points. Before, we knew it was bad. Now, we'll know how bad it is.

Legend mode makes this all possible.

Working With Google's WebGL Globe

2011-05-05T19:44:00.000-07:00

Today, Google released a data visualization that shows search volume by language across the globe. It uses WebGL, so you'll need a recent version of Chrome and decent video drivers to see it.

It's different than the normal "things on a globe" visualizations you see in — for instance — Google Earth, because it incorporates height as an additional dimension. Google Maps and Google Earth give flat perspectives: You can only guess magnitude based on clusters of the familiar red, upside-down teardrops.

By itself, the visualization would be a five-minute folderol to enjoy on a lunch break. But Google's Data Arts team released all the source. Take your own data, muck with it a bit, and you too can have an interactive globe dripping in an oh-so-modern, shades-of-black aesthetic.

I took the bait.

Over the course of today, I took some of our game analytics data and built a local WebGL globe visualization. We have an established workflow for creating latitude/longitude tables from our data — this isn't the first map visualization I've done — and I built on top of that. Once I extracted the information I wanted, I wrote a quick Ruby script that converts the exported data into the format their code looks for.

I can't share a link with you — it's on our internal network — and I can't tell you what I mapped. But I can tell you that it got lots of oohs and aahs in the office.

I can also tell you that it wasn't a simple "replace their data with our data" exercise. If you're planning on doing something with the tool, here are a few things I figured out by trial and error.

Note: This has since been fixed. Don't rely on their README: It's wrong. At least, the JSON format is. It's much better to read the globe.js file to see what it wants, though that requires you to know JavaScript. Rather than a complex set of nested arrays, the code prefers one long array with latitude, longitude, and magnitude strung together like beads on a string. (There's an optional "legend" mode that requires a fourth point, but I haven't played with it. I assume it lets you define different data series. Their addData method takes the data block and a set of options, and one of those options is format: The default is magnitude — three data points — and you can specify legend — four data points.)
A portion of their code multiplies the magnitude by 200 to ensure the small numbers in their data — percentages, I suppose — become big enough for bars that reach high into the sky. For our particular data set, I had to divide the number by 200 in order to get the bars to be correct once they multiply it by 200.
The default coloring of the lines relates to the height, but it looks like you can pass a custom "get the color" function for different logic.
Their source code relies on a top-level directory on your website named /globe for the location of the map image that wraps onto the sphere. You can change that easily enough. Search for "imgDir" in the source.

One-Button Loadtesting With EC2

2011-01-16T09:06:00.000-08:00

I recently realized a minor ambition.

As I've said before, EA's loadtesting group doesn't meet all my needs: They come in near the end of a project and test the nearly-finished system, which means that it's a big headache all around. Fixing performance issues means rummaging through months — or even years — of code to try and suss out issues they're seeing. Also, the mere act of getting them up and running is usually a big disruption for one or more members of your team.

I had loadtesting scripts running already, but they suffered in a few ways. JMeter is good at what it does, but I want easier programmatic control of the scripts (especially for measuring things it's not so good at, such as the lag time on asynchronous processes under load). Also, I didn't have a loadtesting environment. I was running the scripts on a dev server against that same server. I could have put together an environment of multiple servers, but that would incur expense to the studio for machines that were largely idle.

Then I read an article somewhere about a company that uses EC2 as a dynamic loadtest environment. Yes, I thought, that's what I want.

Off and on over the last few weeks, I put together a Ruby script that can, with a single command, set up a full environment in Amazon's cloud-based servers. That one command instantiates an RDS database, an EC2 server that will act as our application server, and another EC2 server that acts as an instance for running The Grinder, a loadtesting tool that another team in the studio is using and that I liked for its extensibility. That one command also builds a custom version of our war file (to handle all the dynamic addresses you get from EC2/RDS), sets up access permissions between all the servers, and runs checks to make sure the whole environment is ready to go.

A second command fires off the loadtest itself, running the test script through The Grinder and collecting the results back to the local machine.

Finally, a third command shuts the whole system down. I do have one command that runs all three steps, but at the moment I tend to run each one manually.

Each run takes less than an hour for now, so the cost to the studio is on the order of $.28 per loadtesting run. Not something you'd want to run continually, but certainly something you could run once a day. As we move along in development, we'll probably need to spend more to set up bigger, more realistic environments, but that will also be when the studio has more budget for my project. The environments we need will scale up alongside the money we have.

One of the key pieces that I set up was the concept of a profile. A config file lets you specify which script to run and how many of each item to set up. So, for my first profile, I just set up one each of the different servers. But you could imagine some profile later down the road that sets up 2 of each machine with some sort of autoscaling system. And you could imagine one much further down the road that sets up something approximating our production environment and runs tests against it. All that is mostly supported.

The advantages to this system are huge. First, it makes real loadtesting feasible even early on in our project. But it also empowers developers to do rapid iteration on performance fixes without having to push them live and see if they make a difference. See a problem in the results, figure out a fix, run the script again to test. Repeat and then release when you can prove that your fix makes a difference. Because each script fires up its own environment, I'll be able to distribute performance fix work to my team.

This ad hoc loadtesting tool is already proving its worth, and I've only just started employing it. I can't wait to see how effective it is going forward.

Dynamic Feature Disabling

2010-11-23T07:01:00.000-08:00

Often, when I'm rolling out a new, substantial feature, I add a config file property that marks it as enabled or disabled. This lets the team shut it down quickly if it's causing problems. It's a good practice, overall.

Keeping the status in a config file is easy to maintain, but it's problematic. Being able to "switch off" a feature in a config file really means, for most Java deployments, rebuilding and redeploying the war file, which can take a while if you're including a bunch of big libraries.

Recently, inspired by two pieces I read about how disabling features helps scale a site and mitigate risk, I came up with an enhancement to my older idea, and it's one of those brain-dead-obvious changes that I wish I had done before: A feature's enabled or disabled status now lives in the database.

I have a simple table (with Hibernate object and service) that stores a feature name, its current enabled status, and the date on which that status changed. Any other service or controller in the system can query that service to see if a given feature is enabled. (Since I run in the Spring framework, access to the service is a simple matter.)

What that means practically is that I can disable an entire subsystem across all instances of my application simply by updating a database row. (In the live system, these will probably be cached and refreshed every few minutes so that incoming requests aren't slowed down.) It also means I can get the status of the systems across an entire bank of servers with a simple query.

I put it in as groundwork for one feature — capturing real incoming traffic to a site to then play back as loadtesting scripts — but quickly back-ported a few other systems that have single points of entry. Our profiling system, for instance, can now be turned off application-wide simply by setting the enabled status of "Profiling" to false.

Of course, once you have subsystems that have a dynamic mechanism for checking their enabled/disabled status, it's a small step to enabling/disabling on a per-user basis. Which means we can gradually roll out new features, checking the load on the system at each pass and fixing bugs.

It's not an earth-shattering idea, but I was immediately entranced by the power it gives to my system.

Integration Tests

2010-10-02T10:05:00.000-07:00

I've been a fan of unit tests for a couple of years now. Once I buckled down and wrote some, I realized how powerful they could be. They give you more stability out of the gate and they provide good regression tests. They allow me, the developer, to keep moving forward in tasks and minimize the time I spend going backward to debug or test older functions.

But I've always held the stance that unit tests shouldn't test things such as interactions with Hibernate or Spring. Those are well-tested frameworks with strong community support. Writing unit tests that touch those layers always struck me as a waste of time.

That said, most server-side Java code ends up moving into those layers. Virtually everything I write ends up talking to a database. And the server code itself is a loosely knit tapestry of services that chat with each other. That code can certainly have bugs — incorrect queries, edge conditions, bad assumptions, whatever. So how do I get tests against it?

Integration tests. Like unit tests, integration tests demonstrate that function A returns result X given inputs Y and Z. The difference is that in an integration test, you are testing the interaction between systems versus the simple in and out of a self-contained function.

I finally decided that I wasn't getting the test coverage I wanted with just unit tests. I was finding subtle bugs tucked away in database calls and service-to-service communication. Mock objects — code that presents the expected interface to a layer without providing the full functionality — only get you part of the way. If your mock object is keeping information in a local map instead of hitting the database, you're not testing that the query to the database returns the right thing.

Once I decided to incorporate integration tests, however, I went down a bit of rabbit hole. Your services need to talk to each other, so you need to wire them up to each other. But in the real, running code, I use Spring to manage all that. Fortunately, I can use Spring in the test environment, too. And Spring allows me to have a supplemental bean configuration file that overrides the production-code config file. So, for instance, I can have a bean named "telemetryService" that overrides the bean of the same name in the main config file. The test version doesn't actually send telemetry information. It effectively becomes a mock object. (Though in that particular case, it's done with a simple boolean.) I have an S3 service layer in the test config file that points to a test S3 bucket instead of our development one. Any beans that aren't overridden are pulled from the main file.

My integration tests do have to call a method to set up that configuration, however. Since the Spring config doesn't change at runtime, I have a static utility method that checks to see if the configuration is set up and, if it isn't, sets it up. This violates the principle that a test should not rely on existing state from another test, but that configuration load is time consuming.

But services don't just talk to each other. They also talk to the database. You don't want them to point to a database you actually use, so you need a test database. And that test database needs to a) be kept up to date, b) have some test data, and c) not let data linger (to ensure that no test ends up passing because of the coincidence of some other test running before it).

For the last two requirements, I incorporated DBUnit. DBUnit can do unit testing against the database, but I don't really need its form of that. Because I'm using Spring to set up my app in test mode, all my services can continue to use Hibernate for database work. (They can also use my JRuby-based query files.) But DBUnit offers two key services: It can take a data file of test data and load the database with it, and it can wipe any desired tables clean between test runs. When a test suite starts, it calls a method that ensures the Spring configuration is read (if it hasn't already been) and wipes/reloads the test database. When a test suite starts, I know that there is predictable data in the database and that that's the only data in the database.

What about between tests, though? I could do a wipe and reload around every test, but that's time-consuming. Instead, I have a setup method that configures a transaction and a teardown method that rolls back that transaction. All of this means that developers have to remember to do all this when writing an integration test, which I dislike, but I hope to integrate some tools that will automate that.

My test database also needs to be current with the latest schema. For that, I incorporated Autopatch, which will automatically get your database up to date when a code base runs against it.

So far, so good. But integration tests take a while to run, and I worry that developers might, faced with progressively longer test runs, disable tests on local builds. To keep our tests spry, I separate integration tests so that they're only run if you tell Buildr you're in the "longtests" environment. (That environment also specifies the test database properties.) The build system, which runs within minutes of any check-in, always runs them, however, so even a developer who gets hasty has an automatic monitoring.

All of this took three solid days to get working properly, but now that's it running, I got exactly what I wanted: the ability to have much deeper test coverage. The other day, I wrote a test that called service methods that altered the database and moved files around on S3. At each step in the chain, I could check the state of each component and verify that it had worked succesfully. And I knew that it was executing real queries and running real code. No mock objects.

The ability to have deeper test coverage means I've set myself a goal of writing those tests. Every feature I work on now has tests against it, even if they require database work or multiple services. And when I find a bug in my existing code, I force myself to write a test to reproduce it and then work on the code until the test passes. That way I get free regression testing whenever a check-in happens.

I have noticed one negative aspect, however. Unit tests, by their very nature, force you to write smaller and smaller methods that are easier to test. With no limit on the amount of activity you can do in an integration test, however, I find that my methods aren't channeled into smaller pieces. I try and do that because I know that ultimately produces more maintainable code, but my integration test system doesn't give an almost automatic push the way unit tests do.

Big methods or not, I still get to run a complete suite of extensive, realistic tests with a single command. That's pretty powerful.

Config Files As Programs

2010-09-28T22:25:00.000-07:00

While working on a few recent tasks, I've hit upon a small but powerful technique: My configuration files are actually programs in their own right.

Expression languages of some form are de rigueur in modern frameworks. Especially in server code, where you need to deploy the same code base to different servers and have them point to different databases, mail servers, or whatever. In Spring, for instance, you can write something like ${project.database.url} and it will assume you mean to use not the literal string but the value of the project.database.url property, which is probably different in your dev environment and your production environment.

But my technique is vastly more powerful. I'm sure others have thought of it, too, but it sort of snuck up on me.

The first time I used this trick was when I put my work's SQL into Ruby scripts. I did that for readability's sake, but as I added more queries, I began to do the things one does does with programs. I refactored two similar queries into a method that took a differentiating argument and returned the query. I declared constants to hold common values. I started to ponder a Ruby module that I could import for various utility functions. The end result from the Java code's view was a set of query definitions. But within the query definition itself, I had normal programming tools at my disposal.

On a personal project (which I hope to write about soon), I decided to use YAML for the config files. I've gotten used to the format for config files, thanks to my work with AppEngine, and Ruby can easily work with it. My original idea for the config file was just fixed dates and times when I wanted my script to wake up and record an Internet radio stream. But then I realized that I wanted to record some streams on every weekday or on the first Sunday of the month.

So I thought, what if, before I feed the file to the YAML parser, I first run it through ERB, Ruby's powerful templating system? As long as the YAML parser sees something that looks like YAML, it doesn't matter how it gets there, right?

Right. I defined a method in my config file — in my config file! — that would return the next weekday after the given date. Then, when defining the value for a field, I called that method to calculate the value. My "YAML" file looks something like: start_date: <%next_weekday.strftime('%Y-%m-%d')%> 18:00. Which won't give you anything useful from the YAML parser. But run it through the ERB parser, and the YAML parser sees start_date: 2010-09-28 18:00, which is perfectly valid YAML syntax and is what you intend for the value to be.

Words With Friends Bingos, Redux

2010-09-03T09:54:00.000-07:00

In my last post, I described writing a program to find the five-letter combinations most likely to yield a bingo in Words With Friends.

Then I realized there was a subtle bug.

Consider the word abreast. When you look at how Ruby's combination method splits it up, you'll see two entries equivalent to STARE, because there are two a's. This is correct from a programming view, but from a player's perspective, it doesn't matter. Having STARE is sufficient to get to abreast. You don't view the a's as different.

Here is the new word list with the corrected code. TEARS and TIERS are still your best bet, but the list has shifted a bit and the numbers are different.


aerst 1295
eirst 1237
aeirs 988
aeist 984
einst 983
einrs 972
eorst 949
aelrs 940
eerst 920
aeins 854
aeirt 816
aenrs 805
aenst 799
eeirs 792
aelst 788
eilst 782
aeint 765
aeils 765
ainst 751
eilrs 748

And here's the corrected script.


# find the 5-letter combos most represented in 7- or 8-letter words

wordlist = ARGV[0]

combos_to_count = Hash.new(1)

File.open(wordlist,"r") do |file|
   line = file.gets
   while(line)
      line = file.gets      
      if line then
          line.chomp!
          next if line.length != 7 && line.length != 8
          chars = line.split(//)
          unique_combos = []
          chars.combination(5) do |combo| 
              combo.sort! {|a,b| a <=> b}
              unique_combos << combo.join
          end
          unique_combos.uniq!
          unique_combos.each {|combo| combos_to_count[combo] = combos_to_count[combo] + 1}
      end
   end
end

sorted_combos = combos_to_count.sort {|a,b| b[1] <=> a[1]}
(0...20).each {|num| puts sorted_combos[num][0] + " " + sorted_combos[num][1].to_s}

Optimize For Bingos In Words With Friends

2010-09-03T07:04:00.000-07:00

See the updated version here.

Somewhere in my library of books is one called Everything Scrabble, a guide to improving your Scrabble game. I read through it a few years back and have probably forgotten half its contents, but I do remember the author's comment that highly ranked Scrabble players often swap tiles in an effort to increase their chances of hitting a bingo (a word that uses all seven letters on your tray).

I've been playing Words With Friends (screen name: linechef) and have been thinking about that Scrabble strategy. Bingoing is less of an advantage in WWF. They're only worth 35 points instead of 50, and they often open up large swaths of bonus tiles for your opponent to gobble up. But they are usually worth it if you can hit a bonus tile yourself.

Everything Scrabble provides the scoop on which five tiles you should keep in your tray to maximize bingoing, but my mind let that knowledge go into the abyss. So I decided to write a program to rediscover it.

You can imagine how the program works: Go through every seven- and eight-letter word in the Words With Friends dictionary, the ENABLE wordlist, and find every unique combination of five letters in it. Keep track of how many copies of that combination you've seen throughout the dictionary, and then sort.

Let's start with the answers first. Here are the top 20 combinations, along with the number of times they occurred (the ENABLE list has 172,823 words):


aerst 2413
eirst 2199
einst 1780
eorst 1698
eerst 1646
einrs 1596
aeist 1587
aeirs 1484
aelrs 1468
aelst 1350
aenst 1346
eilst 1338
eiprs 1286
aeirt 1273
aeprs 1266
aenrs 1258
aeins 1245
einrt 1238
deirs 1233
ainst 1232

And here's the Ruby code. Ruby 1.8.7 added a combination method to the Array class, which makes this program tiny. You tell it how big you want each subset to be, and you pass it a block which takes each subarray as an argument.


wordlist = ARGV[0]

combos_to_count = Hash.new(1)

File.open(wordlist,"r") do |file|
   line = file.gets
   while(line)
      line = file.gets      
      if line then
          line.chomp!
          next if line.length != 7 && line.length != 8
          chars = line.split(//)
          chars.combination(5) do |combo|
             combo.sort! {|a,b| a <=> b}
             combo_str = combo.join
             combos_to_count[combo_str] = combos_to_count[combo_str] + 1
          end
      end
   end
end

sorted_combos = combos_to_count.sort {|a,b| b[1] <=> a[1]}
(0...20).each {|num| puts sorted_combos[num][0] + " " + sorted_combos[num][1].to_s}

Once you have the code, you can start asking other questions. Is the list different for words with exactly seven letters? A little bit:


aerst 609
eirst 493
eorst 450
aelrs 413
eerst 404
einst 385
aeprs 379
einrs 348
aelst 346
eilst 335
acers 331
aeirs 330
aenst 327
aeist 322
deirs 320
aders 318
eiprs 317
eilrs 297
aerss 296
deers 295

Note that there are only 676 permutations of the other two letters you could fit in your tray, so with STARE you can bingo with almost any combination. I leave learning those 609 words as an exercise for the reader.

What about a different dictionary? Here's the same run I first did but with the Scrabble Official Dictionary, Third Edition.


aerst 2395
eirst 2182
einst 1759
eorst 1670
eerst 1628
einrs 1581
aeist 1562
aeirs 1473
aelrs 1458
aelst 1335
aenst 1327
eilst 1317
eiprs 1278
aeprs 1258
aeirt 1257
aenrs 1239
aeins 1232
ainst 1224
einrt 1219
deirs 1218

Of course, some of this is common sense: EST gives you the superlative of many words. Likewise, ER gives you the comparative along with the prefix RE and the "person who does" suffix. Those four letters plus vowels other than U and Y form top contenders in many of the runs.

If you're playing WWF (or Scrabble), work to keep STARE in your tray, playing or swapping other tiles, until you get a bingo. Placing it on the board, of course, is a different matter.

Rails Scaffolding In Java

2010-08-25T08:25:00.000-07:00

I recently needed to add a new business object to a system I'm developing at work, and it was a bit of a drag.

If you follow any of the normal Java patterns, you probably know what I'm talking about. You need the file for your business object. You need a service class responsible for returning instances or sets of instances from the DAO layer. You need a SQL file that defines the database table the business objects will be stored in. In our case, we also have queries in resource files, so we need a file for that as well. You need to edit your Spring config files to point to the new service. And so on. Your environment probably has some overlap with mine, and probably requires pieces mine doesn't.

You can simplify a lot of this, of course. You can make a service base class with Java Generics that will give you a lot of the type-safe methods you'd want. IDEs will let you set up templates, but you'll still have to click through a few menu options to get you the files you need. And the more files, the more clicks. And I like running an IDE-neutral team, so I wouldn't want to do something specific to one workflow.

I wanted a better way. Specifically, I wanted what Rails provides. You type a command at the command line, and you get all the files you need for working with the object. (XCode offers similar functionality.)

And I thought, "Well, why not?" My build system is written in Ruby, and Ruby's ERB templating system is built in to the language.

It took about an hour to get the main system up and running. I now type buildr :biz_object object=com.ea.foo.TestTest and I get a src/main/java/com/ea/foo directory, a TestTest.java and TestTestService.java file in that same directory, a sql file with the table definition (named test_test), and an empty query file. I also get a perforce changelist (via p4Ruby) with the description filled in and all the new files added.

Here's the heart of the code. template_to_final is a hash of template file name locations to the end file destination. The local variables exposed by the call to binding include the package name, the Java object name, and the SQL-friendly name:


       template_to_final.keys.each do |key|
           File.open(key) do |file|
               b = binding
               erb = ERB.new(file.read)
               outfile = File.new(template_to_final[key],"w")
               puts "Creating #{outfile.path}"
               outfile.write(erb.result(b))
               outfile.close
           end
       end

To give you an idea of what the templates look like, here's some code from the java file templates:


package <%=java_pkg%>;

@Entity
@Table(name="<%=java_sql_name%>")
@SequenceGenerator(name="objIdGenerator",sequenceName="object_id_seq",allocationSize=1)
public class <%=java_obj%> {

I don't yet go the full Rails route and specify all the properties on the command line, but this takes care of getting the boring parts of business object development out of the way, enforcing consistent naming schemes, and ensuring that the developer doesn't forget to check in some file. I also don't yet modify my config files, but that won't be too tough to add.

Scripting Campfire

2010-08-05T07:22:00.000-07:00

My team uses Campfire, a web-based chat tool from 37signals, to communicate throughout the day. We've divided it into various rooms, one of which is a Status room where we do a virtual version of the daily stand-up that many teams do: a quick meeting where everyone says what they're working on. (One of our team members is in England, and our hours vary a bit, so a classic stand-up isn't practical.)

About a month ago, one team member started posting the date in the status room before anyone gave their update. For some reason, Campfire itself doesn't do this reliably, and when you have a room with nothing but status updates, it's not always clear where one day's messages end and another's begin.

Usually one or two people on the team did this, but I followed suit on a couple of occasions, and of course thought about automating it. We already had a bot account for some very early automation, and Campfire has a good web service. Insert Tab A into Slot B.

Here's the relevant Ruby source code, which I have hooked up to a cron job (Note that if you want to script Campfire, I suggest creating a "Bot Test Room" that you can use for experiments without spamming your real rooms):


require 'date'
require 'json'
require 'net/http'
require 'net/https'
require 'uri'

def send_text_to_campfire(text)
    message = {
       :message => {
           :type => "TextMessage",
           :body => text
       }
    }
    send_to_campfire(message)
end

def send_to_campfire(message)
    url = URI.parse("https://<your base URL>/room/<your room number>/speak.json")

    request = Net::HTTP::Post.new(url.path)
    request.basic_auth(<your auth token>,<any password string>)
    request.content_type = 'application/json'
    request.body = message.to_json
    request.content_length = request.body.length

    http = Net::HTTP.new(url.host,url.port)
    http.use_ssl = true
    response = http.start do |http|
        http.request(request)
    end

    puts response.body.to_s
end

# construct message
today = Date::today
formattedDateString = sprintf("%02d/%02d/%4d",today.mon,today.mday,today.year)
dateStatusString = "=== Today is #{Date::DAYNAMES[today.wday]}, #{formattedDateString} ==="

#send status message to campfire
send_text_to_campfire(dateStatusString)

Filling in the date each day isn't a huge time savings: It will take us a lot of days to recoup the time I spent automating a five-second typing task. But once I figured out the gist of posting to Campfire, I started adding new functionality. Our "Today is ..." status message now includes a few choice statistics — gleaned from our telemetry system — about gameplay from the previous day. I also wrote a "canary" script that does a health check on our dev server and posts to our general chat room if it seems to be slow.

Naturally, some of my co-workers have suggested writing an adventure game on top of the API. That may be a bit silly, but it does emphasize a point I often make: You can't really imagine all the possibilities for a technology until you get your hands dirty a bit and play with it.

This whole experience underlines again why web applications should have APIs. The lack of a date is probably a Campfire bug, but we don't have to wait for them to fix it. And we've added functionality that is only relevant for our team, resulting in something that more closely ties in with our real needs.

Watch Your Users Use

2010-08-01T10:57:00.000-07:00

Time and time again, I re-learn this lesson: Nothing makes your system more usable than watching your users use it.

As I mentioned in my last post, I'm building a telemetry system on top of Google AppEngine so that my studio can understand how our games will be played. As I've demoed pieces of it, I've put together web pages and interactive charts and other nice little visualizations. I've made JavaScript libraries to make it easy to build those sorts of things, and I've demoed some of the prettier ones to the stable.

But I also put in a way to export data as tab-delimited fields. I figured this would be a good last resort if the JavaScript version wasn't quite there yet, or if we hadn't written a specialized view on the data, but I figured we'd favor those.

Naturally, then, my principal users have all just used the tab-delimited export and not pushed beyond that. In particular, they grab a bunch of tab-delimited data, import it into Excel, and then do manipulations on it there. Though there are macros and various other things, it still seems to take a senior engineer a couple of hours to compile all the data for his weekly reports.

I know this because I sit with him a fair amount as he explains his process.

The other day, I was thinking of his process and realized that Python can create Excel spreadsheets. So I could remove one minor barrier by just letting him get the data in Excel format instead of tab-delimited text. Easy.

But that doesn't buy him much. For his weekly reports, he trims some of the unneccesary columns (my exports grab all the fields and dump them out, including some that are system-level fields), runs some macros to insert formulas to give him percentages, sorts the data, and then copies and pastes that into an email. He does that for about 10-15 reports, in one form or another.

What if, I thought, I could let him define the format of the spreadsheet the system generates? Then he could say, "give me this data, but put it in this layout." That would shave a big chunk of time from his flow.

One thing about working with him is that he's not an online engineer. This turns out to be very good, because it makes me think in terms of letting him modify config files instead of modifying server code. (A medium-term goal is building a real interface on top of my system so that anyone in the studio can interact with it, but for now, config files are how we do things.) One of our other online engineers used the system by writing a custom request handler that interacted with the AppEngine datastore directly to generate a customized HTML page. Very neat, but if he were my only user, my system wouldn't be very evolved. I'd just say to someone new, "write a request handler."

I mentally sketched out what the config file would need to contain, what abilities it would need to give him, and within 45 minutes had a system in place that will let him generate something much closer to the final spreadsheets he needs for his reports.

A key component of the new feature is the Django templating language that comes with AppEngine. I actually dislike the language for HTML generation, but for letting users construct simple templates, it's pretty good. One important feature: It can render a template contained in a string, not just one contained in a file. And you can give the renderer a context, filling in some variables that are accessible within the template. In my case, I create a context that contains the current piece of aggregation data (which is a summary object of a large number of events) and the current row number.

From his perspective, to generate a report about how much damage a creature is doing in our testing, he creates a YAML config file that looks like this:


header:
    - value: Name
    - value: Avg Damage
    - value: Total Damage
row:
    - value: "{{aggregation.groupByValue}}"
    - value: "{{aggregation.average}}"
    - value: "{{aggregation.sum}}"

Then, when he requests data in Excel format, he can tell it to go through the layout he specifies. Rather than getting a dump of every piece of data in each object, he gets a spreadsheet with just the information he needs. The {{}} is part of the templating system and translates as "spit out the result of this expression."

Let's say he also wanted to capture the number of times that creature did damage. That's actually available to him in another field in the aggregation, but let's say it wasn't. My system also supports formulas. So he could do this:


header:
    - value: Name
    - value: Count of Attacks
    - value: Avg Damage
    - value: Total Damage
row:
    - value: "{{aggregation.groupByValue}}"
    - formula:"ROUND(D{{row_num}} / C{{row_num}},0)"
    - value: "{{aggregation.average}}"
    - value: "{{aggregation.sum}}"

That formula will become, in the output for the first row, ROUND(D1/C1,0).

This is still a long way from user-friendly, but I think it's going to give him back an hour and a half of time each week. And building the system this way means that a user-friendly version just needs to write the format into this config file, and away it will go.

Google AppEngine

2010-07-20T18:01:00.000-07:00

Three months ago or so, I had an interesting idea for Google's AppEngine, an environment for building web applications. I wanted to use it for telemetry.

Telemetry, literally, means measuring something from afar. (Merriam-Webster's definition is somewhat circular if not outright tautological.) In the gaming industry and perhaps others, it means capturing information about how users interact with your system. Think of stats packages for web pages; they're the same thing. You also hear the terms business intelligence and plain old data analysis.

I want to be clear: This isn't about watching you, in particular, play a game. If you die in a particular spot on a particular level when our newly announced Darkspore ships, I don't really care. But if lots of people die in that same spot, it tells you something about the difficulty of that spot, doesn't it?

I needed something that could take in a large stream of data without hiccuping and allow me to process it and report on it. I went with AppEngine.*

A big selling point of AppEngine is scalability. Google obviously knows something about running big systems, and they take control of scaling your app. At any given point, I don't know how many servers are running my code: AppEngine starts up new instances as traffic grows and shuts them down when traffic slows. It's not like Amazon.com's EC2, where you manually say, "I want this many servers. No! Now I want this many!"

Closely tied to its scalability is the datastore behind AppEngine, which is a proprietary NoSQL technology called BigTable. When you run queries against BigTable, you're using a MapReduce algorithm — the same algorithm that allows Google to index and search the web so quickly. I had a potentially huge set of relatively unconnected data: AppEngine seemed like a good fit.**

And, for the most part, it has been.

But the datastore that makes AppEngine so powerful has some quirks that I've had to work around. Keep in mind that it's not a relational database, so it doesn't have the same behaviors. Most notably, it's not very good at aggregating data: sums, averages, whatever. And remember where I said that telemetry is all about aggregation? In the long run, I'm hoping BigQuery will make this easy, but in the mean time, I had to build my own aggregation system on top of AppEngine's cron jobs and task queues. The system is pretty neat, including support for a "group by" concept so that I can get views on subsets of the data in a given aggregation, but I'll be happy to let someone else handle that heavy lifting.

Another issue I've run into is that AppEngine sets hard caps on certain aspects of the system. For instance, any given request can't run too long or it will be terminated. On my own servers, if a webpage for an admin tool takes thirty seconds to load, I'm not too upset. It's an admin tool; I can wait. Google isn't so forgiving, presumably to prevent some process of yours from running amok on their machines. This has led to some early performance tweaking that's probably, in the long run, better to get out of the way now.

But these have been interesting problems to solve, so I haven't minded them so much as been annoyed by them. We're currently pulling in 100,000 events or so on any given day (as testers play the game and developers work on it), and I've only had to do optimizations on the reporting side and aggregation system. There's a whole other round of work I can do on optimizing writes, but I haven't bothered yet. Granted, 100,000 events isn't that significant, but it's nice to defer some of the deeper tuning until we're a bit closer to shipping. Meanwhile, we've got graphs and charts and spreadsheets with our telemetry data served up in a useful way to different people in the studio.

It's always hard for me to gauge the ease-of-use of a web system for someone who's not a long-time server engineer, but AppEngine feels like it is very good for either the people who know nothing about web stuff or people who know a lot about it. For middle ground types — maybe a bit of PHP or other dynamic site system — who won't be facing scalability issues, I wonder if Rails might be a better choice.

But I do know that one of our non-online engineers was able to install a new index on AppEngine for a query he wanted without even asking me. I didn't know about it until I happened to see his check-in note. That's pretty powerful. AppEngine supports both Python and Java, but I deliberately went the Python route first, despite not knowing the language at all, because there's more Python knowledge in general in my studio. And that language I didn't know? The code I wrote to take in telemetry events is virtually unchanged from the first day I wrote it. So it's easy to get started on AppEngine, and the cost is free well into the needs of most websites.

If you're interested in developing on AppEngine, I recommend the yet-to-be-released Code in the Cloud.The writing in the early drafts is a bit too exuberant, but there's solid information from the get-go, and Pragmatic Programmers delivers polished final products.

* Yes, yes. I don't trust Google, either, though, as always in blanket statements, the AppEngine team seems to have their hearts in the right place. What's to prevent them from pilfering through our data and gleaning their own information about our users? Well, there is a clause about that in the Terms of Service — they won't do it — but I'm not naive. We won't be storing information that anyone else could use to trace back to our users. An ID that makes sense inside EA? Yes, sometimes. Anything that makes any sense anywhere else? Nope. Our legal department was pretty firm on that.

** What if AppEngine goes down on our big launch day? Am I nervous about tying my company's game to a service that may not be there when I need it? In this case, not really. Let's say AppEngine goes down for a full 24 hours. The whole thing, not just pieces. Such an event would be unprecedented in my experience, but still. So we lose a day of telemetry data. Big deal. People will play the game one day the same way they played it the day before and the day after. Since I'm dealing with aggregates, I can cheerfully ignore missing data.

What's The Probability Of Two Boys?

2010-05-26T22:00:00.000-07:00

There is a link going around about the most recent Gathering For Gardner. The writer hooks the reader with an intro featuring Gary Foshee (a top-notch designer of "secret box" puzzles, though they're rarely boxes), who poses this question to the crowd of mathematicians, puzzlers, and magicians: "I have two children. One is a boy born on a Tuesday. What is the probability I have two boys?"

The article then describes the Gathering. (I'm on the invite list, but I've not yet been.) It eventually explains the answer, but these probability questions never make sense. So I wrote a program to illustrate the first oddity, which is that announcing you have two children and one is a boy makes the probability of the other one being a boy only 1/3. Basically, create 10,000 sets, and remove any that are two girls. Now count up the total number of pairs left and the total number of those that are two boys, and you end up with something around one-third.



num_bbs = 0
total_pairs = 0

(0...10000).each do |count|
  children = [Kernel.rand(2),Kernel.rand(2)]
  next if children[0] == 1 && children[1] == 1

  total_pairs = total_pairs + 1
  num_bbs = num_bbs + 1 if children[0] == 0 && children[1] == 0
end

puts "#{num_bbs} pairs of boys out of #{total_pairs} valid pairs = #{(num_bbs.to_f/total_pairs.to_f) * 100}"

It's weird, but it's true.

We Rule: Maximize Your Time

2010-05-18T21:37:00.000-07:00

I've been playing We Rule on my iPhone. I have to say, I don't really get the point of it. It seems like the only way to progress is to plant and harvest crops and exchange services with other players. But there are tons of things like trees and banners and what not that don't seem to do anything. Should I place a water tower? Who knows?

But this is not a game review. When you plant items and harvest them, you get paid some amount of gold depending on the crop. Planting each crop takes some defined amount of money (usually), and each crop requires some defined amount of real-world time to mature.

So, naturally, I wondered: Which plants are the best investment?

A simple spreadsheet offers the answer*. I wrote down the profit of the crop and divided that by the number of minutes it would take to grow. I did the same thing for experience points, which allow you to level up. (This list reflects the crops I currently have access to. I'll update as I go.)

Crop	Profit	Profit/Minute	XP/Minute
Magic Asparagus	1250	.87	.24
Potatoes	200	.56	.22
Pumpkins	160	.89	.36
Strawberries	120	1.33	.56
Carrots	260	.36	.15
Squash	180	.6	.25
Beans	340	.24	.10
Onions	100	1.67	.67
Wheat	20	4	1.60
Corn	5	6.67	1.33

Your definition of a good investment may be different than mine. Clearly the best investment is to plant and harvest corn like a madman every 45 seconds. You have fun with that. Wheat is somewhat more tolerable, with a 5-minute harvest time, but I like to set up a bunch of crops and then go do something else until they're done. Or set up a bunch overnight to have ready the next morning.

Strawberries and onions are good investments, with onions beating out strawberries for both cash and experience points. Plus, they only take an hour: That's about right for mid-day play. In the crops that take all day, Magic Asparagus is your best best, with beans not even close. Beans also give you one of the worst XP/Minute ratios. Don't plant beans.

Derive your own fun theories from the list.

*Yes, this is a programming blog. Spreadsheets are programs, too, though they're rarely treated as such.

Rails Rules

2010-05-14T07:41:00.000-07:00

I get now why people love Rails.

I had a personal project in mind: a web-based application for organizing all the notes you collect while working on an article. Like the index cards you used to use for school reports. I decided to try Ruby on Rails, a web application framework built on top of the Ruby language, mostly to see what all the fuss was about.

Rails is the framework of choice for a wide range of startups because it lets you get up and running quickly. It works by driving home a brutal truth: Your website is not that unique.

Some sizable percentage of what you need to do for a real-world website is what everyone needs to do for a real-world website: working with databases, adding CRUD functionality, rendering HTML, mapping handlers to URLs, and so forth. "All of this has happened before and all of it will happen again."

Rails acknowledges this and takes care of most of that functionality out of the box. A lot of what it does seems like magic, but really it just imposes naming conventions that will let it do the right thing most of the time. Put your User objects into a users table, and you rarely have to write SQL. Organize your URLs as /controller/action/id, and you don't have to map anything manually. Call a controller method create, and it will automatically be called to handle POST operations. And you don't even have to remember any of this: The scripts that come with Rails generate tons of the code for you, so adding a new business object into the database and HTML-based CRUD pages to manage said objects take two scripts. Two lines at a command prompt, and poof.

I gave it a whirl and came away impressed. In five hours of flying to the East Coast this weekend, I, who know very little about Rails and something about Ruby, had a functional site for creating, viewing, and editing each of the key object types my application needs. I even added a whole bunch of "it would be nice if it did this" types of features. All on the way to New Hampshire. I wrote minimal amounts of code to do it, too. I have an app that I can use — indeed, I've started using it for real-world stuff.

On the way back, I started worrying about the user interface, and that's where I began to flounder a bit. Rails is smooth sailing if you're using all of its tools, but if you want to use something such as jQuery for your front-end JavaScript/AJAX, things get tougher. Or perhaps it's that the book I used doesn't explain the depths well enough to let me figure it out. A friend of mine says it's easy, and I'm sure it is, but I haven't quite grokked everything I need to do yet.

But even with this hiccup, Rails is clearly a valuable tool that any web developer should consider.

Ruby + Outlook + Perforce

2010-04-13T12:11:00.000-07:00

At my work, we have a practice that I've always found curious. Usually, when people check something in, they send out an email to the whole studio with the change list.

This is, by itself, a fine practice. The curious part to me has always been that Perforce (our version control software) has the ability to send out emails for every check-in. My old boss would subscribe to every check-in in our branch, for instance, so she could keep tabs on what the group was doing. But that's not what we use.

Over time, I've come to understand the rationale of our redundant workflow. People can attach screenshots (I do work for a game company, after all) and the subject line can provide project categorization and a short description that perforce doesn't know about. Also, culturally speaking, if everyone in your company does this and you don't, people have a harder time figuring out what you do.

But I still find the workflow annoying. You submit your changes in Perforce. Then you go to the "submitted changelists" pane and double-click on the changelist you just submitted. You select all the text and then copy it (this view has more info in it — such as the list of files — than the text you wrote when submitting, so you can't just use that). You alt-tab over to Outlook, open a new message, address it to the studio email address, attach your pithy subject, and then copy and paste in the changelist info. (You also use this moment to attach screenshots, if relevant.) You send the email and then alt-tab back over to Perforce (or your IDE) to do more work.

I finally decided to automate the process a bit using Ruby to tie together Outlook and Perforce. I set it up to work with my needs, so it may be of limited use to anyone else. For instance, we have a custom of using "mini" to indicate minor, one-liner types of fixes. Otherwise, I tend to use "submit." Also, sometimes I want to aggregate a few recent changelists in one email. That's what the -count argument does. I also put all the config info (username, password, etc.) into a separate yaml file so that I can distribute the script without sending around my network password. Finally, I specified a sendMode of either Display or Send. Display opens the email for you and lets you customize it. Send just kicks it out the door. The former is useful for screenshots and the like. The sendMode and project config file options provide defaults, but they can be overridden. Sometimes I do work in another functional project and need to change the email accordingly.

You'll need p4Ruby to make this work. (I think the OLE stuff is built in to the Ruby for Windows installation.) Perforce returns information in sort of an odd way: a changelist will have the list of revisions as one field and the list of files as another. The indexes line up, but it takes a bit more work to get the info you want.

There are no doubt better ways to do this: I'm still fumbling around with Ruby.


require 'win32ole'
require 'P4'
require 'yaml'

is_mini = false
subject = ""
num_cls = 1

def assert_value(obj,message_on_fail)
   if !obj
      puts message_on_fail
      Kernel.exit
   end
end

# load the yaml settings first
assert_value((File.exists? 'p4_email.yaml'),'You must have a p4_email.yaml file in the same directory')

config = YAML.load_file 'p4_email.yaml'
sendMode = config['sendMode']
project = config['project']

assert_value(config['p4User'],'No p4User specified!')
assert_value(config['p4Password'], 'No p4Password specified!')
assert_value(config['p4Client'], 'No p4Client specified!')
assert_value(config['p4Host'],'No host specified!')

# now parse ARGs. In particular, see if the user has overridden config settings
ARGV.each_index do |index|
   if ARGV[index] == '-sendMode'
      sendMode = ARGV[index+1]
      # check value
      assert_value(sendMode == 'Display' || sendMode == 'Send',"Invalid sendMode value: #{sendMode}")
   end

   if ARGV[index] == '-project'
      project = ARGV[index+1]
   end

   if ARGV[index] == '-mini'
      is_mini = true
   end

   if ARGV[index] == '-subject'
       subject = ARGV[index+1]
   end

   if ARGV[index] == '-count'
      num_cls = ARGV[index+1].to_i
   end
end

puts "count: #{num_cls}" 

assert_value(project,'No project specified! Add to p4_email.yaml or use the -project command-line argument')

# set up p4 connections
p4 = P4.new
p4.client = config['p4Client']
p4.password = config['p4Password']
p4.user = config['p4User']
p4.host= config['p4Host']

p4.connect
p4.run_login

# retrieve recent changelists
lists =  p4.run_changes('-u',p4.user,'-m',num_cls, '-s','submitted')

#get the id
msg_body = ""
(0...(num_cls)).each_with_index do |obj,index| 
    cl_num = lists[index]['change']

    #get the full details for that cl
    cl_full = p4.run_describe(cl_num)[0]
    cl_action_list = cl_full['action']
    cl_rev_list = cl_full['rev']
    msg_body = msg_body + "Change #{cl_num} by #{p4.user}@#{p4.client}\n\n"
    msg_body = msg_body + cl_full['desc'] + "\nAffected files ...\n\n"
    cl_full['depotFile'].each_index do |index|
        msg_body = msg_body + cl_full['depotFile'][index] +\
                   "##{cl_rev_list[index]} " + cl_action_list[index] + "\n"
    end
end
p4.disconnect

#compose email
outlook = WIN32OLE.new('Outlook.Application')

message = outlook.CreateItem(0)
submit_type = "submit"
if is_mini
   submit_type = "mini"
end
message.Subject = "p4 [#{project}] #{submit_type}: #{subject}"
message.Body = msg_body
message.To = '[studioemail]'
# todo: should invoke the method by using reflection
if sendMode == 'Display'
   message.Display
elsif sendMode == 'Send'
   message.Send
end

Thoughts On Core Data

2010-04-04T12:46:00.000-07:00

I started a new iPhone app, and I decided to use the Core Data framework.

For my first app, I built an object wrapper around calls to sqlite, the embedded database built in to the iPhone frameworks. Core Data didn't exist, so everyone had to roll their own solution to this problem. I thought about just using my original solution again — it's well tested, it's a few tweaks from total reusability, and I know SQL well — but my iPhone programming is mostly about learning new technologies, so I gave Core Data a try.

Core Data is basically an ORM system. I've used a number of these over the years; I've even written some, including, in a minor way, the sqlite wrapper I mentioned above. All the ones I've seen abstract away the notion of a "database" so that the bulk of the system just sees objects without knowing their origin.

Here are some of my initial thoughts on Core Data.

Core Data abstracts the database away so much that you can't actually get to it. I recognize that Core Data can run on top of any number of storage solutions, but I feel like if I know it's running over a database, I should be able to manipulate the database myself. Bulk updates of database info — versus loading each object and modifying it — are just one scenario where that would be useful.

Objects managed by Core Data have to extend a single base class. This isn't a huge problem for my model, but it does mean you use up the one inheritance you have in Objective-C. Java has the same limitation, and most of its ORM solutions don't require you to extend a class, which gives you more flexibility in the long run.

Migrating a model should not be an "advanced" topic. One minor change to a model, and you have to nuke the data for your app, which is a bother when you're actually using it. Yes, there are a range of ways to accomplish your goal. But in my first iPhone app, I just wrote a few lines of SQL and had them run against the database at startup: Migration to new models was a snap.

The NSFetchedResultsController is a delight to use. With a few short lines of code, you have a model object you can use to drive table views of data.

Maybe I haven't read up on it enough, but when Core Data is running against a database, I'd like to see explain plans for its queries and be able to check its index usage.

Running arbitrary queries is extremely verbose, again because of the inability to run SQL directly. I wanted the ability to display a unique list of existing non-null values for an object's property in my app so that a user could either enter a new one or select an existing one. In SQL, that would be something like SELECT DISTINCT property_column FROM object_table WHERE property_column IS NOT NULL ORDER BY property_column. The Core Data version of this is:


    NSFetchRequest *request = [[NSFetchRequest alloc] init];
    [request retain];
    NSEntityDescription *entity = [NSEntityDescription entityForName:@"CallSlip" inManagedObjectContext:[self managedObjectContext]];
    [request setEntity:entity];
    [request setResultType:NSDictionaryResultType];
    NSExpression *keyPathExpression = [NSExpression expressionForKeyPath:researchAreaField];
    
    NSExpressionDescription *expressionDescription = [[[NSExpressionDescription alloc] init] autorelease];
    [expressionDescription setName:researchAreasKey];
    [expressionDescription setExpression:keyPathExpression];
    
    [request setPropertiesToFetch:[NSArray arrayWithObject:expressionDescription]];
    
    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"%@ != nil", researchAreaField];
    [request setPredicate:predicate];
    
    [request setReturnsDistinctResults:YES];
    
    NSSortDescriptor *descriptor = [[[NSSortDescriptor alloc] initWithKey:researchAreaField ascending:YES selector: @selector(caseInsensitiveCompare:)] autorelease];
    NSArray *descriptors = [NSArray arrayWithObject:descriptor];
    [request setSortDescriptors:descriptors];
    
    NSError *error = nil;
    NSArray *results = [[self managedObjectContext] executeFetchRequest:request error:&error];

That version isn't exactly shorter.

Compared to other, similar frameworks, I'd rank Core Data as decent. I imagine it's scalable enough for a client application, where you probably don't have to worry about anything larger than 50,000 records. And, if you don't know SQL, it's probably better than just dumping an object tree into a file. But if you know databases, you're likely to find it frustrating as often as you find it useful.

Dynamic XML Schema Elements With JAXB

2010-03-17T09:35:00.000-07:00

I solved an interesting problem the other day at work, and I wanted to write about it here, mostly so I don't forget how I did it.

I can't talk about what I'm doing at work at the moment, but I came up with a scenario from a different industry that presents some of the same problems.

Let's say you're building a web service for a stock trading desk. You generate a variety of reports in XML and JSON so that you can build a robust, AJAX-y front end solution.

Imagine that the head of the desk wants a report that gives a summary of all the stocks traded that day, complete with volume bought and volume sold. The desk trades different stocks each day, of course, and there are a vast number of valid ticker symbols that could appear, with more coming online as companies go public and with some disappearing as companies get delisted.

You might end up with an XML structure that looks something like this:


<report>
    <tradedStocks>
        <stock>
            <symbol>AAPL</symbol>
            <bought>100</bought>
            <sold>100</sold>
        </stock>
        …
    </tradedStocks>
</report>

With a corresponding JSON structure like this:


{"report":{"tradedStocks":[
    {"symbol":"AAPL","bought":"100","sold":"100"},
    …]}}

So far, so good. Now imagine that the head of the desk wants to treat AAPL differently. Instead of being mixed in with the other stocks traded that day, it should be at the head of the list and printed in green.

When you code this special-case logic on the front end, it will probably look something like this (in JavaScript):


    function findSymbol(symbolName) {
        for (stock in report.tradedStocks) {
            if(stock.symbol == symbolName) {
               return stock;
            }
        }
        return undefined;
    }

    aapl = findSymbol("AAPL");

Even wrapped in a function, that's a bit cumbersome. Plus, if the desk has traded hundreds of stocks, that iteration can be time-consuming, especially if the head of the desk wants to call out stocks that the desk shouldn't be trading: The code goes through the entire list of stocks only to return undefined.

It would be easier and more readable to be able to do something like this:


aapl = report.tradedStocks.AAPL

But that would necessitate an XML structure in which the element name stock was replaced by the element name AAPL. Which would in turn mean that the elements allowed under tradedStocks were drawn from a very large list of ever-changing element names. In essence, subelements under tradedStocks could literally be anything.

XML doesn't really allow for that. It assumes you have a well-defined structure. And tools like JAXB build on that.

You could put the special-case logic on the server-side, of course. Grab the AAPL data from the list of traded stocks, and make a sub-element of report be AAPL with further subelements showing the data. That would repeat data, which may not be the end of the world, but what about the next time the head of the desk wanted special-case logic? More server-side custom logic.

Here's how I solved the problem.

First, I changed the logic in our XML formatter. My view layer is isolated from the rest of the application, and within it it shuffles the model to different formatters depending on the format that was requested: xml goes to the XML formatter, json goes to the JSON formatter, and so forth.

The first incarnation of our XML formatter did the obvious thing: It just used the JAXB engine to spit out the XML based on the JAXB annotations. But I made it act more like our JSON formatter, which receives events from a parser that analyzes the JAXB annotations and constructs the output based on those events.

Why switch? Because I wanted custom annotations. By making our XML formatter act as an event listener too, I could interject events based on annotations that JAXB doesn't know about. The new XML formatter (and the JSON formatter) wouldn't notice anything odd about the data, because it would just be another event.

Next I created a FlattenableMap annotation, which our JAXB parser spies and interprets as "take this map, and for each key-value pair, fire an appropriate event." To use the above example, there would be a Map in the report object that would key stock ticker symbols to stock objects. In that case, our parser would say "I'm starting a complex object whose name is 'AAPL'." All the infrastructure would then fire appropriately, and you'd end up with


<report>
    <tradedStocks>
        <AAPL>
           <bought>100</bought>
           <sold>100</sold>
        </AAPL>
    </tradedStocks>
</report>

On the front-end side, a JavaScript programmer merely writes:


   report.tradedStocks.AAPL

This implementation also means that no matter what stocks show up in the report, they'll be pushed out to the client (assuming the stocks get loaded into the Map). There's no need to maintain the code to add "allowed" stocks. So it's automatically maintainable based on the real data in front of it, even if that data changes (which, in my case, it certainly will).

You could fairly point out that this means we're not using valid XML. You could certainly not construct a DTD or schema for this logic. But the reality is that eventually, we probably will have needs defined enough to allow us to construct the equivalent of report.AAPL, at which point we can have a more rigid schema. In effect, our schema is still so much in flux that every week or so creates a different schema. My code keeps up with those changes, even without me doing any new work.

Highest-Scoring Word In WordCrasher

2010-03-04T20:43:00.000-08:00

I've become a fan of WordCrasher on the iPhone. Think of it as <insert your favorite word game> meets Tetris: You tap on bubbles, which are falling from the top, to make words and clear them from the screen.

I've unlocked most of the achievements, but there are two secret achievements that have escaped me. I'm convinced that one of them is finding the highest-scoring word in the dictionary. The in-game view of the leaderboards, as of this writing, shows just four people who have managed to find a 2,300-point word, the highest recorded score.

What is the highest-scoring word? Well, I don't know. But I wrote a Ruby script to make an educated guess. I assumed Kevin Ng, the developer, used the Official Scrabble Dictionary as his dictionary (though his game omits at least ort and gams).

So I wrote this script, which takes a path to a dictionary file as a command-line argument:



$letterScores = {
  'a' => 10,
  'b' => 20,
  'c' => 20,
  'd' => 20,
  'e' => 10,
  'f' => 30,
  'g' => 30,
  'h' => 20,
  'i' => 10,
  'j' => 50,
  'k' => 20,
  'l' => 10,
  'm' => 30,
  'n' => 10,
  'o' => 10,
  'p' => 20,
  'q' => 80,
  'r' => 10,
  's' => 10,
  't' => 10,
  'u' => 10,
  'v' => 30,
  'w' => 30,
  'x' => 50,
  'y' => 30,
  'z' => 50 }
  
def calc_word_score(word) 
   sum = 0
   word.split(//).each do |char|
      sum = sum + $letterScores[char] if $letterScores[char]
   end
   sum * word.length
end


File.open(ARGV[0]) do |file|
    file.each_line do |word|
        score = calc_word_score(word.downcase)
        isBest = (score == 2300)
        puts "#{score} #{word}" if isBest
    end
end

I put in 2,300 as a score because that's what people have achieved. However, in the basic Scrabble dictionary, there's no word that scores exactly 2,300; there's one word that scores 2,350: zyzzyvas. So instead of the official Scrabble dictionary, I used the Enable wordlist (both wordlists are available from the National Puzzlers' League

With the Enable list, I found two words that scored exactly 2,300: showbizzy and whizzbang. I have yet to get a board where I can spell any of these, but I'm going to be trying all of them as soon as I can.

Tail Recursion

2010-02-06T14:38:00.000-08:00

The annoying thing about being a writer who has focused a lot on learning his craft is that I now have a constant editorial chatter now when I'm reading. Typos, awkward sentences, factual problems. They all crop up and prevent me from just taking in what I'm reading.

I was reading through a Scala book the other day, and I noticed this blurb in a section about tail recursion.

(If you don’t fully understand tail recursion yet, see Section 8.9).

8.10 Conclusion

The editor in my brain pounced on this end sentence, which cross-references to the same section the reader has just finished.

It took a beat before the programmer side of my brain woke up and noticed the joke.

Tail recursion is when the last instruction in a method is a call to the same method. To take an example from the book,


def boom(x: Int): Int =
         if (x == 0) throw new Exception("boom!")
        else boom(x - 1) + 1

In this specific case, boom calls itself on the last line of the method. That's tail recursion.

And the last sentence in that section is a perfect example.

Print To URL Via Smartphone

2010-02-01T07:23:00.000-08:00

According to this Mashable post, Microsoft has unveiled a new "tagging" system that would let print publication have a smartphone-readable link so that readers could visit a webpage referenced in an article by pointing their smartphones at it.

JAXB -> JSON

2010-01-30T12:37:00.000-08:00

I'm working on a new project at work, and it has a by-now-standard RESTful API web service layer. And, of course, like all such layers, it needs to support output in XML or JSON.

Supporting XML is easy in Java, thanks to a technology called JAXB. Among its many capabilities is one which lets you annotate Java objects and generate an XML document from those annotations.

For instance, you could write an object with the following:


@XmlElement(name="name")
private String name = "Joe";

Pass that to the JAXB marshaller, and you'd get this XML:


…
<name>Joe</name>
…

And the reverse is true: Make JAXB parse the XML, and you'd end up with an object whose name instance variable was set to Joe.

So our XML version was done in no time. But JSON support is a less-entrenched technology, and thus there's no elegant built-in solution. My initial idea was to just use one of the JSON libraries out there and have it construct a JSON object from the XML document generated by JAXB: Dump the XML into a buffer, convert it, and print that text to the HTTP response output stream. That works (though it's not very efficient), except in this particular case:


<things>
    <thing>
         <name>Joe</name>
    </thing>
<things>

Your schema might define things as a collection with zero or more items, but the XML-to-JSON converter doesn't see the schema, and so it creates a things object instead of a things collection. Once there are two items in the collection, everything works fine. (It's possible that if we had an actual schema doc somewhere, the converter could know about it. But since the annotations define the output, we don't have an xsd file around at the moment. )

I looked at various options for mapping objects, but most felt like duplicate work: I'd have to maintain a JSON mapping in sync with the XML mapping, a setup that's guaranteed to get messed up in some subtle way someday. What I really wanted to do was use the JAXB annotations as the definition for the JSON output.

Jersey looks like it's trying to accomplish the same thing, but it also seemed (when I first looked at it) to be not yet ready for prime time.

So I came up with my own solution. While I can't post the code, I can give you a good sense of how it worked.

I created a concrete object,JAXBBridgeParser. I also created an interface, JAXBBridgeParserListener. You throw a JAXB annotated object and an implementation of JAXBBridgeParserListener at the parser, and it uses reflection to find the annotated fields and methods in that object. For each annotated field/method, it calls some appropriate method on the listener.

In addition to the ubiquitous startParsing and finishParsing messages, the parser fires special-purpose messages at the listener. The easiest scenario is what I call the "simple type" field or method. A String, a Java primitive, a Date, etc. In that case, the parser says to the listener, "Here's the name of this field and here's the value." In the JSON scenario, this translates to a key-value pair.

Next up is what I call the "complex type" field or method. This is a value that is itself an annotated object. In that scenario, the parser first tells the listener that it's beginning a complex object with a given name; then it recurses into a processObject method with that new object. That will in turn trigger its own "simple type" or "complex type" messages. When it comes out of the recursive call, the parser tells the listener that it's done with the complex object of the given name. This corresponds to a JSON key-value pair where the value is an object.

Finally, I have to worry about collections. These could contain simple types or complex types. When the parser sees an annotated field or method that is a collection, it tells the listener that it's starting a collection via the beginCollection method in the interface. For each item in the collection, it sends a message to the listener telling it that it's processing an object inside a collection. There are separate methods for simple types and complex types in collections. When it's done with the collection, it tells the listener that it's finished. In JSON, that corresponds to a list that might look like this: ["a","b",{c: "c",d:"d"}].

The end result works like a charm: My JSON objects line up perfectly with our XML documents, and I don't have to do anything to get them there. The JSON listener is about 20 lines of code. Any object that can be converted to XML can also be converted to JSON simply by passing it through the system. (A custom Spring ViewResolver figures out the right converter to use based on the request, so anything that can be served up as XML also automatically gets a JSON version.)

And I can support new formats pretty easily as they rise in prominence in the web world. I've thought about wiring up an HTML formatter that would give default browser versions of the data. HTML is a little trickier because I'd want some of the items — object IDs, for instance — to be attributes and some of the items — names, for instance — to be page content. But it should be doable. I've also thought of wiring up YAML support, which should be brain-dead simple, just because I can.

My particular scenario is pretty simple: We don't use some of the deeper features of JAXB, so I don't have to worry about handling them. I do, however, support @XmlJavaTypeAdapter, running the referenced adapter on a field before the parser sends the value to the listener. And my version has the downside that I, and not some larger open-source community, have to support and extend it. Still, it was a pleasant little exercise, and it's working well.

If you go this route, I encourage you not only to have lots of unit tests to catch subtle edge cases, but also to set up a more behavior-focused test. In my case, I made a test listener that simply counts the number of messages it gets from the parser. I set up an object with a fixed set of annotations, and then passed it and the listener to the parser. My "unit test" then checks the counts for each message in the listener.

Polyglot Programming

2010-01-23T19:32:00.000-08:00

I recently read through The ThoughtWorks Anthology, a collection of essays by Big Thinkers in the realm of systems design. The essays were largely interesting, but one in particular resonated with me: Polyglot Programming. The author made a compelling case for using the Java Virtual Machine — a robust, mature, well-tested infrastructure — as a platform in which any number of languages can co-exist.

Java's a good language, of course, but it's not good at everything. Why not mix in other languages that can run in the virtual machine but offer strengths in the face of Java's weaknesses, asked the author. I've toyed with this idea before, especially with adding Scala's potential for highly concurrent code, but the essay lit a new fire in me.

I came up with a way to try out the concept. We have a bunch of queries in our service layers, and Java blows when it comes to formatting long strings. Without the ability to have one string span multiple lines, you end up with something like this:


String query = "SELECT table_a.*,table_b.* FROM table_a,table_b,table_c " +
                          "WHERE table_a.some_column = table_b.some_column " +
                          "AND table_b.some_column = table_c.some_column " +
                          "AND table_c.id = :idValue";

Easy reading, right? Not only is it annoying to read, it's error prone. I almost always forget a space in one of these long strings, causing SQL exceptions that don't show up until runtime.

Ruby, like many other scripting languages, allows for a "here document" which basically says, "Treat this text following double less-than signs as a double quote and just pull in everything after it until you see the same text again." In Ruby, you might write the query above as follows:


query = <<QUERY
SELECT
     table_a.*,
     table_b.*
FROM
     table_a,
     table_b,
     table_c
WHERE
     table_a.some_column = table_b.some_column
     AND table_b.some_column = table_c.some_column
     AND table_c.id = :idValue
QUERY

Which is more readable. I admit this is not a monumental problem, but it did offer an opportunity.

Enter JRuby. JRuby is a Ruby interpreter written in Java. Curiously, it now outperforms the C-based interpreter in lots of benchmarks, which is not only a testament to the maturity of the JVM but to the dedicated open-source team that have devoted themselves to improving JRuby. JRuby's main benefit is that you can access the sweeping Java API from within your Ruby scripts, but you can also invoke Ruby scripts from your Java code.

I made a new class called QueryContainer that would serve as a facade for managing the Ruby invocations and giving Hibernate Query objects back to the service layer. No other layer in the code would need to know about invoking Ruby: QueryContainer would translate the scripts into objects useful elsewhere in the system. Inside each Ruby script, I made a class to act as a namespace (because I opted for a singleton of the Ruby interpreter instead of multiple copies), and then inside each class defined hash literals that looked something like this:


QUERY_1 = {
:type=>AppConstants::SQL,
:query => <<QUERY
SELECT
     table_a.*,
     table_b.*
FROM
     table_a,
     table_b,
     table_c
WHERE
     table_a.some_column = table_b.some_column
     AND table_b.some_column = table_c.some_column
     AND table_c.id = :idValue
QUERY
}

What's that AppConstants::SQL thing? AppConstants is a Java class in our system that has some globally useful constants. Because it's JRuby, I can use constants from my Java classes. We have two query languages in our system: normal SQL and Hibernate's abridged SQL. QueryContainer needs to know which query language it is because Hibernate defines a createQuery method for HSQL and a createSQLQuery method for SQL.

But it gets more complicated. If you have a SQL query that returns everything you need to construct a Hibernate object, you need to tell Hibernate what kind of object it is. (You don't need to do this for HSQL.) I added an entityClass key to the SQL hash literals, and had it reference a Java class object (.java_class when you're in JRuby, since .class has meaning in the Ruby world. In other words:


QUERY_1 = {
:type => AppConstants::SQL,
:entityClass => BusinessObject.java_class,
…
}

Here's the final flow. Some method in the service layer wants to run a query. It calls a method in the base class called getQueryForKey, passing in the query key it wants. That base class method calls a similar method on a QueryContainer instance variable held by the base class. QueryContainer was initialized with the Ruby script that will act as a resource, and it reaches into it to find the keys in the hash literal with the same name as the key that's been moving through the chain. e.g.,: QuerySet1::QUERY_1[:query]. If it's an HSQL query (QuerySet1::QUERY_1[:type]), QueryContainer just constructs a regular Query object. If it's a SQL query, QueryContainer constructs a SQLQuery object and calls addEntity on it, passing in the Java class from the :entityClass key of the hash literal.

So how does it work? Well, on the one hand, it accomplishes what I wanted. My queries have been factored out into new files, and they've been re-produced in a format that's easier to read and less error-prone. The entire rest of the system is ignorant of their source. It makes the case for adding languages that have strengths (in this case, the relatively minor advantage of string literals that can span multiple lines) to a deploy.

But on the other hand, JRuby seems to have added a sizable chunk of memory to our app. Shortly after I put in this system, our dev server started running into OutOfMemory errors on a regular basis, a process that I've contained somewhat by disabling some other systems. And this is with a singleton of the interpreter. I've found little information about this, and so I'm wondering if JRuby is the way to go. I haven't hooked up a profiler yet to determine the real source, but it's the only thing that's changed.

I've started looking at Groovy as an alternate. At least if I go that route, only QueryContainer needs to change.

Menu For Hope Source

2010-01-20T09:51:00.000-08:00

For the past few years, I've run the raffle program that generates the winners for Menu For Hope, an annual fundraising event started by Pim that incorporates food bloggers from around the world and raises tons of money for hunger relief.

It's been a couple of years since I first posted the source code, so I thought I'd post the current version. It hasn't changed much: I made the parsing a bit smarter and added support for giving people tickets even if they asked for fewer tickets than they bought (I used to skip those as invalid). Looking at it now, I see a number of ways it could be cleaner and better architected (can you say regex?), but it continues to work. If you see any bugs, feel free to mention them!

Keep in mind that I don't run the program blindly: I run it and then look over every line of output to make sure the program is behaving the way I expect. If it's not, or I see an error, I usually end up fixing some minor data issue (People often think O is the same as 0, for some reason, or they write UW01x2 UW02x3 instead of 2xUW01, 3xUW02. My program handles the latter but not the former because it attaches the 2 to the second string in the field instead of the first.)

Here's the main class.


  public class MFHRaffle {
  
  
   public static void main(String[] args) {
     List prizeCodes = new ArrayList();
     try {
  RandomAccessFile prizeFile = 
      new RandomAccessFile("prizeFile.txt","r");
  String curPrize;
  while ((curPrize = prizeFile.readLine()) != null) {
      prizeCodes.add(curPrize.trim());
      System.out.println("Added prize code: " + curPrize);
  }
     } catch (IOException ioe) {
  System.err.println("Error reading prize code file");
  ioe.printStackTrace();
     }
    // divide args into buckets
    // there's only one input arg (file) so we can size list in advance
    List commandArgs = new ArrayList(args.length - 1);
    Map params = new HashMap();
    String filename = null;
    for (int i = 0; i < args.length; i++) {
     if (args[i].startsWith("-")) {
      commandArgs.add(args[i].substring(1,args[i].length()));
      if (args[i].startsWith("-oneprize") ) {
       params.put("oneprize",args[i+1]);
      }
     } else {
      filename = args[i];
     }
    }
  System.out.println("Using file: " + filename);
  
  if (commandArgs.contains("testrandomdraw")) {
   testRandomDraw();
  }
  
  boolean debug = true;
  if (commandArgs.contains("debug")) {
   debug = true;
  }
  
  MFHDataParser parser = null;
  if (commandArgs.contains("csv")) {
   parser = new CSVDataParser();
     } else {
      parser = new ExcelDataParser();
     } 
     parser.setDebug(debug);
            parser.setValidPrizes(prizeCodes);
     
     Map<String,List<String>> entries = parser.extractData(filename);
     Map<String,Integer> prizeCounts = new HashMap<String,Integer>();
     Map<String,String> prizeToWinner = new HashMap<String,String>();
     
     //produce a sorted list
     List<String> sortedPrizes = new ArrayList<String>(entries.keySet());
     Collections.sort(sortedPrizes);
     
     System.out.println("*******************************");
     if (!commandArgs.contains("oneprize")) {
   // for every entry in map, throw list to randomDraw
   for (Iterator<String> prizeIt = sortedPrizes.iterator();
    prizeIt.hasNext();) {
     // drumroll please...
     String prize = prizeIt.next();       
     List<String> bidders= entries.get(prize);
     prizeCounts.put(prize,new Integer(bidders.size() * 10));
     
     String winnerEmail = randomDraw(bidders);
     prizeToWinner.put(prize,winnerEmail);
   }
  } else {
   String prizeCode = (String)params.get("oneprize");
      String winnerEmail = randomDraw(entries.get(prizeCode));
      prizeToWinner.put(prizeCode,winnerEmail);
  }

   
  // tab-delimited output for fatemeh
  System.out.println("********** TEXT ****************");
  for (Iterator<String> prizeIt = sortedPrizes.iterator();
   prizeIt.hasNext();) {
    String prize = prizeIt.next();
    System.out.println(prize+"\t$"+prizeCounts.get(prize)+"\t"+
     parser.getNameForEmail(prizeToWinner.get(prize)) +"\t"+
     prizeToWinner.get(prize));
  }
  
  // html markup for brett
  System.out.println("*********************************");
  System.out.println("********** HTML *****************");
  System.out.println("<table rules=\"rows\" >");
  for (Iterator<String> prizeIt = sortedPrizes.iterator();
   prizeIt.hasNext();) {
    String prize = prizeIt.next();
    System.out.println("<tr><td>"+prize+"</td><td>$"+
        prizeCounts.get(prize) + "</td><td>" +
        parser.getNameForEmail(prizeToWinner.get(prize)) +
         "</td><td>" + prizeToWinner.get(prize) +
          "</td></tr>");
  }
  System.out.println("</table>");

           
 }

Here's the base class for parsers:


public abstract class MFHDataParser {
     
    private boolean debug=false;
    
    private char[] delims = {',',' ','.',';'};
    
    private Map<String,String> emailToName = new HashMap<String,String>();
    
    private List<String> validPrizes = new ArrayList<String>();
    
    public void setValidPrizes(List<String> prizeList) {
     this.validPrizes = prizeList;
    }
    
    protected void mapEmailToName(String email, String name) {
     this.emailToName.put(email, name);
    }
    
    public String getNameForEmail(String email) {
     return this.emailToName.get(email);
    }
    
    public abstract Map<String,List<String>> extractData(String filename);
    
 /** Do our darndest to figure out what prizes a donator has mentioned on
  *  a given line. Note: Be sure to complain if we don't recognize a prize.
  *  
  */
    protected List<String> extractPrizes(String prizes, int amount) 
     throws NoPrizeFoundException {
    
     String ucPrizes = prizes.toUpperCase().trim(); //for consistency's sake
     if (isDebug()) {
      System.out.println("Incoming prize string " + ucPrizes);
     }
    
     // basic strategy
     // find two numbers followed by a delim (, space, ., eol, ;)
     // back up and find two letters before it
     // then back up (not into an earlier code) and find a #
     // can't use java's regex abilities because i need to divide into larger chunks
     
     // put that many copies into the list
     // verify that size of list = donation /10. bark if not
     // List = one of each real raffle ticket (5xUW03 -> 5 entries in List)
     
     List<String> prizeList = new ArrayList<String>();
     List<Integer> prizeCounts = new ArrayList<Integer>();
     
     if (ucPrizes.length() < 4 ) {
      throw new NoPrizeFoundException(ucPrizes);
     } else if (ucPrizes.length() == 4) {
      // exact count. easy case, but verify that it's legit
      
      String prizeCode = findPrizeCodeInTextBlock(ucPrizes); 
      if (!validPrizes.contains(prizeCode)) {
       System.out.println("INVALID PRIZE CODE: " + prizeCode);
         }
      for (int i =1;i<= amount/10; i++) {
       prizeList.add(prizeCode);
      }
     } else {
      // in this case we need to walk through the list, divided it into
      // chunks, and find the prize code in each
      int chunkStart = 0;
      for (int i = 0; i < ucPrizes.length();i++) {
       if (i == ucPrizes.length() - 1 || 
        isDelim(ucPrizes.charAt(i))) {
        String curChunk = null;
        if (isDelim(ucPrizes.charAt(i))) {
          curChunk = ucPrizes.substring(chunkStart,i);
        }
        
        if (i == ucPrizes.length() - 1) {
         curChunk = ucPrizes.substring(chunkStart,i+1);
        }
        if (curChunk.length() < 4) {
         continue;
        }
        String prizeCode = findPrizeCodeInTextBlock(curChunk);
        if (prizeCode.equals("")) {
         continue; // not in this text block
        } else if (!this.validPrizes.contains(prizeCode)) {
         System.out.println("INVALID PRIZE CODE: " + prizeCode);
           }
        
        
           prizeList.add(prizeCode);
        int prizeCount = new Integer(
         findPrizeQuantityInTextBlock(curChunk,prizeCode));
        if (prizeCount == -1) {
         prizeCounts.add(new Integer(1));
        } else {
               prizeCounts.add(new Integer(prizeCount));
           }
        
        chunkStart = i + 1;
       }
      }
     }
      
  // expand prize list as needed
  if (prizeList.size() == amount /10) {
   return prizeList; // if there are as many prizes as the amount
         // would suggest, do one ticket each
  } else if (prizeList.size() == 1) {
   // create an expanded list that has one entry for each ticket
   List<String> newPrizeList = new ArrayList<String>(amount/10);
   for (int i = 0;i < (amount /10); i++) {
     newPrizeList.add(prizeList.get(0));
   }
   return newPrizeList;
  } else {
   // we have a mix of amounts and quantities
   List<String> newPrizeList = new ArrayList<String>(amount/10);
   for (int i = 0; i < prizeList.size();i++) {
    Integer count = prizeCounts.get(i);
    for (int j = 0; j < count.intValue(); j++) {
     newPrizeList.add(prizeList.get(i));
    }
   }
   return newPrizeList;
  }
 }
    
    protected int parseAmount(String amtString) {
        if (amtString.trim().equals("")) {
            return 0;
        }
     return (int)(Double.parseDouble(amtString));
    }
    
    /** Takes a guess at the prize quantity in a given text block */
    private int findPrizeQuantityInTextBlock(String chunk, String prizeCode) {
     // make a spaced version so we can look for UC 01 as well as UC01
     // walk over the string looking for numbers, skipping the prize code
     String spacedPrizeCode = prizeCode.substring(0,2) + " " +
                           prizeCode.substring(2,4);
                
        String newChunk = chunk.replace(prizeCode,"");
        newChunk = newChunk.replace(spacedPrizeCode,"");
                           
     for (int i = 0; i < newChunk.length(); i++) {
      if (Character.isDigit(newChunk.charAt(i))) {
       if (i < newChunk.length() - 1 &&
        Character.isDigit(newChunk.charAt(i+1))) {
        // two-digit quantity
        char[] digits = {newChunk.charAt(i),newChunk.charAt(i+1)};
        return Integer.parseInt(new String(digits));
       } else {
           return Integer.parseInt(newChunk.substring(i,i+1));
       }
      }
  }
  
  // one last check. some bidders wrote "TWO" instead of 2
  if (newChunk.indexOf("TWO") != -1) {
   return 2;
  }
  
  return -1;
    }
        
 /** Returns the offset of something that looks like a prize code. */
    private String findPrizeCodeInTextBlock(String chunk)  {
     
     // look for 2 letters followed by 2 numbers => prize code
     for (int i = 3; i < chunk.length(); i++) {
      if (Character.isDigit(chunk.charAt(i)) && 
       Character.isDigit(chunk.charAt(i - 1))) {
       if (chunk.charAt(i-2) == ' ') {
        if (Character.isLetter(chunk.charAt(i-3)) &&
         Character.isLetter(chunk.charAt(i-4))) {
            return chunk.substring(i-4,i-2) +
                 chunk.substring(i-1,i+1);
        }
       } else {
        if (Character.isLetter(chunk.charAt(i-2)) &&
         Character.isLetter(chunk.charAt(i-3))) {
            return chunk.substring(i-3,i+1);
        }
         }
      }
        }
        return "";
    }
    
    protected boolean isDebug() {
     return this.debug;
    }
    
    public void setDebug(boolean debug) {
     this.debug = debug;
    }
    
    private boolean isDelim(char c) {
     for (int i = 0; i < this.delims.length; i++) {
      if (c == delims[i]) {
       return true;
      }
     }
     return false;
    }
    
}

And here's the Excel-parsing subclass of Data Parser (you can see that it's got some logic that should be in DataParser, but in truth we've always used the Excel format, so it hasn't been an issue.


public class ExcelDataParser extends MFHDataParser {

    public Map<String,List<String>> extractData(String filename) {
     Map<String,List<String>> retVal = new HashMap<String,List<String>>();
     try {
      Workbook wkbk = Workbook.getWorkbook(new File(filename));
      Sheet sheet = wkbk.getSheet(0);
      for (int i = 0; i < sheet.getRows(); i++) {
    if (i == 0) { continue; }// skip headers
    String name = sheet.getCell(0,i).getContents();
    String email = sheet.getCell(1,i).getContents();
    String date = sheet.getCell(2,i).getContents();
    if (isDebug()) {
     System.out.println("amount: " + sheet.getCell(3,i).getContents());
    }
    int amt = parseAmount(sheet.getCell(3,i).getContents());
    String comment = sheet.getCell(4,i).getContents();
    if (email == null || email.trim().equals("")) {
     throw new IllegalArgumentException("No email found at line " + i);
    }
    mapEmailToName(email,name);

    
    if (comment == null || comment.trim().length() == 0) {
     System.out.println("No comment on line " + (i+1));
     System.out.println("");
     continue;
    }
    List<String> prizes = extractPrizes(comment,amt);
    if (isDebug()) {
     System.out.print( "prizes for line " + (i+1) + " ");
     for (Iterator<String> prizeIt = prizes.iterator();
          prizeIt.hasNext();) {
          System.out.print(prizeIt.next() + " " );
     }
     System.out.print("\n");
    }
    
    if (prizes.size() != amt / 10) {
     System.out.println("Line " + (i+1) + 
     " does not have the right number of prizes for the amt");
     System.out.println("");
    }
    
    // compress the lists down to MFHPair, which includes an email
    // and a count. Insert into map, keyed by prize code
    Collections.sort(prizes); // make sure they're in order
    String curPrize = "";
    int curCount = 0;
    for (Iterator<String> prizeIt = prizes.iterator();
         prizeIt.hasNext();) {
         String prizeFromList = prizeIt.next();
         List<String> bidders = null;
         if (retVal.containsKey(prizeFromList)) {
          bidders = retVal.get(prizeFromList);
         } else {
          bidders = new ArrayList<String>();
          retVal.put(prizeFromList,bidders);
         }
         bidders.add(email);
     }
     System.out.println("");
     }
     return retVal;
  }
  catch (Exception e) {
   System.err.println("There was a problem: " + e);
   e.printStackTrace();
   System.exit(1);
  }
     return null;
    }
}

buildr

2010-01-16T09:38:00.000-08:00

A co-worker of mine seems to subscribe to some mailing list where he hears about every framework and tool that ever comes into existence. And he seems to bring up each one as a potentially neat and useful addition to our tool belt at work. I tease him about it, but he is good at finding interesting things.

About one-third of his suggestions strike me as worth investigation. About one-third of those make it past a quick docs read and into my "we should incorporate this" mental bucket.

buildr is the most recent tool to run this gauntlet and end up in our source tree. buildr is a build tool that runs on top of rake, Ruby's equivalent of make, but with a Java app mindset. The mentality behind buildr seems to be, "Just because we're building Java packages doesn't mean we need to have a tool written in Java with some clunky XML config system. Maybe a scripting language would be a better choice. And, since Java build tools aren't a new frontier, let's solve the problems that are commonplace in them while we're at it."

Java's standard build tools have notable flaws. The original authors of ant decided that ant should not be a programming language. The problem is that you often want some flow-of-control/variable setting for specialized tasks in your build, and you certainly want some ability to modularize. Because ant doesn't give you lots of ways to do these things (though there are add-ons that fill this gap), you end up with verbose XML files with an annoying amount of repetition. (If you're in this boat, you might want to check out the "Refactoring Ant Build Files" essay in The ThoughtWorks Anthology.)

Maven tried to fix some of ant's problems with a convention-over-configuration mindset: Set up a directory structure in a standard way, and maven's plugins will do the right thing with a minimum of configuration. Deploying a normal war required about three lines of XML. Maven also introduced a notion of dependencies and repositories. In an ant system, you find a library you need, download its jar into your source tree, and check it in. In the Maven universe, you just put an entry in your build file, and Maven goes and fetches the jar from a network-based repository.

But should you need to stray from the convention, Maven gets in your way. Want to use the ant technique of downloading a library for some reason (like, for instance, it's an Oracle driver that can't be published from Maven repositories because of licensing)? You can set up an internal repository, but it's a fair amount of work for a developer on a small team with a lot of other things to do. You can customize Maven (and ant, for that matter) with plugins written in Java, but that always seems clunky.

Buildr tries to solve these problems. It does all the dependency stuff that Maven does, but it also allows you to construct an artifact object (the term for a built resource) around an arbitrary file. Setting up buildr at work, I faced the problem that the Oracle drivers I need aren't in the global Maven repository anymore. So I downloaded the jar, stuck it in a lib directory, and just created an artifact reference to that file. (One caveat: buildr doesn't yet do the "transitive dependency tracking" that Maven does. In Maven, if you need a library, the system will fetch it and all the libraries that that library depends on. buildr requires you to list every single dependency.

Buildr also supports hierarchical/environment-specific properties as a feature. You often need to have properties files that contain different values for different environments (database connection URLs, for instance), and you usually have to roll your own solution. buildr acknowledges this common need and just addresses it.

And if you need to do something specialized, you can write Ruby code inline. No building a Java library that conforms to some API and then configuring the build tool to use it.

I've got local builds working in buildr, but I need to convert our automated build systems to use it. Once those are in place, I plan to move forward with buildr as our sole build tool.

Programmers And Statistics

2010-01-10T12:04:00.000-08:00

A recent post on Slashdot links to this (strongly worded) article urging programmers to learn some statistics.

I am certainly not one of the programmers he mentions who pretends to a deep knowledge of statistics. I took it in college, got an A, and forgot just about everything except calculating a mean and a median. I can define a standard deviation, with a bit of hand-waving.

But his article is inspiring, especially because I want our team to own the load testing of our application. I (somewhat) recently bought Statistics In A Nutshell, so maybe I'll pick it up again at some point.

Loadtesting As Functional Testing

2009-12-17T07:29:00.001-08:00

I'm a fan of automated testing that exercises code and checks its results.

But I don't believe in striving for 100 percent coverage. Should I really spend time writing unit tests that exercise our database interactions given that Hibernate, our ORM layer, is heavily tested and community-vetted? I create mock versions of the service layer we put on top of our ORM to allow testing of the service clients, but I don't (usually) write tests to exercise the services themselves.

This works out fine. Except when I need to do a big refactor in the service layer, as I did yesterday. I moved common functionality to a base class and used Java Generics [pdf] to define methods in the base class that would, in the context of the child class, compile as if they returned the type appropriate to that child class.

How could I ensure that I hadn't broken anything before checking in the code? I thought of our load testing scripts, which I've been diligent about maintaining. They do a full CRUD cycle on a variety of object types.

I didn't need to test load, but this would exercise virtually all the code I had changed. I brought them up, set the thread count to 1, pointed them at my machine, and pressed go. Sure enough, they uncovered a few bugs. I fixed them, re-ran the scripts, and they all passed.

Automated testing is good. No matter what form it's in.

Facebook's Puzzles

2009-12-11T19:01:00.000-08:00

A co-worker of mine pointed me to Facebook's Puzzle Page, a suite of challenging puzzles. The puzzles exist to find skilled programmers that Facebook might like to hire by letting those programmers sift themselves out, rather than taxing Facebook employees' time with trying to find wheat in the chaff of job applicants who come through normal channels.

But even if you're not interested in working at Facebook, they're a nice set of programming challenges. Especially if you're learning a new language and want some meaty exercises.

Even as someone who has challenged interviewees with logic puzzles — a standard practice for software companies — I find these types of puzzles to be interesting but not necessarily useful in finding good hires. Yes, it helps you find the bright folks who can figure out clever algorithms.

But those aren't necessarily the people who make good software engineers. Software engineering is about delivering quality code on schedule and within the budget. It's about writing code that other people will be able to work with and maintain, even years later. It's about debugging, but it's also about exposing bugs as early as possible. It's about understanding trade-offs, risks, and benefits. I spend a very small percentage of my time sussing out clever algorithms: I spend a very large percentage figuring out how to deliver scalable, high-quality features in a cost-effective way.

Being a good puzzle solver doesn't preclude those skills, of course, but it doesn't point to them, either. Puzzles test that you're bright in one way; they don't tell your employer if you can actually do your job.

On the other hand, Facebook is no doubt swamped with would-be employees, so these puzzles offer a way for someone to prove that they can be internally motivated (the only effective form of motivation) and self-starting. And they do test for one sort of programming intelligence. And, finally, I haven't yet figured out a way to test for the kind of programmer I look for, other than interviewing people.

Learning Ruby

2009-12-09T22:27:00.000-08:00

Recently, I decided to learn Ruby, the programming language du jour of Internet startups.

The main reason was an essay in The Thoughtworks Anthology, a collection of essays about software development from some of my field's luminaries. The essay argued for polyglot programming, especially since the Java Virtual Machine enjoys wide distribution; a mature, scalable implementation; and a number of languages that compile down to Java bytecode. It showed a typical "isBlank" method in Java and an implementation in Ruby (in particular, JRuby) that was one-third the length and much more readable. The end result: code that's easier to maintain overall.

There were a few other reasons. If I'm laid off, Ruby knowledge would be a good resume item. I'm also drawn to the language's "Behavior Driven Development" testing tools Cucumber and RSpec, which a co-worker showed me. But the Thoughtworks essay was the catalyst.

I bought the "pickaxe" book and dug in.

As I've said before, I have a standard programming project I assign myself when I'm learning a new language: a solver for various types of word puzzles that you'd find in the monthly publication of the National Puzzlers' League. By the end of writing it, I have a good sense of the language's syntax, I've dug into the established class library a bit, and I've even learned something about performance: The dictionary I use is 838,000 lines, which means that I often have to optimize the code that runs in the loop over those lines.

I got my program up and running pretty quickly, and it's easy to see why people like Ruby. Even complicated tasks come together in some form pretty quickly in a much less verbose form than they would in, say, Java. I like that the language includes mixins, a way to fake multiple inheritance, and I like that you can add behavior to an existing class (which you can also do in Objective-C via categories) without subclassing it. I added a method to the String class with no effort.

And, though I haven't tried this yet, Ruby can plug in to the Open Scripting Architecture (do they still call it that?) on Mac OS X so you can control your applications via a Ruby script. At this stage in my life, I'd prefer writing a Ruby script to writing something in AppleScript.

Next, I'll try and put up a simple Ruby on Rails app. I haven't fully embraced Ruby as the one true path, but it's definitely a worthwhile addition to your programming toolkit.

Knowledge Creates Possibilities

2009-11-29T08:46:00.000-08:00

When I first left college 16 years ago — a bio major with a head for computers — I worked for Bank of America. I helped design software for a lucrative trading desk. One of the strongest memories I have from that time is showing a piece of software to the head of the trading desk, a tall, nervous, white-haired man who stuttered when his thoughts outpaced his ability to voice them. My boss and I had taken his requirements and created software that met them. He looked at the software and said, "That's nice. I wonder: Could it do this, too?"

I don't remember what the "this" was, but it wasn't a feature he had originally mentioned. This happened again. And again. And again. Couldn't he just tell us what he wanted the first time around? It took me a year or so to realize what was happening: He couldn't define the features he wanted because he didn't know what was possible. Once he saw one feature, it expanded his horizon of what was doable, and he could then make the mental leap to bigger, and ultimately better, feature ideas.

It may be the most important thing I learned on that job.

It's certainly one reason that Agile programming made sense to me when I first learned about it: The mindset urges you to get functional software in front of customers as early as possible so that they can give feedback. The stated reason for this idea is that customers can tell you what's important and get what they want. I have always added a mental note: It also lets them know what's possible.

But there's no line between users and developers here. I think everyone works this way. Well, everyone except those few people who are truly visionaries.

A co-worker and I are both prototyping features that require APIs. Being modern web programmers, we're building them as RESTful APIs. REST simply means that you present data organized hierarchically by "nouns," or resources (compare flickr.com/photos and flickr.com/photos/melissanicole, for instance). You typically present it as XML or JSON via HTTP so that even inexperienced programmers can retrieve the data and parse it.

He showed me a page he had set up at the top level of his API. It didn't reference any particular resource. Instead, it presented a list of all the calls in the API. I thought it was a good idea: Rather than have the documentation in some separate place, he had put it right in front of the user. At the time, he was updating this manually. I've maintained API documents before, and it's a pain in the ass.

But seeing his page, I realized that I could generate a similar page by having the system itself analyze its own internals. Look at all the request mappings you've set up, see which ones are under the path for your API calls, and add them to the page. Voila! Documentation that keeps itself up to date. (See below if you want some of the details.)

It took some effort. I dug around in Spring code, read through tips on forums, and yelled at my computer. But by the end of the day, I had a page of API documentation, in the normal format of the API, that I would never have to maintain or remember to update. New API calls would get added without me thinking about it. A simple annotation on the method that handles the request gives a human-readable definition of what the call does. (I also added an exclude capability for calls that we don't want to expose.) You could argue that the time I spent on this feature was greater than the time I would spend maintaining the documentation. That may be true.

But a week later, I had an epiphany.

As I've said, I'm experimenting with the idea of having load testing scripts that run early and often for new features. To that end, I had a JMeter script where I manually added the API calls as I added them to the system.

See the problem?

What if I could make a load testing script that would hit that top-level page and use it to figure out all the other calls that were possible? I'd have a script that would, with no maintenance on my part, load test every API call as I added it.

Or when someone else added one. They wouldn't even have to know about the load testing script, and their call would get load tested.

It took a bit of digging in JMeter's sparse documentation, but I got it working after a few hours. I add an API call, it shows up in the generated documentation page, and then the loadtesting script picks it up. Beyond my initial excitement of seeing the flow work, I stopped thinking about it. (Again, see below for some tips.)

I felt like the head of the trading desk from all those years ago. I didn't start with a vision of a load testing script that would dynamically adjust itself. But when I implemented one feature, that script came into view. And now I will always see it as a possibility, and that knowledge will create new possibilities I can't see today.

A Self-Documenting API
I'm using Spring 3.0 as a framework, and that's what made this work. If you're in the same boat, there are a few things that will frustrate you along the way.

You'll want a reference to the DefaultAnnotationHandlerMapping object that Spring maintains. However, you can't autowire a reference into your class, because Spring doesn't, by default, register the DefaultAnnotationHandlerMapping as a bean. But you can create a bean of your own for that class, and then it will work. (Note that you'll probably want to Autowire for an array of AbstractUrlHandlerMapping objects, in case you add more down the road.)

DefaultAnnotationHandlerMapping will give you a map of all the request mappings in your system, but the values in that map are controllers, not methods. You'll have to use reflection to go through all the methods in the class and figure out which ones actually have the @RequestMapping annotation for the given request mapping.

On that note, keep in mind that you may actually be getting a CGLIB proxy for the controller, so you'll need to get the actual class name in order to use reflection.

A Self-Discovering Load Testing Script

Because my API documentation page returns its results in XML, I was able to use JMeter's XPath Extractor to derive all the URLs to hit. Couple that with a ForEach controller, and you've got a loop that will iterate over each URL in the document. Put an Http Request Sampler in that loop, set the path field to the iteration variable created by the controller, and you're hitting whatever the current URL is. That's the basic format.

Spring's request mapping syntax looks something like /foo/bar/{id}, where {id} gets replaced at request time by whatever's in that space. (This is, incidentally, an awesome feature.) But /foo/bar/{id} is not a useful URL. I wrote a preprocessor for the Http Request Sampler that replaces {id} with some other value. You may be clever enough to do it with a Regex preprocessor; I did it with a BeanShell script that takes the current path — /foo/bar/{id} — and transforms it to /foo/bar/1234. Then it puts that value into a new variable called correctedPath.

JMeter allows you to use variables in the name field for a component. This is very useful. If you leave the sampler's name as Http Request Sampler, every single URL will get put in the "Http Request Sampler" test name, which will make your load testing reports useless. If you set the sampler's name to — using the above example — ${correctedPath}, the test name will get set to /foo/bar/1234, and each discovered URL will get its own line item in your report.

Learning JMeter

2009-11-15T11:42:00.000-08:00

At work recently, I set up automated load testing for some new features I'm working on. There are teams within EA that do large-scale load testing, but I have a need they can't meet. I want to know, every single day, if our performance metrics have changed in a marked way even against a development server. If they have, the change probably came from code that was checked in that day. If performance went down, our team can find the fix quickly or acknowledge the new baseline. If performance went up, we can figure out why and see if the fix can be applied in other places.

I downloaded JMeter and started giving myself a crash course on the software. I've used it in the past, but it was much simpler in those days. So were my requirements.

A simple case was pretty easy: hit a couple URLs that don't require parameters but do return dynamic data. I set up a thread group, the "how many threads should I use" required container, then I added HTTP requests that hit the URLs to that thread group, and then I added a "listener" that parses the responses. That gave me basic familiarity with the tool and allowed me to set up an automatic, nightly run of any checked-in JMeter tests. We use Hudson as a build system, and it has a nice little JMeter plugin that will graph the response times over time and also give you a simple per-run breakdown of performance.

But then I wanted a test suite that would hit a URL, parse the response, and run subsequent tests based on the data (an ID) that came back in the first response. There are plenty of hints that JMeter can do this kind of thing, but it took a lot of muddling to get it right.

I often say that I should contribute to open-source projects not as a professional programmer but as a professional writer. Documentation is often sparse and unclear, even on an established project like JMeter. Where is the "cookbook" section with "you want to do this common thing; here's how" entries? Where is the list of best practices?

You read their manual, you read the FAQ and the wiki, and you still spend a lot of time bumbling about, poking this and prodding that to see if the errors clear up. And I've barely touched the more powerful features.

Here are a few of the things I learned along the way, which I'm posting mostly for my own benefit.

JMeter can't do anything with a response — even show it to you — without knowing the MIME type. On the one hand, that forced me to return a clean response. On the other hand, I feel like JMeter should fall back to plain text if it has no other information.

Extract data from a response with a post-processor. If you want to use the response data from one http request in subsequent calls, you make the post-processor a child of that http request. (You can make a post-processor a sibling, in which case it runs after every request in the thread group.) I used a regular expression post-processor that matched the entire response. Even though there was no other data, I still needed to enclose the regex in (). I also needed the $1$ default template, even though I wasn't doing anything with multiple values.

It seems like every JMeter tutorial suggests that you add a Graph Results listener to your test plan. It shows you a graph of your response time. Ooh. Aah. Pretty. Also? Useless for debugging. I moved through errors at a rapid pace after I added a View Results Tree listener to the thread group. With that listener, you can drill down on the request/response for each and every server call. I also found the Summary Report listener to be more useful than Graph Results. Keep the pretty picture in to show your boss — it doesn't add time — but add the others to make your life easier.

Learn to love the JMeter functions. They're buried in Chapter 19 of the JMeter manual, but they're essential for making scripts that can be re-used in multiple places. I sprinkled the ${__P(name)} function throughout the text fields of my scripts so that I could fetch command-line properties such as TARGET_SERVER and TARGET_PORT. That means that running my scripts against a different environment will require nothing more than a different set of command-line arguments (prefaced with -J).

Profiling With Spring's AOP

2009-11-13T09:20:00.000-08:00

Recently, at work, I wanted to set up an easy way to get lightweight profiling information about method calls in our system. There are lots of heavyweight profilers out there that give you great features at the cost of bogging down the machine they're profiling. Fine if it's a development box; less so if it's a production machine your users are using.

Because we're using Spring, we have access to a pretty decent aspect-oriented-programming system. (AOP is basically a way to add system-wide concerns to objects without those objects having to be aware of the systems in any way: no subclassing or encapsulation.) Since profiling is a classic cross-cutting concern — something you need throughout the system but don't want any individual object to know anything about — Spring's mechanism works pretty well.

In my final system, a programmer only needs to add a @Profile annotation on a method, and it will generate profiling information in the logs. I didn't want to profile every method all the time because of the noise it would generate and the performance hit it would cause. The programmer can refine the output of the profiling system a bit, but the default is to show method name, args, return value, and timing information.(Spring's AOP framework is limited to non-private methods that are proxied by the Spring system, so private methods and intraobject calls don't get tracked. If that becomes a problem, I'll add in true AspectJ support, but that shouldn't require any real changes in my code, since Spring's support uses AspectJ syntax.)

Here are the key parts.

First, the profiling method, with a pointcut/advice combination that says "run this code around any method annotated with a @Profile annotation.



    @Around("@annotation(profileAnnotation)") 
    public Object profileMethod(ProceedingJoinPoint pjp, 
                Profile profileAnnotation) throws Throwable {
        
        StringBuilder buf = new StringBuilder("method: " + 
                               pjp.getSignature().getName());
        if (profileArgs(profileAnnotation)) {
            buf.append("; args: ");
            if (pjp.getArgs() != null && 
                pjp.getArgs().length > 0) {
                buf.append("[");
                for (Object o : pjp.getArgs()) {
                    buf.append(o.toString() + ",");
                }
                buf.append("]");
            } else {
                buf.append(" (no args)");
            }
        }

        long currentTime = System.currentTimeMillis();
        Object retVal = pjp.proceed();
        if (profileReturnValue(profileAnnotation)) {
            buf.append("; returnValue: " + 
                        String.valueOf(retVal)); // valueof prevents NPEs
        }

        if(profileTiming(profileAnnotation)) {
            buf.append("; timing: " + (System.currentTimeMillis() - currentTime));
        }
        log.info(buf.toString());
        return retVal;
}

The Profile annotation looks like this:



/** Annotation to demark methods that should be profiled. */
@Retention( RetentionPolicy.RUNTIME )
@Target( ElementType.METHOD )

public @interface Profile {
    ProfilingType[] value() default {ProfilingType.ALL};
}

And the ProfilingType enum looks like this:
public enum ProfilingType {ARGS,RETURN,TIMING,ALL}

The default output (ProfilingType.ALL) looks something like this:
method: [methodName]; args: [each arg as String]; returnValue: [returnValue]; timing: [some number of milliseconds]

Sad, But Probably True

2009-11-05T17:14:00.000-08:00

Reading the website for Apache Jmeter, I noticed this line: "JMeter's target audience is the testing community, which is generally not the hangout of developers or technical people."

Except for those developers who understand that the sooner you catch an issue, the cheaper it is to fix. Which, granted, is not as many as there should be.

Rounded Corners on UITableView

2009-10-07T19:56:00.001-07:00

The iPhone UI is a soft, graceful collection of rounded corners. Your application icon is cropped and smoothed out with the ubiquitous curves. The highlight on the selected tab of a tab bar has tiny ones. Grouped table views, the corners of navigation items, the little buttons in toolbars. You can't escape the smooth arcs.

Which would be fine if Apple didn't require you to hand craft them yourself on your UI components. You might expect a "draw this with rounded corners" method. Actually, given their prevalence, you'd expect it to be a setting on UIView: cornerRadius or something like that.

But since that doesn't exist, every developer has to write his or her own take on it. So far, mine has become fairly standard. My UIView classes implement drawRect to set a curvy clip region. I have a utility method that takes any rect, with a desired radius, and returns a path for a rounded rectangle. Add that to the graphics context, call CGContextClip, and voila, I have voluptuousness. This has worked well.

Then I tried to do it on a UITableView embedded in a larger view. It's a view, right? So this should work, right? Except that UITableView's drawRect method doesn't really do anything. The corners don't get clipped if you subclass it and clip within drawRect.

Searching on this topic suggests that this is a common problem. I read various ideas, didn't really like them for one reason or another, and then came up with my own strategy. Why not overlay my table with a view that would act as a window? It would draw the part that I normally clip off and then make the interior transparent. And with a bit of work, I could make it a reusable view I could put atop other views that didn't behave the way I wanted. No more mucking around with clipping in random subviews!

I am not a graphics programmer. Not only did this take me a while, but there are probably better ways to do it. However, you may find yourself wondering how to do the same thing, and so I'm sharing my results.

Here's my init method. I used bgColor to represent the color on the outside of the corners, and I set the actual backgroundColor to clearColor I tried using clearColor as the fill color for my rounded path, but that didn't work. It was transparent over the view's background color, which meant the whole rectangle was filled in with the view's background, not the view underneath.



- (id)initWithFrame:(CGRect)frame {
    if (self = [super initWithFrame:frame]) {
        bgColor = [UIColor blackColor].CGColor;
        self.backgroundColor = [UIColor clearColor];
        self.userInteractionEnabled = YES;
    }
    return self;
}

And here's the drawing code. I used EOFillPath to ensure that my interior was empty. Yes, I hardcoded the radius. Yes, that's lame. I'll fix it in a refactor pass.


- (void)drawRect:(CGRect)rect {
    // Drawing code
    CGContextRef ctxt = UIGraphicsGetCurrentContext();
    
    CGContextSetFillColorWithColor(ctxt,bgColor);
    CGContextAddRect(ctxt,rect);
    CGPathRef curvePath = [Utils roundedCornerPathFromRect:rect withRadius:10.0];
    CGContextAddPath(ctxt, curvePath);
    CGPathRelease(curvePath);
    
    CGContextEOFillPath(ctxt);
    [super drawRect: rect];
}

That's the result of a fair amount of fiddling, and it now works. Except for one problem: You've covered the view you want the user to see with another view, effectively removing the user's ability to interact with it. To get around this, I created a "coveredView" reference within RoundedCornerOverlay and set it in the view controller managing the whole view. I overrode nextResponder in my view to return this reference. Then I overrode the various event methods you inherit from UIView. The final step was to realize I needed to convert incoming points to be relevant within the covered view.



- (void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self nextResponder] touchesBegan:touches withEvent:event];
}

- (void)touchesMoved:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self nextResponder] touchesMoved:touches withEvent:event];
}

- (void)touchesEnded:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self nextResponder] touchesEnded:touches withEvent:event];
}

- (void)touchesCancelled:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self nextResponder] touchesCancelled:touches withEvent:event];
}

- (UIView *)hitTest:(CGPoint)point withEvent:(UIEvent *)event {
    return [[self coveredView] 
             hitTest:[self convertPoint:point toView:[self coveredView]] withEvent:event];
}

- (BOOL)pointInside:(CGPoint)point withEvent:(UIEvent *)event {
    return [[self coveredView] 
            pointInside:[self convertPoint:point toView:[self coveredView]] withEvent:event];
}

Independent Developer

2009-10-03T17:31:00.000-07:00

I assume that if anyone reads this blog — anyone? hello? — they also read my main blog, An Obsession With Food. So you've probably already seen the announcement: I launched 1.0 of my iPhone app, which helps home cooks organize the prep tasks for a dish.

What that really means is that for the first time in my professional career, I'm an independent developer. Of course, I also work for a company that can pay me. I am not planning my retirement based around sales of my application in the App Store. But I'll be the one doing the marketing — ugh — tech support, and project management. Now I'll have users. At least I hope I will.

So far, I've been the primary user of this app. In a way, this is good, because it's forced me to use the app day in and day out, finding the things that annoy me and then fixing them. But it's also bad. The app is tailored to the way I do things, even though most people probably have different menu planning strategies. I've had testers, of course, who have offered feedback. But, as of today, I have four times as many users as I did before I launched. I'm hoping they'll take the time to write me and offer suggestions, but who knows what will happen?

The Simplest Thing That Could Possibly Work

2009-09-14T08:42:00.000-07:00

One of the maxims of Agile development — and especially Extreme Programming — is to write the simplest thing that can possibly work.

There are a few driving forces behind this motto. First, a core tenet of Agile development is to get testable, functional code up and running as soon as possible. If you don't spend a lot of time writing complex code, you can get it running sooner. And complex code is harder to maintain and thus more likely to have subtle bugs. Second, Agile assumes that requirements will change. This in part because requirements always change, but also because the mere act of showing a user a current piece of software will give them new ideas for the features that will really solve their problems. Users aren't good at articulating what they need until they see something "tangible" that isn't what they need. If you get a piece of code up sooner, you give your users the chance to steer the application in the direction they really want, not the direction they thought they wanted. And when they present this abrupt change in direction — as they inevitably will — you don't have to dismantle a complex system.

A close cousin of the "simplest thing" motto is "You're Not Gonna Need It." Experienced programmers tend to think in terms of frameworks they can build, good object-oriented principles, and so forth. But if you spend a lot of time building a nice, modular system with layers and abstraction, you may have wasted your time and effort if there will only ever be one module, one implementation of a given interface, or one class in a layer.

This is not to say that you should be sloppy, or churn out quick and dirty, copy-paste code. You are a professional and should act like one. Your "simplest thing" should be tested and it should be clean code. As the requirements change beyond what your "simplest thing" can do, you can then start to build layers on top of it, refactor to make it more modular, and so on.

All these arguments came back to me when I had to implement templatable text at work. I wanted text that a non-programmer could modify, but it needed to allow variable substitution.

Java's localization system allows this to some degree, except that its variables are numeric, not symbolic. So item zero is always the same, item one is always the same, etc. regardless of where they show up in the string. This makes sense for localization, but is fragile for more general templates. What if we wanted the text to use a different set of variables from one week to the next?

There are a number of general-purpose templating systems for Java: Probably the best-known is Velocity. As I started researching what it would take to implement it, however, the "simplest thing" motto came back to mind. Not that Velocity is difficult to implement, but we only needed to support five variables, and we didn't need any of the flow-of-control logic that Velocity offers.

So I wrote a very simple set of methods that allows me to do string replaces on the five symbolic variables I allowed. The (internal) user can write any text, using some or all of those variables once, twice, or whatever they want. And just in case we ever need to move to Velocity, I used the same variable naming syntax to define our variables.

The code is simple. It has noticeable restrictions (can't use undefined variables, can't define new ones), but in this case, it works exactly the way I need it to. It satisfied the business need, and truthfully, will probably never need to support more. When I needed to add an additional piece of templated text with the same variables, it took about 15 minutes including 10 minutes of testing.

Tutorial System, Part 2: Variadic Methods

2009-09-04T00:07:00.000-07:00

In my last post, I mentioned a simple tutorial system that I wrote for my iPhone app: When I pass a key to the system, new users get a text box explaining the purpose of each screen, tap to close it, and then never see it again.

It worked well enough, though my testers have some usability comments that I need to address. But as I started working on version 1.1 of my app*, I realized that I had new needs. I had written, as I try to remember to do despite the temptation to overdesign, the simplest thing that could possibly work. That no longer fit the bill. (Writing the simplest thing that can possibly work assumes that design requirements will change. But since you can't predict those changes, even when you're the one creating them, you shouldn't waste time designing for requirements that don't yet exist. Address the problem in front of you, get functional code up and testable, and then refactor to simplify.)

I wanted the system to handle upgraders, so that upgraders would see text just about the new features, while new users would see a screen giving an overview of the page, including the new features. The key was that when a new user dismissed the tutorial box, s/he would never see the one for upgraders (because there would be no need).

I toyed with a few ideas, but I finally settled on one: Rather than passing a single key to the system, I'd pass a list of keys. The system would iterate through the keys until it found one the user hadn't seen, show the tutorial text for that key, register that the user had seen the key, and then register every subsequent key in the list.

In order to make this work, I used a variadic method, one with an indeterminate number arguments. It's been a while since I've been deep in a C-based language, so I had to refresh my memory about implementing them. (Curiously, the Objective-C bible, Programming in Objective-C, doesn't tell you how to implement them. Perhaps the new edition does.) They rely on the stdargs.h macros, va_start, va_arg, and va_end. In Objective-C, the convention is that the last item in the list be nil. For example, see NSArray's initWithObjects: method.

If you want to implement your own variadic method in Objective-C, here's the basic pattern (you don't need to import stdarg.h if you're importing the Foundation classes):



 - (void) method: (id) arg, ... {
        va_list argList; // special datatype for the argument list
        va_start(argList,arg); // sets up the iteration
       
       id curArg = arg;
       while (curArg != nil) {
             // do something
             curArg = va_arg(argList,id); // pass the datatype you expect to find
       }
       va_end(argList);
}

Here's the code in the context of my Tutorial system:



+ (void) showTutorialForKey: (NSString *)key, ... {
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    va_list argPtr;
    va_start(argPtr,key);
    
    NSString *curArg = key;
    
    while (curArg != nil) {
        if ([TutorialSystem userHasSeenKey:curArg]) {
            curArg = va_arg(argPtr,id);
            continue;
        }
        
        NSString *titleKey = [NSString stringWithFormat:@"%@ Title",curArg];
        
        NSString* title = 
          [[NSBundle mainBundle] localizedStringForKey:titleKey value:@"" table:@"Tutorial"];
        NSString* text = 
          [[NSBundle mainBundle] localizedStringForKey:curArg value:@"" table:@"Tutorial"];
        
        TutorialController *controller = 
          [[TutorialController alloc] initWithTitle:title tutorialText:text];
        [[[self appDelegate] window] addSubview:[controller view]];
        
        // clear the user for all the rest of the args
        while (curArg != nil) {
            [TutorialSystem registerUserSeesKey:curArg];
            curArg = va_arg(argPtr,id);
        }
    }
    
    va_end(argPtr);
    [pool release];
}

* I should note that 1.0 of my app is not yet out, despite being published on the App Store, because of some bureaucracy with my employer. Assuming it's going to get resolved. I've started on 1.1.

Tutorial Mode For An iPhone App

2009-05-31T09:33:00.000-07:00

One of my testers suggested that I have a "tutorial" for new users of my app: Help screens that pop up the first time a user visits a feature. There are many ways to do this, of course, but here's the implementation I designed. A client invokes the tutorial system in a simple way:



      [TutorialSystem showTutorialForKey:@"Welcome"];

The TutorialSystem class takes over the rest through a series of class methods. (There's never an instance of a TutorialSystem object. Not a very object-oriented approach, but the tutorial system doesn't need to maintain state between invocations.)

Obviously I only want to show a tutorial screen once, so showTutorialForKey: first checks to see if the user has already seen the screen. I have an "application state" dictionary file that gets used for miscellaneous persistence: the last tab the user looked at before shutting down, the date they were looking at in the "by day" view, and so forth. The tutorial system uses the same dictionary to keep track of which tutorial screens the user has seen. In particular, it looks for a key named "hasSeen" followed by the name of the key passed in to the showTutorialForKey method. For the example above, the dictionary would have a key named "hasSeenWelcome."



+ (BOOL) userHasSeenKey: (NSString *)key {
    if (ALWAYS_SHOW_HELP) {
        return NO;
    }
    NSMutableDictionary *prefs = [[TutorialSystem appDelegate] appState];
    NSNumber *didSee = [prefs objectForKey:[TutorialSystem prefsSeenKeyForKey:key]];
    
    return didSee != nil && [didSee intValue] != 0;
}

(I have a #define that lets me always have tutorial screens show up. This is helpful for testing.)

If the system decides that it needs to show the tutorial screen, it uses the passed-in key as the key for a line in a strings file, the resource files Cocoa uses for localization of text. It also uses the key plus the text " Title" as the key in the strings file for the title of the tutorial screen. That means there's a line in my strings file that maps the tutorial body text to the key "Welcome" and another line that maps the tutorial screen title to "Welcome Title."

From there, it loads an xib file that I built in Interface Builder and lays it over the existing view. That file contains a screen-sized view that doesn't allow user interaction, but has an opaqueness level of .4. That way the underlying screen will still be visible. The xib file also contains a "tutorial screen" view that contains three subviews: a UILabel for the title, a UITextView for the tutorial body, and a UILabel that says "Tap to continue." The tutorial screen view handles all the user interactions, and interprets any touch to mean "go away." Here's what it looks like.

Finally, of course, I register that the user has seen the tutorial screen by placing the appropriate entry into the application state dictionary.

Here's the full showTutorialForKey: method:



+ (void) showTutorialForKey: (NSString *)key {
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    if ([TutorialSystem userHasSeenKey:key]) {
        [pool release];
        return;
    }
    
    NSString *titleKey = [NSString stringWithFormat:@"%@ Title",key];
    
    NSString* title = [[NSBundle mainBundle]
                                   localizedStringForKey:titleKey value:@"" table:@"Tutorial"];
    NSString* text = [[NSBundle mainBundle] 
                                           localizedStringForKey:key value:@"" table:@"Tutorial"];
    
    TutorialController *controller = 
                               [[TutorialController alloc] initWithTitle:title tutorialText:text];
    [[[self appDelegate] window] addSubview:[controller view]];
    
    [TutorialSystem registerUserSeesKey:key];
    
    [pool release];
    
}

Simulator Versus Device

2009-05-28T07:43:00.000-07:00

I had an amusing realization this morning as I finished up a feature for my iPhone app. I implemented a "touch and hold" mechanism (see this excellent description of how to do it) and tested it last night on the simulator, Apple's "virtual iPhone" that makes it faster to turn around and test new code. It worked perfectly.

This morning, I put it on my phone, and it barely worked at all. It took me a few minutes, but I finally figured out the issue.

In my original code, touchesMoved dismissed the timer that eventually puts an indicator in place that you're in editing mode. On the simulator, that's not a problem: You take your finger off the mouse. But in the real world, your finger does not sit perfectly still. Mine doesn't, at any rate. So I had to add literal wiggle room. If you stay within a small radius of touches, the code won't dismiss the timer. Now it works just the way I want.

I Am Not a Graphics Programmer

2009-05-24T07:32:00.000-07:00

I consider myself a good programmer. I can comfortably chat about anything from high-level topics such as methodologies and architecture to the minutiae of multithreaded code and Java virtual machine idiosyncracies. I understand how to solve a wide range of scalability issues. I have a good eye for the subtle details that cause bugs. I know at least the basics of when one algorithm is better suited for a task than another.

Put me in front of a graphics library, however, and I freeze up. I've been doing some simple graphics work in my iPhone app — filling an area with a gradient, for instance — and it's been slow going. I don't have the terminology in my head for this work to come naturally. Transforms, alpha channels, drawing paths, blending modes. I might as well be reading Greek. And it doesn't help that Quartz, the graphics drawing system on the iPhone, is all based on C, a procedural language, instead of Objective C, an object-oriented language. It's been a very long time since I've worked in a low-level procedural language on a regular basis. (I will say, though, that Quartz tries its best to encapsulate the C code in macros that provide a straightforward coding experience. The graphics folks around me at Maxis think that it's pretty sweet, which makes me wonder what sort of libraries Microsoft gives to Windows programmers.)

Some of my frustration comes from simple lack of experience. I didn't used to know a lot about server-side programming, after all. As I do more graphics programming, I suppose more of it will come naturally, and I will learn more of the subtleties. But I've also never been drawn to that side of the coding fence. I like to say that I have no design sense. This isn't strictly true: I did the design work for Obsession With Food, and I've got some stunning quilt tops that I designed. But I'm always surprised when something I design comes out well.

Yet here I am, literally a one-man development team for this application. I can manage the architecture just fine — and it's pretty good — but there's no one I can hand off the graphics work to. It's just me. I speed through the object-oriented development side and slow to a crawl when I'm dealing with Quartz. In the long run, it's good for me to expand my horizons. But in the short term, the part of me that's used to quickly getting code up and running is growling at the long periods of time it takes me to implement the simplest little things in the graphics world.

Back to figuring out why my transparent blue is sometimes the only thing in the rectangle and is sometimes on top of the gradient that's supposed to be gone at that point. And why doesn't my grey rectangle show up at all? All this for a tiny-but-necessary bit of visual feedback on an already functional feature.

Federal Data Coming to a Web App Near You

2009-05-22T16:32:00.001-07:00

The Federal government is releasing tons of data in web-friendly formats. See the perfectly named data.gov.

Must. Finish. One. App. Before. Starting. Another.

Real-World iPhone Rotation, Part 1

2009-05-15T13:53:00.000-07:00

When I researched the techniques for supporting rotation (portrait vs. landscape) in my iPhone app, I found a few tutorials that all did the same thing: Stick a label in the middle of an otherwise unadorned view and override one method.

Simple, yes? Yes. If you're writing a boring application that just has one line of text in the middle of the screen.

My application has a tab bar. It has multiple screens. It has custom views that pop up from the bottom of the screen. It is not well served by the tutorials I found. Here, then, is one part of my tutorial, based on my own experience.

First, the default mode for any UIViewController is to not support rotation. It's easy to change that if you have a custom class that inherits from UIViewController: Override shouldAutorotateToInterfaceOrientation: to return YES. (You can get fancy with the argument passed in and only support certain orientations, but I didn't.)

But what if you're using a UITabBarController? The same idea applies, unfortunately. Make a subclass of UITabBarController and override that method. In Interface Builder, then, set your custom class as the handler. My class, RotatingTabController, does nothing except override that method and return YES. Yay for class bloat! There may be some way to just set a property in a normal tab bar controller and have it magically start handling rotation, but I haven't found it. So a brand-new class that exists solely to override a boolean return value it is.

However, once you've overridden that method for UITabBarController, you don't need to do it for all the views that hang off of it. But you do need to make sure your views behave when rotated.

The rotation animation on the iPhone is snazzy: Views arc around the screen and rearrange themselves. Get too wrapped up in the idea that you're rotating the view, however, and you'll hurt your head. Instead, think of it as resizing the view, since that's actually what it's doing. It's much easier to grok if you imagine going from this:

to this:

If you're using a built-in data view (table view, image view, etc.) as the sole item on the screen, it probably already behaves the way you want. If you're using a custom layout like this one, which includes a UITableView, controls, and buttons, you'll need to use the Size Inspector in Interface Builder. To control positioning, the Size Inspector uses the "springs and struts" metaphor: Views are locked to superviews (or not) based on the "struts" around the edge , and they expand or contract within their superview based on the "springs" on the inside of the rectangle. Click a strut or spring to enable it or disable it.

The first step in reocnfiguring your view is to set up the autosizing for the main UIView that contains all your subviews. This one's easy: Just enable every strut and spring. That ensures that each edge of the view stays a fixed distance from the corresponding edge of the superview (the application window) and changes its width or height as the window resizes.

I set up the UITableView in the picture the same way: Its top edge is a fixed distance from the top edge of the view (to make room for those other controls), and I want it to expand or shrink based on the containing view. It will expand to fill the width of the superview in landscape mode, and it will add height (and shrink its width) in portrait mode.

I wanted the button in the upper left to stay in the upper left, regardless of the orientation. In that case I enabled the strut on the left side and the strut on the top. I disabled every other strut and every spring. That ensures that the button doesn't change its size based on the orientation and that it stays in the upper left. Not surprisingly, I set up the add button in the upper right and the Today/All control next to it to be locked to the upper right corner, again without shrinking or growing.

A little bit of tinkering got all my views to look nice in either rotation. But I add my own custom views to the UIWindow object, and that is simply not working yet. The view pops out of the edge next to the home button, regardless of the phone's orientation, and it resizes in a sloppy way. When I figure that out, I'll post part 2.

Refactoring Pays Down Technical Debt

2009-05-09T14:29:00.000-07:00

One of my first posts on this site mentioned technical debt, a brilliant metaphor for the way that hurrying imperfect code out the door provides short-term gains but long-term costs for new features. Enough technical debt, and your feature development slows to a crawl. One way to pay down technical debt is to refactor code: Refactoring's main role is to adapt a code base to requirements that weren't on the table when the programmers started typing.

I've talked about continual refactoring before, which I do as a general rule in any code I find myself in, but that is usually directionless cleanup. A deliberate refactoring phase is even better. (As long as you don't fall into the "rewrite everything" trap that Netscape found itself in, which caused such a long delay in version 6 that the company essentially handed the entire browser market to Internet Explorer.)

In my iPhone app, I have three "detail" views that, when I first wrote them, had only surface-level similarities.* They had different DAL requirements, used different objects, and so forth. But as I added new functionality to each, such as my ScheduleButton widget and my date picker widget, they began to look more alike than not. I entered a bug for myself to refactor all the common code into one base class.

A week later, I did the work. There was no design requirement driving this: I just knew the code would end up in a better place at the end. I abstracted common functionality into superclass utility methods and used the template method design pattern to put a flow into the base class that used abstract methods ("abstract," because Objective C doesn't support this) to fill in view-specific details.

A week after that, I realized I needed to change the method signature for my date picker delegate methods to support a new feature. That meant going into all the delegates and updating the code. A simple search/replace wouldn't do what I needed.

Before my refactor, that fix would have affected eight methods in four classes. After my refactor, it affected four methods in two classes. I halved the development effort it took to support this new feature. But I also halved the testing effort. Since all my detail views descend from a base class that handles all the date chooser work, I only had to test one of them — plus one unrelated view — to ensure that all three were working. If I need to add a feature across all the detail views, it will take me one-third the time it would have before.

* As always, I should point out that my iPhone app just isn't that big. I'm unlikely to ever have a vast amount of technical debt in it. Still, it illustrates the point in a microcosm. Larger systems have proportionally more complexity and thus more opportunity for technical debt.

Girls On Rails

2009-05-01T08:36:00.000-07:00

Martin Fowler has a nice essay about a recent controversy in the Rails community: A presenter gave a talk from a slide deck garnished with scantily clad pretty women. The Rails community, he notes, eschews typical corporate values. But they can't ignore decades of professional discrimination against women.

Database Updates, Redux

2009-04-30T12:41:00.000-07:00

Before I came up with my solution for updating the embedded database in my iPhone app, I had a different strategy that I couldn't figure out how to implement. Of course, after I had the new solution in place, I figured out how to implement my first idea.

I wanted to figure out how to construct method names on the fly and invoke them. That way, I could have dbChangeScript0,dbChangeScript1, and so forth, and then invoke them in a while loop that ran until the application didn't have a method with the relevant name. I wanted JavaScript's eval function in Objective C. I didn't see how to do it, though.

Today, I discovered NSSelectorFromString, a function that transforms a string into a selector (essentially, a function pointer). I used it to implement Plan A as follows:


- (void) installChangeScripts {
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];

    int versionNum = [self dbVersionNumber];
    SEL curSelector = NSSelectorFromString([NSString stringWithFormat:@"dbChangeScript%d",versionNum + 1]);
    while ([self respondsToSelector:curSelector]) {
        [self performSelector:curSelector];
        [self updateDbVersion:versionNum +1];
        
        versionNum = [self dbVersionNumber];
        curSelector = NSSelectorFromString([NSString stringWithFormat:@"dbChangeScript%d",versionNum+1]);
    }
    [pool release];
    
}

Is this a better version? I've lost the self-documenting method names such as makeVersionTable, but I've removed an in-memory array and a #define that I had to remember to update when I updated the array. In other words, I've increased the code's maintainability. (The code looks shorter, but only because I also factored out the version update into a new method.) Overall, I consider this a win.

The Ruby on Rails community has a mantra: convention over configuration. In Rails, you can get a full website up and running just by following a few naming conventions — ones you probably would have used anyway. A class of name X will translate to a database table named Y and so on. Compare that to Spring, a powerful Java framework that requires constant work in the XML-based configuration files.

In essence, I voted for the convention side of the fence in this code. By following a consistent naming scheme, I can write a small method that will do all the work. All I have to do is add a new method with a predictable name. Compare that to an array that I had to remember to configure properly.

On-The-Fly Database Updates

2009-04-30T07:28:00.000-07:00

I love the fact that the iPhone comes with an installation of SQLite*. As someone who's worked with relational databases and the SQL language for his entire professional career, I happily use it to store the information I need for my iPhone app.

But with an embedded database comes the need to alter its structure on the fly as my data requirements change. I'm still in beta testing, so I could tell all my testers to nuke their data the next time they install my application, but I'd prefer to avoid that, and I certainly won't tell my future customers to do that.

Instead, I came up with a simple system for making the app run its own database updates. I have a class, DBAccess, that puts an object-oriented wrapper around the very procedural SQLite calls. I created a new method, installChangeScripts, that looks like this:



- (void) installChangeScripts {
    SEL changeScripts[CHANGE_SCRIPTS];
    
    changeScripts[0] = @selector(makeVersionTable);
    changeScripts[1] = @selector(addTaskDuration);
    
    int versionNum = [self dbVersionNumber];
    
    for (int changeScriptIndex = versionNum + 1; 
                changeScriptIndex < CHANGE_SCRIPTS; 
                changeScriptIndex++) {
        [self performSelector:changeScripts[changeScriptIndex]];
        NSString *versionUpdateQuery = 
              [[NSString alloc] 
                      initWithFormat:@"update app_version set version_num = %d",
                                                     changeScriptIndex];
        [self executeUpdate:versionUpdateQuery];
        [versionUpdateQuery release];
    }
}

The gist is straightforward: Make an array of method selectors (Objective C's way to find a method at runtime), iterate through them starting at the database's current version number + 1, and update the version number after the script has run. The only tricky part is that the version table doesn't exist in the default version of the database, so I have logic to handle the case where it doesn't exist:



- (int) dbVersionNumber {
    int versionNum = -1;
    // get the current version. handle this case differently since it drives the rest
    // of the logic
    NSArray *queryResult = [self executeQuery:@"select version_num from app_version"];
    if ([queryResult count] > 0) {
        // handle the result 
        NSDictionary *versionResult = [queryResult objectAtIndex:0];
        versionNum = [[versionResult objectForKey:@"version_num"] intValue];
    }
    [queryResult release];
    return versionNum;    
}

Originally, this logic was in the installChangeScripts method, but that was an easy refactor: self-contained logic that only produces a single value used by the rest of the method. See the Extract Method refactoring sequence.

Not surprisingly, the first change script creates the version table.



- (void) makeVersionTable {
    [self executeUpdate:@"create table app_version (version_num integer)"];
    [self executeUpdate:@"insert into app_version values (0)"];
}

There's at least one other good refactoring target here: The setup of the list of install scripts should be in its own method because it's a separate concept from actually running the scripts. I just need to remind myself how to deal with calloc, free, and other denizens of the pointer/array abyss.

* I also love the group's motto, "Fast. Small. Reliable. Choose any three." A nice spin on the various "choose two" sayings such as "Fast. Scalable. Cheap. Choose two."

Old-School Coding Techniques

2009-04-29T21:27:00.001-07:00

Computerworld takes a look at old-school programming techniques we should all be happy to forget. But what about old-school techniques that are still relevant or should be revived?

Meaty Learning Projects

2009-04-28T10:15:00.000-07:00

My job description implicitly includes understanding current technologies, philosophies, and methodologies. Usually, this is just a surface-level knowledge about what's going on in the industry, but occasionally I decide to dig deeper.

For instance, right now I'm learning Scala because I think it will solve some thorny performance issues at work. But it's a Catch-22: I don't want to introduce a whole new language without knowing if it will really benefit us, but I can't know if it will benefit us until I try it with a meaty project. And it's even worse than normal in this case: The feature I really want to explore — actors — requires a decent chunk of knowledge before I can take advantage of it.

Over the years, I've developed a solution to this problem. I have a substantial, but not overly complicated, program that I write whenever I come to a new platform. I write a program that assists me in finding solutions to the kind of thorny word puzzles you get from the National Puzzlers League. It requires me to open a file, read in words, and see if they match the pattern the user is searching for. No network connection. No database work. No scalability issues.

It's a problem space I understand, so I can dive in without much thought. I get a deeper knowledge of the language than I'd get from working through a tutorial's abstract problems. It forces me to probe and dig through the libraries to accomplish what I need. I don't get a thorough understanding of the language, but I get pretty far.

Overall, it's a plan that's worked well for me.

Article On EasyMock

2009-04-28T08:27:00.000-07:00

IBM's DeveloperWorks features an article about EasyMock, a framework for building mock objects for unit testing.

The Root Of All Evil

2009-04-24T11:13:00.000-07:00

In my most recent round of interviews, I chatted with the founder of U1 Technologies. He and I had a lot in common philosophically and, with a slight shift in timing, I might very well have been working for him instead of Maxis.

One of the things that caught my attention was his phone screen questions. I've been on both ends of these things, and they usually consist of simple questions that ensure that the candidate isn't obviously inflating his or her resume. Things like: "Explain how an Iterator works." or "When would you use a List versus a Map?" (Yes, I've actually been asked these.)

Nathaniel's first question reflected his company's decision to only hire senior engineers: "When should you optimize your code?" Fifteen years of software engineering provided a quick answer: "When you know there's a performance problem with it."

"Premature optimization is the root of all evil," is a quote that's been attributed to both Donald Knuth and C.A.R. Hoare. (Most people cite Knuth, who cites Hoare, who denies saying it, according to Wikipedia.)

This may sound at odds with one of my other favorite programming axioms: "The earlier you find the bug, the cheaper it is to fix." But optimizing code usually goes hand in hand with making it less maintainable and reusable. Specializing algorithms, caching values, adding different control structures, and more all add complexity. If you don't have performance numbers to back up that work, you're adding the complexity and (probably) not getting any benefit from it.

You may have inefficient code that rarely gets called. You may think something's inefficient when in fact the compiler or optimizer or processor does just fine with it. Or your code could actually be inefficient and it might just not matter enough to affect the user.

I've seen this in practice for years. And yet the temptation to optimize is always there, coaxing even the best programmers into a fen of tangles and mazes and confusion.

While I was working on the "real time time update" feature of my reusable date picker pop-up, I set it up so that the delegate would be called for every UIControlEventValueChanged message from the UIDatePicker.

I envisioned the date and time changing as the wheels in the date picker spun to their new locations, so I decided to be smart about calling the delegate method.

I could have written this in my event-handling method:


    if ([delegate respondsToSelector:@selector(dateChangedWithoutSaving:)]) {
           [delegate dateChangedWithoutSaving: date];
   }

But I assumed that respondsToSelector would be a relatively expensive call. I also assumed that the runtime binding to dateChangedWithoutSaving: method would be costly. And I didn't want that cost slowing down the user's interaction with the date picker wheels. Instead, I made a boolean in the controller that tracked whether or not the delegate actually supported the method and set it at the same time I stored the reference to the delegate:


    - (void) setDelegate: (id) delegate {
         ...

          delegateRespondsToContinuousDateChanged = [delegate respondsToSelector:@selector(dateChangedWithoutSaving:)];
    }

I changed my event-handling logic accordingly:


    if (delegateRespondsToContinuousDateChanged) {
        [delegate dateChangedWithoutSaving:[myDatePicker date]];
    }

Then I fired up my app.

And discovered something I didn't know. That UIControlEventValueChanged message only gets sent once the date picker wheels have finished spinning. Rather than sending a message for every number that cruises by, it only sends, essentially, one message per gesture. My optimization didn't buy me much, since the code can't possibly interfere with the spinning wheel animation.

So what did my "clever" optimization get me?

My own implementation of setDelegate (rather than the one created by Objective C's synthesize keyword)

An extra layer of logic in my event-handling code.

I've added more complexity (the extra layer of logic) and more lines of code (the setDelegate method) with the concurrent possibility of bugs.* Performance gain: virtually none.

Premature optimization is the root of all evil.

* Yes, of course, this isn't much extra complexity or code. But add up enough items like this, and you get a significant amount of technical debt

Delegation And Redesign

2009-04-22T23:19:00.000-07:00

One side effect of programming for the iPhone (or Mac OS X) is a heightened awareness of the delegation pattern. It seems like every class in the framework has delegate information.

When I started my UI redesign work in my iPhone app, I saw a perfect opportunity for implementing my own delegate pattern.

The idea behind delegation is straightforward. Object A holds a reference to Object B. Object A goes about its merry way performing its duties, except that every now and then it says to Object B, "Hey, I need you to handle this, okay?" To everyone else, it looks like Object A did the work. A good example is the UITableview in the iPhone framework. It supports two delegates, one for supplying the table's data, one for handling the table's behavior. The data delegate has methods such as numberOfSectionsInTableView; the behavior delegate has methods such as tableView:didSelectRowAtPath:. The table view doesn't care about anything else those objects might do: It just expects them to do what it needs.

The pattern encourages code reuse (I have a single instance of a delegate that handles common behavior for all of my text fields) and separates business logic from other layers (I have specialized objects that act as my tableview data sources, leveraging a model object in different ways, all without modifying UITableView in any way). It lets you make complex object compositions from subsystems. It makes breakfast for you in the morning. It's great. (As with all general-purpose design patterns, you may need to consider performance and threading issues when putting it into practice.)

In Objective C, it's easy to use a delegate:


       id delegate; // set by an outside client
      [delegate invokeDelegateMethod]

Even though the compiler doesn't know if delegate implements invokeDelegateMethod, it compiles this code with only a warning. (id is shorthand for "an otherwise untyped object.") Of course, if you try and run this, and delegate does not implement invokeDelegateMethod, you'll get a runtime exception. This may be what you want, but you can also use:


     id delegate;
     if ([delegate respondsToSelector:@selector(invokeDelegateMethod)]) {
         [delegate invokeDelegateMethod];
     }

This makes the method optional. You can be formal and specify a protocol that the object must declare to get yourself some type safety, but it's not strictly necessary. Java has similar "rummage around in an object's belly and see what it contains" tools in its reflection API, so you could easily hook up delegate patterns the same way in that language.

For my redesign, I had several places where I needed to pop up a view that had a UIDatePicker, a Done button, and a Cancel button. The idea being that the user needs to pick a date and tell me when s/he's done. The delegate pattern seemed to be the perfect approach, since I had one class that I needed to reuse throughout the application and have it fit seamlessly into the flow of each area.

In my first pass, I implemented two delegate methods, dateSaved: (NSDate *)date and datePickerViewDismissed.

I made the first one optional: Why should the date choosing view care whether the client actually wants to know about the date that was chosen? I made the second method required (i.e., I don't check to see if the delegate supports the method before calling) because that's the method that triggers the client to release any memory it used for the view. I wanted to make sure that I'd know right away if I forgot to implement that method.

I wired the view into the first use case: A screen that shows one date at a time would let the user jump to any arbitrary day, rather than nexting and previousing a bunch. Everything worked fine, so I started to wire it into the next part of the app, where you set the time for a single task. That's when I had the eye candy idea for my ScheduleButton. With a bit more work, I could make a delegate method for the date picker view that would fire as the date changed, which meant that I could update the button's text as the user spun the wheels.

My new version added a new delegate method, dateChangedWithoutSaving: (NSDate *)date. But what if the user cancels? My architecture didn't support that, so I changed the datePickerViewDismissed method into datePickerViewDismissedWithButton: (DateChooserButton)btn (DateChooserButton being an enum representing the two choices). Now a client could grab a copy of the date, bring up the view, update the ScheduleButton helper object as the date changed, and then reset it to the original date if the user pressed cancel.

Enhancing UIButton. Or Not.

2009-04-21T19:52:00.000-07:00

As part of a UI redesign in my iPhone app, I realized I had a few concepts that would be repeated in various places, so of course I wanted to bundle them into reusable components.

One idea was a button whose text would always read "<some fixed prefix>: <some mutable date>." The date changes as the containing view is visible, but the prefix is fixed early in the button's lifecycle.

I made a new class, called ScheduleButton, that subclassed UIButton. I added methods for setting the date format, the actual date, and the prefix. These call down into a single "updateButtonText" method that constructs the appropriate string and would have used the inherited UIButton methods to change the text.

So far, so good.

I dragged a UIButton object from Interface Builder into my app, and set the class to ScheduleButton (instead of UIButton, the default). I launched my app and saw the button,but without a border.

I could probably figure out the reason for this. I suspect it has something to do with the fact that UIButtons are made with a class method, buttonForType, rather than a normal alloc/initWithFrame flow (at least from a client's perspective: I imagine initWithFrame gets called in there somewhere).

But after hitting my head against several other issues during this redesign, I pretty much just said, "Fuck it." (Did I mention that I'd swear on this blog? Well, I will.) Instead, I converted ScheduleButton into a helper class that holds a reference to a UIButton. The controller still interacts with ScheduleButton, using the same methods, and they feed into one centralized method, but that one method now invokes setTitle on the contained UIButton rather than inheriting it. That means that other objects can muck with the UIButton on their own, without my helper being the wiser, though that's not likely to be a problem since I'm the sole user. (There's probably yet another solution involving Objective-C's category functionality, but I don't want to modify every instance of the UIButton class in my application: I just want to modify particular objects.)

I tried to lend credence to this implementation by convincing myself that this was an instance of the Decorator design pattern, but it's pretty clearly not. A proper Decorator shares a parent with the decorated object and implements every method by calling into the (completely contained) decorated object. It then adds extra functionality that the decorated object doesn't know about.

Ah, well. Sometimes you just need something to work.

Evolutionary Architecture and Emergent Design

2009-04-21T12:01:00.000-07:00

IBM's DeveloperWorks has an interesting series on finding the design inherent in systems through test-driven development and other strategies.

Constant Refactoring: A Practical Benefit

2009-04-19T14:14:00.000-07:00

As I program, I continually refactor the code I'm in. It's habit at this point. I rarely get the opportunity to do a massive change, but even the basic work I do — removing duplicated code, generalizing logic, dividing up large functions, and so on — reaps unexpected benefits.

Refactoring partly makes code more maintainable, but its main purpose is to make the code flexible enough to respond to new design requirements. I try to code on the assumption that change will come, but if I could predict the exact direction of every future change and thus craft a perfect design, I'd be writing this from my French Chateau.

Here's a simple example I recently found in my iPhone app.

One of the tabs lets you move between dates, one at a time, to see your schedule for that particular date. In my initial design, I didn't let you change dates because that view exists to show you what's going on today. But I quickly realized that I wanted to at least see what tomorrow would bring.

When I made that change, I had two buttons that would switch to either the next date or the previous date. They each called separate methods — nextDay and previousDay — that were fairly similar to each other. I was working through the basic functionality, so I didn't register the commonality. I think this is common enough: You're so focused on one thing that you don't notice something else.

On a later pass through the code, fixing something else, I realized how similar the methods were, and I wrote a new method that would just adjust the date based on some time interval. I gutted nextDay and previousDay and replaced them with calls to the new method that passed either 86400 or -86400. (The number of seconds in a day.)

After that, one of my testers suggested that I let the user swipe across the date to move to the next or previous day, depending on the direction. Once I figured out the gesture system for the iPhone, I just made each swipe call either of the two appropriate methods, which in turn called into that common method.

But then I realized that I needed to add animation to show the user what was happening. I put the animation logic into the generalized method and made it smart enough to know that a negative number meant animate one way, while a positive number meant animate the other way (animate from the left in the first case, because you're moving to the previous day, and animate from the right in the second). That handled the animation for the two cases.

A couple of my testers asked for the ability to jump to an arbitrary day. I decided to look into implementing that this morning, and once I opened the code, I realized it would be a snap. I bring up a UIDatePicker, the "choose from a list" UI in the iPhone, and then just pass the time interval between the user-selected date and the current date to that one common method. The logic is the same, and the code animates appropriately: If you jump to some arbitrary, far-future date, the screen wipes from right to left, as you'd expect. Aside from bringing up the date picker, I wrote exactly one line of code to enable the feature.

For this simple example, I probably would have found my way to the same end result: an all-purpose method for jumping to any given date. But the fact that I had refactored earlier — just because — made the solution obvious and quick.

When I started this feature, you couldn't switch dates at all. Piece by piece, I built the infrastructure so that now you can jump to any date with gestures and animations. That wasn't the original design, but constant refactoring allowed me to eventually support this radically new direction with a minimum of "from scratch" code.

iPhone Mail Reformatting

2009-04-19T09:20:00.000-07:00

I recently wired up the ability to email lists from an iPhone app I'm writing. Setting this up was pretty straightforward (see the Apple docs on the subject).

But since I'm sending lists, and since my list might have headings with one level of subitems, I wanted to create an email that looked like this:


Heading
   Item 1
   Item 2
   Item 3

I added the appropriate spaces to my email body, URL encoded it with NSString's stringByAddingPercentEscapesUsingEncoding method, and shipped it off to Mail with [[UIApplicaton sharedApplication] openUrl:]

What I saw when Mail started was this:


Heading
Item 1
Item 2
Item 3

I assumed my indent code didn't work, so I looked at the result in the debugger. The encoded string had the three spaces in front of each item; Mail had decided it didn't like the spaces at the beginning of the line. I tried using a tab instead. No dice.

Thinking that I could trick Mail by putting a different character at the beginning, I sent it email that should have looked like this:


Heading
-   Item 1
-   Item 2
-   Item 3

What I got was:


Heading
- Item 1
- Item 2
- Item 3

Mail not only doesn't like white space at the beginning of the line; it doesn't like multiple spaces at all.

I find this very annoying. I'm sending Mail a piece of text that I want it to use, and it's making its own decisions about that text. Mail would let me type " Item 1"; it just won't take it from a passed-in URL.

Grr. In the end, I decided to just live with the last version of the list. A friend suggested "--Item 1" but I don't like the way the dashes run into the text in that form. Maybe "-- Item 1."

Technical Debt

2009-04-19T08:02:00.000-07:00

Some time in the last year, I learned the term Technical Debt. It's a brilliant metaphor for the more hurried, less-thought-out code that crops up in any project with a ship date.

The idea is simple: Just as you actively take on financial debt to give yourself new opportunities, you take on technical debt to ship a product out the door and make money from it. Financial debt is simple money, but technical debt is more complex. Any decision you make to ship the product at the expense of a clean code base qualifies. Every test you don't write is technical debt. Every piece of duplicated code that hasn't been refactored into common functionality is technical debt. Every todo comment is technical debt. Every bug you ship with is technical debt.

But too much technical debt, and you never pay down the principal, in this case the ability to quickly add new features or fix old bugs. That duplicated code? If you never refactor it, every bug fix in that code has a cost proportional to the number of places it's used. And that's if you actually remember where all those pieces are. That todo comment you never go back to? Some day you'll need to enhance the code around it, and that todo will add to the cost of the work. Add enough of these little debts, and the cost for any given feature multiplies.

Unfortunately, while it's easy to measure financial debt, it's tough to measure technical debt, which makes it difficult to use when arguing for time to fix things up in the code base. If you never need to fix a bug in that duplicated code, for instance, you'll never experience the extra cost of fixing it in every location. Of course, the idea that a chunk of code will sit forever untouched once it's written is ludicrous, but you can't sell the idea that something might be a problem someday outside of development teams.

There's also no good way to flag technical debt. Perhaps as programmers we need a more formal way — beyond todo comments — to note down that we made a deliberate choice to take on technical debt. But then we have to advertise to the larger world that we deliberately did something not in the best possible way. I know that won't sell.

About This Blog

2009-04-19T07:53:00.001-07:00

I'm a programmer, both professionally and passionately. This blog will contain all my thoughts, explorations, and ramblings about my programming antics.

For the most part, this blog will cover my personal programming projects, but work topics will sneak in from time to time if they don't have any confidential information.

Most people would call me a Java programmer, but that's just how my career has shaken out. I consider myself an "all-purpose" programmer. The topic itself intrigues me, regardless of the particulars of any one language.