Rails Scaffolding In Java

I recently needed to add a new business object to a system I'm developing at work, and it was a bit of a drag.

If you follow any of the normal Java patterns, you probably know what I'm talking about. You need the file for your business object. You need a service class responsible for returning instances or sets of instances from the DAO layer. You need a SQL file that defines the database table the business objects will be stored in. In our case, we also have queries in resource files, so we need a file for that as well. You need to edit your Spring config files to point to the new service. And so on. Your environment probably has some overlap with mine, and probably requires pieces mine doesn't.

You can simplify a lot of this, of course. You can make a service base class with Java Generics that will give you a lot of the type-safe methods you'd want. IDEs will let you set up templates, but you'll still have to click through a few menu options to get you the files you need. And the more files, the more clicks. And I like running an IDE-neutral team, so I wouldn't want to do something specific to one workflow.

I wanted a better way. Specifically, I wanted what Rails provides. You type a command at the command line, and you get all the files you need for working with the object. (XCode offers similar functionality.)

And I thought, "Well, why not?" My build system is written in Ruby, and Ruby's ERB templating system is built in to the language.

It took about an hour to get the main system up and running. I now type buildr :biz_object object=com.ea.foo.TestTest and I get a src/main/java/com/ea/foo directory, a TestTest.java and TestTestService.java file in that same directory, a sql file with the table definition (named test_test), and an empty query file. I also get a perforce changelist (via p4Ruby) with the description filled in and all the new files added.

Here's the heart of the code. template_to_final is a hash of template file name locations to the end file destination. The local variables exposed by the call to binding include the package name, the Java object name, and the SQL-friendly name:

       template_to_final.keys.each do |key|
           File.open(key) do |file|
               b = binding
               erb = ERB.new(file.read)
               outfile = File.new(template_to_final[key],"w")
               puts "Creating #{outfile.path}"
               outfile.write(erb.result(b))
               outfile.close
           end
       end

To give you an idea of what the templates look like, here's some code from the java file templates:

package <%=java_pkg%>;

@Entity
@Table(name="<%=java_sql_name%>")
@SequenceGenerator(name="objIdGenerator",sequenceName="object_id_seq",allocationSize=1)
public class <%=java_obj%> {

I don't yet go the full Rails route and specify all the properties on the command line, but this takes care of getting the boring parts of business object development out of the way, enforcing consistent naming schemes, and ensuring that the developer doesn't forget to check in some file. I also don't yet modify my config files, but that won't be too tough to add.

Scripting Campfire

My team uses Campfire, a web-based chat tool from 37signals, to communicate throughout the day. We've divided it into various rooms, one of which is a Status room where we do a virtual version of the daily stand-up that many teams do: a quick meeting where everyone says what they're working on. (One of our team members is in England, and our hours vary a bit, so a classic stand-up isn't practical.)

About a month ago, one team member started posting the date in the status room before anyone gave their update. For some reason, Campfire itself doesn't do this reliably, and when you have a room with nothing but status updates, it's not always clear where one day's messages end and another's begin.

Usually one or two people on the team did this, but I followed suit on a couple of occasions, and of course thought about automating it. We already had a bot account for some very early automation, and Campfire has a good web service. Insert Tab A into Slot B.

Here's the relevant Ruby source code, which I have hooked up to a cron job (Note that if you want to script Campfire, I suggest creating a "Bot Test Room" that you can use for experiments without spamming your real rooms):

require 'date'
require 'json'
require 'net/http'
require 'net/https'
require 'uri'

def send_text_to_campfire(text)
    message = {
       :message => {
           :type => "TextMessage",
           :body => text
       }
    }
    send_to_campfire(message)
end

def send_to_campfire(message)
    url = URI.parse("https://<your base URL>/room/<your room number>/speak.json")

    request = Net::HTTP::Post.new(url.path)
    request.basic_auth(<your auth token>,<any password string>)
    request.content_type = 'application/json'
    request.body = message.to_json
    request.content_length = request.body.length

    http = Net::HTTP.new(url.host,url.port)
    http.use_ssl = true
    response = http.start do |http|
        http.request(request)
    end

    puts response.body.to_s
end

# construct message
today = Date::today
formattedDateString = sprintf("%02d/%02d/%4d",today.mon,today.mday,today.year)
dateStatusString = "=== Today is #{Date::DAYNAMES[today.wday]}, #{formattedDateString} ==="

#send status message to campfire
send_text_to_campfire(dateStatusString)

Filling in the date each day isn't a huge time savings: It will take us a lot of days to recoup the time I spent automating a five-second typing task. But once I figured out the gist of posting to Campfire, I started adding new functionality. Our "Today is ..." status message now includes a few choice statistics — gleaned from our telemetry system — about gameplay from the previous day. I also wrote a "canary" script that does a health check on our dev server and posts to our general chat room if it seems to be slow.

Naturally, some of my co-workers have suggested writing an adventure game on top of the API. That may be a bit silly, but it does emphasize a point I often make: You can't really imagine all the possibilities for a technology until you get your hands dirty a bit and play with it.

This whole experience underlines again why web applications should have APIs. The lack of a date is probably a Campfire bug, but we don't have to wait for them to fix it. And we've added functionality that is only relevant for our team, resulting in something that more closely ties in with our real needs.

Watch Your Users Use

Time and time again, I re-learn this lesson: Nothing makes your system more usable than watching your users use it.

As I mentioned in my last post, I'm building a telemetry system on top of Google AppEngine so that my studio can understand how our games will be played. As I've demoed pieces of it, I've put together web pages and interactive charts and other nice little visualizations. I've made JavaScript libraries to make it easy to build those sorts of things, and I've demoed some of the prettier ones to the stable.

But I also put in a way to export data as tab-delimited fields. I figured this would be a good last resort if the JavaScript version wasn't quite there yet, or if we hadn't written a specialized view on the data, but I figured we'd favor those.

Naturally, then, my principal users have all just used the tab-delimited export and not pushed beyond that. In particular, they grab a bunch of tab-delimited data, import it into Excel, and then do manipulations on it there. Though there are macros and various other things, it still seems to take a senior engineer a couple of hours to compile all the data for his weekly reports.

I know this because I sit with him a fair amount as he explains his process.

The other day, I was thinking of his process and realized that Python can create Excel spreadsheets. So I could remove one minor barrier by just letting him get the data in Excel format instead of tab-delimited text. Easy.

But that doesn't buy him much. For his weekly reports, he trims some of the unneccesary columns (my exports grab all the fields and dump them out, including some that are system-level fields), runs some macros to insert formulas to give him percentages, sorts the data, and then copies and pastes that into an email. He does that for about 10-15 reports, in one form or another.

What if, I thought, I could let him define the format of the spreadsheet the system generates? Then he could say, "give me this data, but put it in this layout." That would shave a big chunk of time from his flow.

One thing about working with him is that he's not an online engineer. This turns out to be very good, because it makes me think in terms of letting him modify config files instead of modifying server code. (A medium-term goal is building a real interface on top of my system so that anyone in the studio can interact with it, but for now, config files are how we do things.) One of our other online engineers used the system by writing a custom request handler that interacted with the AppEngine datastore directly to generate a customized HTML page. Very neat, but if he were my only user, my system wouldn't be very evolved. I'd just say to someone new, "write a request handler."

I mentally sketched out what the config file would need to contain, what abilities it would need to give him, and within 45 minutes had a system in place that will let him generate something much closer to the final spreadsheets he needs for his reports.

A key component of the new feature is the Django templating language that comes with AppEngine. I actually dislike the language for HTML generation, but for letting users construct simple templates, it's pretty good. One important feature: It can render a template contained in a string, not just one contained in a file. And you can give the renderer a context, filling in some variables that are accessible within the template. In my case, I create a context that contains the current piece of aggregation data (which is a summary object of a large number of events) and the current row number.

From his perspective, to generate a report about how much damage a creature is doing in our testing, he creates a YAML config file that looks like this:

header:
    - value: Name
    - value: Avg Damage
    - value: Total Damage
row:
    - value: "{{aggregation.groupByValue}}"
    - value: "{{aggregation.average}}"
    - value: "{{aggregation.sum}}"

Then, when he requests data in Excel format, he can tell it to go through the layout he specifies. Rather than getting a dump of every piece of data in each object, he gets a spreadsheet with just the information he needs. The {{}} is part of the templating system and translates as "spit out the result of this expression."

Let's say he also wanted to capture the number of times that creature did damage. That's actually available to him in another field in the aggregation, but let's say it wasn't. My system also supports formulas. So he could do this:

header:
    - value: Name
    - value: Count of Attacks
    - value: Avg Damage
    - value: Total Damage
row:
    - value: "{{aggregation.groupByValue}}"
    - formula:"ROUND(D{{row_num}} / C{{row_num}},0)"
    - value: "{{aggregation.average}}"
    - value: "{{aggregation.sum}}"

That formula will become, in the output for the first row, ROUND(D1/C1,0).

This is still a long way from user-friendly, but I think it's going to give him back an hour and a half of time each week. And building the system this way means that a user-friendly version just needs to write the format into this config file, and away it will go.