Internet Radio Podcast, the Raspberry Pi Edition

I recently bought a Raspberry Pi*, the small-and-cheap-but-powerful computer taking the geek world by storm. The hope is that someday I can use this little board or its successor to teach my baby daughter more about my own interests. But given my current time constraints, I figured I should get one now to ensure I've done at least one significant project with it by the time she's old enough to grok what this thing is.

That's right; I think I can do one useful project with my Pi in the next ten years.

People go crazy with their Pis: rigging up their microwaves to cook food according to the bar code on it, taking pictures from very very high in the sky, and building souped-up irrigation systems that would be the envy of pot farmers everywhere.

I do have ambitious goals in mind, but I wanted to start with something a little simpler. Some of you may remember my Rube-Goldberg-esque "Radio Broadcast Podcast." An obvious failing of my original setup was that it required me to leave my MacBook running. Predictably, this has turned out to add enough friction that the recordings don't get made.

At the time, commenters offered suggestions to fix that obstacle. One was to set up an EC2 server that could run all the time. A good idea, and I'm certainly comfortable with EC2, but I never got around to it. Between launching a big game and having a baby, the priority on the whole endeavor kept dropping. Still another suggestion was to just buy a server for the home. A very good idea, but we didn't have space for it even before we had a baby.

Enter the Raspberry Pi. Since it's just a normal computer that runs Linux, it can also be a server. Not a super-powerful one, but I don't need a super-powerful server for my house. And it's hard to argue with the physical footprint: the Pi and its case are roughly the same size as two or three stacked decks of cards. I just tucked it into a small space behind our TV.

I decided to start from scratch on my radio-to-podcast scripts, mostly because I wanted to fix some problematic and error-prone behaviors from version 1. It ended up being a few independent parts:

1. A Python script runs every five minutes as a cron job and uses a config file to determine if it should be capturing streaming audio. If yes, and streamripper isn't running, it starts the program, figuring out a file name based on parameters in the config file (including adding numbers if a file of that name already exists.)
2. Another cron job looks in the download directory for any mp3 files that haven't been touched in 5 minutes, uploads any it finds to S3 via s3cmd, and then deletes them.
3. Yet another cron job creates a podcast-compliant XML file by getting a listing of items on S3, running them through an awk script that generates the necessary XML, and then taking that output and uploading it to S3.

This may still seem Rube-Goldbergian, but it's designed to be more resilient than its previous incarnation. If streamripper crashes or the network goes out, my script will ensure that the music starts recording at the next opportunity and it won't overwrite the first part. If the Pi can't get to S3 for some reason, the files will stay in place until needed. There's no cleverness around updating the podcast xml file; it's created from scratch every time. (Note that doing a bucket listing on S3 very much exposes you to eventual consistency issues, but even a day's worth of latency is probably fine for my purposes.) 

The onus is still on me to clear old files from S3, but that's something I can do every few months. And once I do, the XML file will reflect that within an hour.

The next step is to run an HTTP server on the Pi so that I can log in and make configuration changes from any of the devices on the network instead of sshing in and modifying the config file directly. And after that? I'm tempted to add AirPlay capability so that if I happen to be home and awake, I can send the current download to the living room speakers.

Overall, I'm pretty impressed with the Pi's capabilities. I doubt I can pile too much onto it, but I'm curious to see what its actual limits are as a household server. Future projects involve using its ability to talk to other electronic components, which is one area where the Pi shines.

* I bought the Maker Shed Starter Kit. It comes with a good book and most of the stuff you'll need to get going and do the book's exercises.

All Together Now: A Look at Concurrent Languages

Over the past several months, I've been taking a look at various languages that advertise easy concurrency and scalability. It's too late to use any of them on SimCity, but I'm always thinking about what I'll build next and how I'll build it, and these languages are on my radar. Java is increasingly cumbersome to me as a language, and its concurrency constructs are too error-prone even for senior programmers, not to mention that the thread model Java exposes has serious performance issues if not carefully managed.

Here's my quick take on the three languages I focused on: Erlang, Scala, and Go. The TL;DR version: I'd use Erlang for infrastructure in a heartbeat, Scala will make your developers more productive at the possible expense of application performance, and Go is fast but less fun to work in.

For each language, I worked through at least one book on the subject and then built something for myself. The personal projects ranged from small to sizable.

Erlang

Erlang was my favorite of the three languages, and it would be hard to ever argue against using Erlang for back-end infrastructure pieces that require high scalability. Say, message queues, or chat systems, or NoSQL databases, or the backbones of prominent first-person shooters.

Like all these languages, it has a high-level abstraction for concurrency, but, unlike the others, it easily supports passing messages between machines, which bodes well for a cluster of servers. It has extensive fault tolerance mechanisms, even across machines, allowing for robust systems. It has support for hot code swapping, opening the possibility of upgrading a system while it's still live and reducing maintenance windows to nil. It has great support for extracting values out of binary data, which is invaluable when dealing with network traffic and proprietary data formats. It's a mature, proven technology. And it has the benefits of functional programming: more concise code that reduces the number of potential bugs and immutable objects that prevent weird thread-safety issues.

On the other hand, I can probably count on one hand the number of other Erlang programmers I've met. And it's not like taking a C programmer and teaching them Java; functional programming is a distinct mental shift from imperative programming, and it can be hard to get your head around it. That means that you can write your Erlang code all you want, but what about the people who will have to maintain your system beyond you? Its small community also means that while there are certainly lots of third-party libraries for it, it's not the vast universe that Java enjoys. And while immutable objects are easy to work with, they're also expensive because new ones are constantly being made.

Scala

While Twitter's Scala School argues for treating Scala as a separate language, it's hard not to compare it to Java, since it compiles down to Java bytecode and runs on the same virtual machine. And in terms of developer productivity, Scala rockets past Java in my book.

As an application layer language, it has tremendous advantages. You can accomplish complicated tasks with much fewer keystrokes. You can use functional paradigms and immutable objects, but also use imperative style and mutable objects if you need performance, or, crucially, if you need to bring another developer on board with your system. You can enjoy the same concurrency abstraction Erlang provides. You can leverage Java's seemingly infinite supply of open-source libraries. You can incorporate it into an existing Java application, giving you the ability to bring it in without rewriting everything. You can even easily build internal DSLs with it to make your system more expressive and easier to maintain.

But, in my own experiments and in anecdotal evidence, it suffers from sluggish performance. All that great functionality makes your developers more productive, but potentially at the expense of speed. This makes sense; all that pretty code needs to be contorted and converted into Java with who-knows-how-many object creations along the way. Obviously Twitter and Foursquare manage to be fairly fast, but how much engineering time is spent to get them there? On the other hand, a system that enjoys greater and easier concurrency than Java might be more scalable and have more consistent performance under load, even if any given call could be faster in another language.

Go

Go is Google's attempt to build a better C, with a focus on developing distributed systems at a scale that Google needs. Its concurrency model is distinct from Erlang and Scala's, preferring the notion of Communicating Sequential Processes to the Actor model, but neither is particularly superior; each has strengths and weaknesses that fit different situations.

The big win with Go is its speed: Go programs are compiled down to machine code. And while the built-in garbage collection probably means that C would win a horse race between the two, Go is a much less error-prone language to work in. Its community is still young, but it seems eager to improve the language, and a wide variety of useful go libraries already exist. It's hard to compete with the many years of robust Java libraries out there, but Go nuts seem to have filled in the most obvious needs.

I have to admit that I dislike working in the language itself; it lacks the cleanliness of Erlang and the depth of Scala. But there's no denying that its concurrency model is easy to work with, and the programs that you create are nice and zippy relative to their Java counterparts.

Others

I've yet to dig deep into Clojure, though it's the obvious next one. I figure if I'm going to be a fan of functional programming, I might as well go into crazy Lisp land. But I'd worry that it would suffer from the same performance problems — for the same reasons — as Scala.

It seems funny to mention node.js in this post, since in some ways it's all about zero concurrency: a single thread of execution is all you get. Of course, under the hood there's lots of asynchronous work, but it's tied directly to the operating system's I/O. We use it for a subsystem in SimCity, and it, like everything, has strengths and weaknesses. It can do lots of I/O tasks concurrently. Lots. But it's very sensitive to slow code, since that code will block the entire thread when it runs. It appeals to the game developers on my team, since single threads, event loops, and performance-sensitive code are the norm for them.

However, it's not very mature, and the libraries for it can be buggy and incomplete at this stage. I think we made the right decision switching our SimCity subsystem to node.js, since it outperforms its Java predecessor by a long shot, but it hasn't been simple or without issues.

Hirsute: Fake Data for the Real World

At a previous job, which sold web services to help school districts track their tests and their progress against state tests, the sales team thought they could be more effective if our demo district was more realistic. The demo district, which had survived years of service, only had a few hundred students. Superintendents wanted to get a sense of what the system could really do for their needs.

You couldn't just show a district some other district's data, though, for confidentiality reasons. So one of our engineers came up with a solution. He took data from other districts and munged it together. He didn't just swap names around, though. Because we tracked demographic information, he squished students together within demographics. So you'd end up with Hispanic names together. And those students would have test scores that mirrored the Hispanic population in your district (since, at that point, we probably had data from nearby districts in your state). Kids of different socioeconomic status would also have similar test scores, and so forth. I seem to remember that one district half-jokingly suggested they just run the district off the demo they saw, since it was so close to their own.

I thought of that recently when working with our loadtesting group. I'm sure EA's centralized loadtesting group is no different than that of any other corporation in that they lack intimate details about our business objects. On Spore, I remember that our loadtesting database was set up with something like 100,000 users (we have 3 million or so now), each of whom had something like 10,000 creations (most users probably have 20). Or all the sporecasts had 5,000 items (most held somewhere in the few tens of creations). Early work on SimCity's system produced similarly unrealistic data. That kind of data makes it hard to tune queries and indexes, figure out caching strategies, and all the other normal stuff one needs to do with data for a system.

What I wanted, I thought, was a DSL that would let me do what we did for school districts: Specify how the data should kind of look, and let the system generate that data for me.

Thus, Hirsute was born. Hirsute is an internal DSL built on top of Ruby and its extensive metaprogramming facilities (an aside: I recommend Metaprogramming Ruby for its solid information, though most emphatically not for its trumped-up dialog and narrative structure). In Hirsute, you build templates that define how objects should look, and then you create collections of objects derived from those templates. You can specify histograms of probabilities so that you don't just get a random distribution among options but a distribution that reflects your real-world requirements. You can read in data from files and randomly combine them to get plausible composite strings. You can then flush the collections out to files ready-made for loading into a database (mysql at the moment).

For instance, here's some Hirsute code from a fictional wine-cellar-management service that I created as a sample (since the data requirements most on my mind are for SimCity, which I can't talk about). There's also a manual that goes into greater detail.

# This defines a tasting note that a single user might write about a single bottle. It pulls descriptors from various files.
a('tastingNote') {
  has :tasting_note_id => counter(1),
      :description => combination(
         subset(
           read_from_file('wine_cellar_aromas.txt') {|text| text + ","},
           read_from_file('wine_cellar_aromas.txt') {|text| text + ","},
           read_from_file('wine_cellar_aromas.txt') {|text| text + ","},
           read_from_file('wine_cellar_aromas.txt') {|text| text + ","},
           read_from_file('wine_cellar_aromas.txt') {|text| text + ","},
           read_from_file('wine_cellar_aromas.txt') {|text| text + ","}),
         subset(
           read_from_file('wine_cellar_flavors.txt') {|text| text + ","},
           read_from_file('wine_cellar_flavors.txt') {|text| text + ","},
           read_from_file('wine_cellar_flavors.txt') {|text| text + ","})
         ),
      :rating => one_of([1,2,3,4,5],[0.1,0.1,0.4,0.3,0.1]),
      :bottle_id => 1, # filled in later
      :user_id => 1    # filled in later
    is_stored_in 'tasting_note'
      
}

tastingNotes = collection_of tastingNote

That sample defines a template for tasting notes. The description field comprises 1 to 6 lines pulled randomly from a file of aromas combined with 1-3 lines randomly read from a file containing wine flavors, all joined to one another with commas. The rating is from 1 to 5, but weighted such that most wines will have either a 3 or a 4. The tasting_note_id is a counter that's incremented with each new object. The bottle_id and user_id fields are filled in later when an actual tasting note is created.

Then you define a collection to hold them. You could also do this by using

   tastingNotes = tastingNote * 6000

Which would create 6,000 tasting note objects based on the formula you provide.

So far, the system is pretty simple, but it gets the job done. And because it's a Ruby DSL, you can always just write raw Ruby to fill in what you need. I definitely plan to keep adding to it, though, with a newborn in the house, maybe not quite yet.

Functional Programming For The Imperative Mind

I've been learning erlang on the side lately, largely because of its increasing role in the universe of scalable systems. I know of a number of highly-scalable pieces of software that are written in erlang. But I also think that shifting programming paradigms every now and then is a good mental exercise.

I've mentioned before that I often write word puzzle solvers (for puzzles you might find in the National Puzzlers' League newsletter) when learning a new language. It's a problem I understand, it doesn't involve a lot of error handling or user interface, and you have to think a little about scale because wordlists can be reasonably big. I'm doing the same with erlang, but I'm focusing on writing a cryptogram solver.

Getting the gist of the language wasn't too difficult, but I struggle with thinking in functional terms. How do I take a problem that I know how to solve in an imperative language such as Java and translate it into functional patterns? It's fine that one can code a Fibonnaci sequence so quickly, but how often do you need to do that in a production system?

I asked some functional programming fans at work, and they gave me some ideas.  "Functional Programming for the Rest of Us" is a six-year-old post that recently made the rounds on Hacker News. It trumpets the benefits of these languages but didn't help me with my problem. Various books on algorithm design were also suggested, because functional programming is really derived from lambda calculus. While I'm looking at those, that obviously increases the friction to learning. It's not as if I'm writing business-critical software in erlang yet, so increasing the work to learn it seems like a good way to let it fall by the wayside.

I still haven't found a good single resource for my not-so-unique situation, but I've begun to piece together a strategy based on bits here and there. The paper "Why Functional Programming?" got me focused on decomposing programs into smaller and smaller bits. That's a good strategy in imperative programming, too, but I've found with functional programming that I need to brunoise my ideas, not just dice them. O'Reilly's Erlang Programming articulated something that I had to begun to come to myself: accumulators. It's hard for me to think of building up a list from the end out, but it turns out I can easily picture how to do it by passing values down through the recursive calls and then returning that list of values when I'm at the end of the chain.

That technique has a benefit: You often naturally end up with tail-recursive code. But I worry it's becoming too much of a crutch for me. Because I can easily think in those terms, it's the technique I often use. I write a "pretty" function for clients of the code, and that largely calls down to a function that uses accumulators to compose a return value.

Are there other good resources for programmers in my situation?

Radio Station Playlist Data Visualization, Part 2

As soon as I did my visualization of 99.7's music selection for a week, I asked the obvious next question: How does 99.7 compare to other "adult contemporary" radio stations?

There's an interactive version that lets you drill down into the graph, but here's a screenshot.

People listen to radio stations for all sorts of reasons, of course, so I don't know that anyone actually cares about this. But it did give me a chance to look at Protovis and compare it to Processing as I learn about data visualization toolkits.

Gathering Data

When I gathered data for my first visualization, I wrote a simple script that grabbed songs from the 99.7 website. I set that up as a cron job on an EC2 instance and let it go.

I did the same thing for the other four radio stations I decided to look at. 97.3 uses the same website tech as 99.7, and KFOG and KBAY share a different website tech, so those got me two stations for the price of one. 101.3 uses yet another system. Once I had my scripts running, I just had to wait until I had the same week's worth of data from all stations. A bit of cleanup on the data, a quick change to JSON from comma-separated values, and I was ready to go.

I decided to use the concept of small multiples to provide a quick comparison between stations, but then showing an enlarged version for deeper exploration. Each small graph in the chart represents one station across the same span of time.

Protovis Vs. Processing

It took me some time to learn Protovis. I feel that only now, after finishing one visualization, do I really have a grasp on how it works. It seeks to be a declarative language, which means that you define the result and let the under-the-hood bits figure out how to get you there, but I found myself struggling against the lack of control.

Processing gives you that control. You have vast amounts of control, but that's because it starts you with a blank slate. You can probably do anything you want, but the flip side is that you have to do everything you want.

But Processing comes with a strong disadvantage: It creates Java applets. Remember those? I barely do, and I was actually writing Java when that's all people did with it. An applet takes a long time to load in a world where website visitors are accustomed to instant gratification from your page. An applet also won't work on your iOS device. So my first visualization was completely unusable by iPad owners.

(Yes, there is Processing.js, but my attempts to use it only frustrated me. It didn't support Java generics, and even when I removed them from my code, it failed with cryptic errors that were impossible to debug.)

As with so many things, deciding on a visualization toolkit means figuring out what's best for your job. If you're doing something complex and custom, you'll probably want Processing. But for a lot of web-based visualizations, I think Protovis will give you what you need once you figure out how to use it. It can certainly do a lot in that space.

I have still more visualizations in mind for this same set of data, and I'm planning on starting with Protovis (or its successor, d3). The Java applet problems are too big.

Menu For Hope Source

For the past few years, I've run the raffle program that generates the winners for Menu For Hope, an annual fundraising event started by Pim that incorporates food bloggers from around the world and raises tons of money for hunger relief.

It's been a couple of years since I first posted the source code, so I thought I'd post the current version. It hasn't changed much: I made the parsing a bit smarter and added support for giving people tickets even if they asked for fewer tickets than they bought (I used to skip those as invalid). Looking at it now, I see a number of ways it could be cleaner and better architected (can you say regex?), but it continues to work. If you see any bugs, feel free to mention them!

Keep in mind that I don't run the program blindly: I run it and then look over every line of output to make sure the program is behaving the way I expect. If it's not, or I see an error, I usually end up fixing some minor data issue (People often think O is the same as 0, for some reason, or they write UW01x2 UW02x3 instead of 2xUW01, 3xUW02. My program handles the latter but not the former because it attaches the 2 to the second string in the field instead of the first.)

Here's the main class.

public class MFHRaffle {


public static void main(String[] args) {
List prizeCodes = new ArrayList();
try {
RandomAccessFile prizeFile =
new RandomAccessFile("prizeFile.txt","r");
String curPrize;
while ((curPrize = prizeFile.readLine()) != null) {
prizeCodes.add(curPrize.trim());
System.out.println("Added prize code: " + curPrize);
}
} catch (IOException ioe) {
System.err.println("Error reading prize code file");
ioe.printStackTrace();
}
// divide args into buckets
// there's only one input arg (file) so we can size list in advance
List commandArgs = new ArrayList(args.length - 1);
Map params = new HashMap();
String filename = null;
for (int i = 0; i < args.length; i++) {
if (args[i].startsWith("-")) {
commandArgs.add(args[i].substring(1,args[i].length()));
if (args[i].startsWith("-oneprize") ) {
params.put("oneprize",args[i+1]);
}
} else {
filename = args[i];
}
}
System.out.println("Using file: " + filename);

if (commandArgs.contains("testrandomdraw")) {
testRandomDraw();
}

boolean debug = true;
if (commandArgs.contains("debug")) {
debug = true;
}

MFHDataParser parser = null;
if (commandArgs.contains("csv")) {
parser = new CSVDataParser();
} else {
parser = new ExcelDataParser();
}
parser.setDebug(debug);
parser.setValidPrizes(prizeCodes);

Map<String,List<String>> entries = parser.extractData(filename);
Map<String,Integer> prizeCounts = new HashMap<String,Integer>();
Map<String,String> prizeToWinner = new HashMap<String,String>();

//produce a sorted list
List<String> sortedPrizes = new ArrayList<String>(entries.keySet());
Collections.sort(sortedPrizes);

System.out.println("*******************************");
if (!commandArgs.contains("oneprize")) {
// for every entry in map, throw list to randomDraw
for (Iterator<String> prizeIt = sortedPrizes.iterator();
prizeIt.hasNext();) {
// drumroll please...
String prize = prizeIt.next();
List<String> bidders= entries.get(prize);
prizeCounts.put(prize,new Integer(bidders.size() * 10));

String winnerEmail = randomDraw(bidders);
prizeToWinner.put(prize,winnerEmail);
}
} else {
String prizeCode = (String)params.get("oneprize");
String winnerEmail = randomDraw(entries.get(prizeCode));
prizeToWinner.put(prizeCode,winnerEmail);
}


// tab-delimited output for fatemeh
System.out.println("********** TEXT ****************");
for (Iterator<String> prizeIt = sortedPrizes.iterator();
prizeIt.hasNext();) {
String prize = prizeIt.next();
System.out.println(prize+"\t$"+prizeCounts.get(prize)+"\t"+
parser.getNameForEmail(prizeToWinner.get(prize)) +"\t"+
prizeToWinner.get(prize));
}

// html markup for brett
System.out.println("*********************************");
System.out.println("********** HTML *****************");
System.out.println("<table rules=\"rows\" >");
for (Iterator<String> prizeIt = sortedPrizes.iterator();
prizeIt.hasNext();) {
String prize = prizeIt.next();
System.out.println("<tr><td>"+prize+"</td><td>$"+
prizeCounts.get(prize) + "</td><td>" +
parser.getNameForEmail(prizeToWinner.get(prize)) +
"</td><td>" + prizeToWinner.get(prize) +
"</td></tr>");
}
System.out.println("</table>");


}


Here's the base class for parsers:

public abstract class MFHDataParser {

private boolean debug=false;

private char[] delims = {',',' ','.',';'};

private Map<String,String> emailToName = new HashMap<String,String>();

private List<String> validPrizes = new ArrayList<String>();

public void setValidPrizes(List<String> prizeList) {
this.validPrizes = prizeList;
}

protected void mapEmailToName(String email, String name) {
this.emailToName.put(email, name);
}

public String getNameForEmail(String email) {
return this.emailToName.get(email);
}

public abstract Map<String,List<String>> extractData(String filename);

/** Do our darndest to figure out what prizes a donator has mentioned on
* a given line. Note: Be sure to complain if we don't recognize a prize.
*
*/
protected List<String> extractPrizes(String prizes, int amount)
throws NoPrizeFoundException {

String ucPrizes = prizes.toUpperCase().trim(); //for consistency's sake
if (isDebug()) {
System.out.println("Incoming prize string " + ucPrizes);
}

// basic strategy
// find two numbers followed by a delim (, space, ., eol, ;)
// back up and find two letters before it
// then back up (not into an earlier code) and find a #
// can't use java's regex abilities because i need to divide into larger chunks

// put that many copies into the list
// verify that size of list = donation /10. bark if not
// List = one of each real raffle ticket (5xUW03 -> 5 entries in List)

List<String> prizeList = new ArrayList<String>();
List<Integer> prizeCounts = new ArrayList<Integer>();

if (ucPrizes.length() < 4 ) {
throw new NoPrizeFoundException(ucPrizes);
} else if (ucPrizes.length() == 4) {
// exact count. easy case, but verify that it's legit

String prizeCode = findPrizeCodeInTextBlock(ucPrizes);
if (!validPrizes.contains(prizeCode)) {
System.out.println("INVALID PRIZE CODE: " + prizeCode);
}
for (int i =1;i<= amount/10; i++) {
prizeList.add(prizeCode);
}
} else {
// in this case we need to walk through the list, divided it into
// chunks, and find the prize code in each
int chunkStart = 0;
for (int i = 0; i < ucPrizes.length();i++) {
if (i == ucPrizes.length() - 1 ||
isDelim(ucPrizes.charAt(i))) {
String curChunk = null;
if (isDelim(ucPrizes.charAt(i))) {
curChunk = ucPrizes.substring(chunkStart,i);
}

if (i == ucPrizes.length() - 1) {
curChunk = ucPrizes.substring(chunkStart,i+1);
}
if (curChunk.length() < 4) {
continue;
}
String prizeCode = findPrizeCodeInTextBlock(curChunk);
if (prizeCode.equals("")) {
continue; // not in this text block
} else if (!this.validPrizes.contains(prizeCode)) {
System.out.println("INVALID PRIZE CODE: " + prizeCode);
}


prizeList.add(prizeCode);
int prizeCount = new Integer(
findPrizeQuantityInTextBlock(curChunk,prizeCode));
if (prizeCount == -1) {
prizeCounts.add(new Integer(1));
} else {
prizeCounts.add(new Integer(prizeCount));
}

chunkStart = i + 1;
}
}
}

// expand prize list as needed
if (prizeList.size() == amount /10) {
return prizeList; // if there are as many prizes as the amount
// would suggest, do one ticket each
} else if (prizeList.size() == 1) {
// create an expanded list that has one entry for each ticket
List<String> newPrizeList = new ArrayList<String>(amount/10);
for (int i = 0;i < (amount /10); i++) {
newPrizeList.add(prizeList.get(0));
}
return newPrizeList;
} else {
// we have a mix of amounts and quantities
List<String> newPrizeList = new ArrayList<String>(amount/10);
for (int i = 0; i < prizeList.size();i++) {
Integer count = prizeCounts.get(i);
for (int j = 0; j < count.intValue(); j++) {
newPrizeList.add(prizeList.get(i));
}
}
return newPrizeList;
}
}

protected int parseAmount(String amtString) {
if (amtString.trim().equals("")) {
return 0;
}
return (int)(Double.parseDouble(amtString));
}

/** Takes a guess at the prize quantity in a given text block */
private int findPrizeQuantityInTextBlock(String chunk, String prizeCode) {
// make a spaced version so we can look for UC 01 as well as UC01
// walk over the string looking for numbers, skipping the prize code
String spacedPrizeCode = prizeCode.substring(0,2) + " " +
prizeCode.substring(2,4);

String newChunk = chunk.replace(prizeCode,"");
newChunk = newChunk.replace(spacedPrizeCode,"");

for (int i = 0; i < newChunk.length(); i++) {
if (Character.isDigit(newChunk.charAt(i))) {
if (i < newChunk.length() - 1 &&
Character.isDigit(newChunk.charAt(i+1))) {
// two-digit quantity
char[] digits = {newChunk.charAt(i),newChunk.charAt(i+1)};
return Integer.parseInt(new String(digits));
} else {
return Integer.parseInt(newChunk.substring(i,i+1));
}
}
}

// one last check. some bidders wrote "TWO" instead of 2
if (newChunk.indexOf("TWO") != -1) {
return 2;
}

return -1;
}

/** Returns the offset of something that looks like a prize code. */
private String findPrizeCodeInTextBlock(String chunk) {

// look for 2 letters followed by 2 numbers => prize code
for (int i = 3; i < chunk.length(); i++) {
if (Character.isDigit(chunk.charAt(i)) &&
Character.isDigit(chunk.charAt(i - 1))) {
if (chunk.charAt(i-2) == ' ') {
if (Character.isLetter(chunk.charAt(i-3)) &&
Character.isLetter(chunk.charAt(i-4))) {
return chunk.substring(i-4,i-2) +
chunk.substring(i-1,i+1);
}
} else {
if (Character.isLetter(chunk.charAt(i-2)) &&
Character.isLetter(chunk.charAt(i-3))) {
return chunk.substring(i-3,i+1);
}
}
}
}
return "";
}

protected boolean isDebug() {
return this.debug;
}

public void setDebug(boolean debug) {
this.debug = debug;
}

private boolean isDelim(char c) {
for (int i = 0; i < this.delims.length; i++) {
if (c == delims[i]) {
return true;
}
}
return false;
}

}


And here's the Excel-parsing subclass of Data Parser (you can see that it's got some logic that should be in DataParser, but in truth we've always used the Excel format, so it hasn't been an issue.

public class ExcelDataParser extends MFHDataParser {

public Map<String,List<String>> extractData(String filename) {
Map<String,List<String>> retVal = new HashMap<String,List<String>>();
try {
Workbook wkbk = Workbook.getWorkbook(new File(filename));
Sheet sheet = wkbk.getSheet(0);
for (int i = 0; i < sheet.getRows(); i++) {
if (i == 0) { continue; }// skip headers
String name = sheet.getCell(0,i).getContents();
String email = sheet.getCell(1,i).getContents();
String date = sheet.getCell(2,i).getContents();
if (isDebug()) {
System.out.println("amount: " + sheet.getCell(3,i).getContents());
}
int amt = parseAmount(sheet.getCell(3,i).getContents());
String comment = sheet.getCell(4,i).getContents();
if (email == null || email.trim().equals("")) {
throw new IllegalArgumentException("No email found at line " + i);
}
mapEmailToName(email,name);


if (comment == null || comment.trim().length() == 0) {
System.out.println("No comment on line " + (i+1));
System.out.println("");
continue;
}
List<String> prizes = extractPrizes(comment,amt);
if (isDebug()) {
System.out.print( "prizes for line " + (i+1) + " ");
for (Iterator<String> prizeIt = prizes.iterator();
prizeIt.hasNext();) {
System.out.print(prizeIt.next() + " " );
}
System.out.print("\n");
}

if (prizes.size() != amt / 10) {
System.out.println("Line " + (i+1) +
" does not have the right number of prizes for the amt");
System.out.println("");
}

// compress the lists down to MFHPair, which includes an email
// and a count. Insert into map, keyed by prize code
Collections.sort(prizes); // make sure they're in order
String curPrize = "";
int curCount = 0;
for (Iterator<String> prizeIt = prizes.iterator();
prizeIt.hasNext();) {
String prizeFromList = prizeIt.next();
List<String> bidders = null;
if (retVal.containsKey(prizeFromList)) {
bidders = retVal.get(prizeFromList);
} else {
bidders = new ArrayList<String>();
retVal.put(prizeFromList,bidders);
}
bidders.add(email);
}
System.out.println("");
}
return retVal;
}
catch (Exception e) {
System.err.println("There was a problem: " + e);
e.printStackTrace();
System.exit(1);
}
return null;
}
}