Tsung vs. JMeter

I first learned of tsung from a colleague in EA's Pogo division, and I was intrigued by his description of the load it could generate. I've been pushing it at Maxis as a result.

There's sound architectural reasoning for its abilities. It's written in Erlang, which delivers highly concurrent systems by eschewing the use of threads in favor of events. The various Java-based loadtesting tools use threads to simulate users, which necessarily limits their abilities.

There's also good anecdotal evidence favoring tsung, not only from the Pogo folks but also from the fact that it's used to loadtest ejabberd, the chat system Facebook uses that is generally considered to be the most scalable around. (It's also written in Erlang)

But how much better is it than JMeter, the go-to Java-based loadtesting tool? JMeter is probably easier to configure -- it has a GUI versus moderately documented XML -- and it probably has a richer feature set. So is it worth going to tsung?

There are some numbers out there, such as this slideshow, which found that tsung could generate up to 50,000 simultaneous users while JMeter couldn't go above 1,000 requests per second, but I wanted to find out for myself.

I set up a simple web server (mochiweb, an Erlang-based web server that also has a good scalability track record -- sensing a pattern here?) that served a simple, static page. Remember the goal wasn't to test the server under load; it was to test how much load the clients could generate. I also ran top to get a sense of how much CPU my process was using. There are probably lots of things wrong with my methodology, but, again, I just wanted a sense of the difference.

Here's some graphs I made comparing CPU usage by tsung for given rates of users. This was on my MacBook Pro with a minimal set of other applications open.

Note that the numbers on the x-axis show the number of users up to that point. The web server responds very quickly in this scenario, so the number of simultaneous users is probably very small.

Tsung Graphs

And here are the graphs I could collect from JMeter, which I configured via the GUI and then ran in command-line mode. Once I tried to get to 100 users per second, the JVM never had enough memory to create all the threads, no matter how I adjusted the heap size. This is, of course, the other major resource that threads consume.

So tsung, even on my machine, could handle about 50 times more users than JMeter, which is roughly consistent with the numbers in the report I link to above. That said, for tiny amounts of load, JMeter compares favorably with tsung and is a lot quicker to get up and running. 

JMeter Graphs

I've noticed a tendency in Java shops to rely solely on tools written in Java. It drives me crazy; why put blinders on to a big chunk of software universe? I (more or less) like Java; I use it all the time. But it's not good for everything, and it's worth knowing what else is out there that might be better at solving the problem at hand. 

Yes, tsung is "harder" than JMeter. It's written in another language. Boo hoo. Use the tool that makes sense; in this case it's hard to argue with tsung's capabilities.