Sunday, September 16, 2012

Tsung vs. JMeter

I first learned of tsung from a colleague in EA's Pogo division, and I was intrigued by his description of the load it could generate. I've been pushing it at Maxis as a result.

There's sound architectural reasoning for its abilities. It's written in Erlang, which delivers highly concurrent systems by eschewing the use of threads in favor of events. The various Java-based loadtesting tools use threads to simulate users, which necessarily limits their abilities.

There's also good anecdotal evidence favoring tsung, not only from the Pogo folks but also from the fact that it's used to loadtest ejabberd, the chat system Facebook uses that is generally considered to be the most scalable around. (It's also written in Erlang)

But how much better is it than JMeter, the go-to Java-based loadtesting tool? JMeter is probably easier to configure -- it has a GUI versus moderately documented XML -- and it probably has a richer feature set. So is it worth going to tsung?

There are some numbers out there, such as this slideshow, which found that tsung could generate up to 50,000 simultaneous users while JMeter couldn't go above 1,000 requests per second, but I wanted to find out for myself.

I set up a simple web server (mochiweb, an Erlang-based web server that also has a good scalability track record -- sensing a pattern here?) that served a simple, static page. Remember the goal wasn't to test the server under load; it was to test how much load the clients could generate. I also ran top to get a sense of how much CPU my process was using. There are probably lots of things wrong with my methodology, but, again, I just wanted a sense of the difference.

Here's some graphs I made comparing CPU usage by tsung for given rates of users. This was on my MacBook Pro with a minimal set of other applications open.

Note that the numbers on the x-axis show the number of users up to that point. The web server responds very quickly in this scenario, so the number of simultaneous users is probably very small.

Tsung Graphs



And here are the graphs I could collect from JMeter, which I configured via the GUI and then ran in command-line mode. Once I tried to get to 100 users per second, the JVM never had enough memory to create all the threads, no matter how I adjusted the heap size. This is, of course, the other major resource that threads consume.

So tsung, even on my machine, could handle about 50 times more users than JMeter, which is roughly consistent with the numbers in the report I link to above. That said, for tiny amounts of load, JMeter compares favorably with tsung and is a lot quicker to get up and running. 




JMeter Graphs




I've noticed a tendency in Java shops to rely solely on tools written in Java. It drives me crazy; why put blinders on to a big chunk of software universe? I (more or less) like Java; I use it all the time. But it's not good for everything, and it's worth knowing what else is out there that might be better at solving the problem at hand. 

Yes, tsung is "harder" than JMeter. It's written in another language. Boo hoo. Use the tool that makes sense; in this case it's hard to argue with tsung's capabilities.

8 comments:

  1. Hi there,

    Regarding JMeter, you might have a bias. I'm no expert, but I recall you have to uncheck "Functional Mode" so that JMeter discards the responses instead of keeping them in memory.

    Anyway, if you're looking for an efficient yet user friendly stress tool, you should have a look at Gatling: Scala Actors and NIO.

    http://gatling-tool.org

    Cheers,

    Stéphane

    ReplyDelete
  2. Thanks for the tip, Stéphane!

    A friend mentioned gatling as well, and it looks very promising: It seems to have all the same architectural benefits as tsung.

    ReplyDelete
  3. Thanks,

    This is exactly what I was searching for as I am curently bumping up against volume limitations with Jmeter. I would be interested to see the difference when the client driver needs to parse out sessions or perform other http manipulations. I have seen some regex enhanced features in the latest version of tsung that suggest tsung may have the edge, but only a controlled test such as yours would reveal this.

    Regards

    Ian

    ReplyDelete
  4. One advantage, which has JMeter - is that there is cloud service, which allows to configure test with up to 80000 users. And this may save time.

    ReplyDelete
  5. Dzmitry, Good point, though there are obvious cost implications there. Still, good to know!

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. JMeter can still get the job done if you're willing to turn to a commercial solution. I work for http://blazemeter.com , and we provide JMeter in the cloud, so that you can run any of your JMeter scripts with up to 100,000 concurrent users. You'll also get real time reporting with a ton of cool graphs <\marketing pitch>

    ReplyDelete
  8. Hello,
    I think your methodology is strange, check this:
    http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/

    JMeter results are strangely really bad, my experience with it shows much different results.

    It would be nice to show the Test plan in Tsung and JMeter so that we understand if your users leave or still work.
    Finally are you sure your machine is showing realistic results ? CPU graph is strange even for TSung

    ReplyDelete