These new frameworks are supposed to facilitate fast development. But, how fast – and scalable – are the applications built in and for these frameworks?
The goal of this post is to get a preliminary answer to this question. NOTE: before you scream “it depends on the problem and any such comparison must include real code not to be a silly toy experiment” let me finish this post I know my tests are flawed, but they do give an indication of relative performance and scalability. With the latter, I mean on one server.
What will be measured is the accepting of an incoming connection, and either delivering a static page or dynamically creating it. All is – of course – using the HTTP protocol. Important: there will be no model objects involved; this implies no access to database or other persistent store..
To counterpoint these new and cool frameworks, we add an old stable contestant to the mix: Apache Web Server. What the heck! We even throw in lighttpd for the fun of it.
If you enjoy minutiae about the exact versions and environment used for this analysis, go the end of the post.
Let the fun begin!
ab (ApacheBench) to measure the time for each request-response cycle, including the loopback socket communication.
The command executed is
ab -n 1000 -c 100 http://localhost:
which runs 100 requests in parallel in ten steps, yielding 1000 requests in total. What I measure is number of requests that can be handled in one second. Since this is using the loopback address and very simple pages, this can be viewed as the upper limit for actual use.
NOW your favorite tool’s performance will be judged! Can you take the truth? Ok, here it comes:
UPDATE 12/10/07:for some paraphenomenal reason, Yariv run almost identical tests the very same day! Although he did only focus on ErlyWeb vs Rails. Anyway, quite similar results to mine: look for yourself.
- Well, for static content, lighttpd is the clear winner, and for dynamic content, it is almost a draw between the two of the four – IMHO – coolest functional languages right now (the other two being F# and Scala), with a slight advantage to the Haskell framework. This is pretty interesting since it is in very early development; ok, admittedly, so is ErlyWeb, but Erlang is not
- The currently most popular framework amongst the kids, Ruby on Rail, is sloooow, just handling some 5% of the requests of the functional language frameworks; and this is on one server! I do not want to imagine the difference once a few servers are added
- What about Java (J2EE, Spring and/or Tomcat JSPs) and ASP.NET!!?? Well, I will include them later…
- Both RoR and ErlyWeb used a rather intricate dispatching mechanism; well, you can clearly see who is the performer in that battle!
- ErlyWeb had a hard time handling 1000 requests coming within an interval of a few seconds, so I had to lower the burst to 500 requests there. It did not have as a severe problem if one bypassed the dispatch and used raw Yaws.
LAMP– Apache 2.2.6 + PHP 5.2.5
RoR– Ruby 1.8.6 + Rails 1.2.6 + Mongrel 1.1.1
ErlyWeb– Erlang 5.5.5 + Yaws 1.68 + ErlyWeb 0.6
HAppS– GHC 6.8.1 + HAppS 0.9.1a
lighttpd– lighttpd 1.4.18
The test box is a MacBook Pro, having a dual core CPU running at some 2.x GHz and ample RAM – enough not to have any disk swapping.
UPDATE 12/10/07:I am using the same box for the (light) client. I do not think that affects relative, but could affect the absolute measures slightly. I did that not to care about transfer of bytes over the wire. Ok, I admit it, I also did it not to have to setup an Amazon EC2 box for this sole purpose
The frameworks are: