Ever wondered what is the best technology stack for building web applications? I’m sure you have.
Last week my company went on a workshop, during which we tried to answer that question. I wrote a requirements specification for a simple web application. Everyone had to choose the technology stack they’re most familiar with and implement the application. Everyone received a benchmark script for the application and at the end of the day we compared the performance results. The application was just enough complex to require at least a day to implement. In the end we multiplied the performance by the level of functionality implemented.
The webapp I chose is the same one for the couldspeed project. It is based on the model of social networking applicatins. Once registered and logged in, users can add other users as friends and submit posts. The homepage will display the 20 most recent posts from all user’s friends including the user himself.
There are 2 entities each with 2 attributes:
- users(email, password)
- post(date, content)
and there are 2 relationships
- users are friends with other users (n to n)
- each post is written by a user (1 to n)
there are 2 web pages: login and home.
From the login page, users can register themselves or log in.
The home page displays the number of friends, the 20 most recent posts (for each post, the date, author and content) and allows users to add posts, friends or logout.
Information has to be persisted somehow.
Each developer got a copy of JMeter and 3 scripts. The 3 scripts excercized 3 different levels of functionality:
- Register Users – registered a number of users (100)
- Make Friends – created around 10 friends per user
- Post Stuff – posted messages
For the first level of functionality, the developer had to implement the functionality behind the Register button. Registered users had to be persisted. This level had a multiplier of 1.
For the second level of functionality, developers had to implement logging in, keeping session information, and the friendship relationship. The homepage should have shown the number of friends. This level had a multiplier of 3.
For the third level of functionality, developers had to implement everything: posting messages and showing the messages from all friends. Since this functionality included a complex query that ate a lot of processing power, the multiplier was 50.
Only three of the contenders implemented level 3 of functionality, but we were able to measure the performance only for the erlang implementation. The other 2 that made it to the 3rd level were ASP+SQLServer and Ramaze. We will soon publish the results of the two missing level 3 implementations.
The winner was Ben with 1111.1 points. Erlang + CouchDB proved to be quite fast and productive in the right hands.
Allthough just one day of development is not nearly enough to measure productivity, we noticed that choosing a particular language that promised a high level of productivity wasn’t as important as choosing the language you’re most familiar with.
Alltough Level 1 of functionality does not provide a meaningful scenario to measure performance, we can see that most of the implementations had the same performance, including Ruby, which has been measured to be 100 times slower than C++, but in our scenario it actually showed the best performance.
Allthough we must point out that both ruby implementations and the ASP implementation showed instability. Errors ranging from 15% to 30% were reported during the benchmarks.
As a comparison, I have done some java implementations before the competition. They all provide level 3 functionality. They all use Servlets in the web tier, but the DAO tier changes: we have JDBC with MySQL, EJB3 with MySQL and pure in memory.
Compared to the Erlang implementation wich serves 15 req/s I have similar results for the JDBC (15 req/s). A “heavyweigth” stack such as EJBs actually manages to produce 38 req/s, but by far the fastest solution is keeping everything in memory that brings us to an impressive 2000 req/s. Obviously the in memory solution cannot be compared to the others because it doesn’t satisfy the persistence requirement. But can still be used as a comparison to see how much performance do we loose on the persistence layer.
We have had requests from other people that were not taking part of the competition to submit their implementations. Since we have no way to measure productivity, all submissions now require level 3 of functionality. All implementations will be published on the cloudspeed website. If you wish to contribute by improving an implementation or submitting your own just send me an email or leave a comment here.
Currently the benchmark is adapted to quickly give a very aproximative measure. 100 users is not a realistic number even for a small website. Our next goal is to create a benchmark that will register up to millions of users, have an average of 100 friends per user and post several million messages.
The final goal is to port these implementations to the cloud and measure their scalability. We don’t only want to see the performance on a single machine, but also how will the performance be when we reach the limits of a single server. Ideally we want to try to run these applications on clusters as big as we can get.
Stay tuned for updates…