SoftSlate Commerce Load Test Analysis

Methodology:

The SoftSlate Commerce load test was performed using Apache JMeter (http://jakarta.apache.org/jmeter/index.html).

To develop our test plan, we examined the server statistics of a real ecommerce Web site that gets over 80,000 page views a day and over 10,000 unique visitors per day. In particular we looked at the peak hour for this site for the previous month, during which 7,040 page views hit the site in one hour. We more than doubled that amount to arrive at the target of 15,000 page views for our one-hour test.

Many ecommerce sites experience low page view to session ratios, and we decided on ratio of two page views for each session. Since creating each session takes a little extra processing, this low ratio would tax the application somewhat more than a higher ratio, which might be more typical.

Since we wanted to simulate the behavior of real users, we then examined the statistics of several sites to arrive at an average time spent between requests. In the test plan, we defined a delay that averages out to 12 seconds between requests, but varies randomly between 4 and 20 seconds.

Next, we described three groups of users for the test plan:
1. One-Hit Wonders: Users who arrive at one of the application's category pages, decide it's not what they're looking for and leave. Our test plan calls for 3000 of these users, spread throughout the hour-long test.
2. Just Browsin': Users who spend a little more time browsing the application's categories and products. We defined 3800 of these users, each of whom hit the application three times before ending their session.
3. Comin' to Buy: Users who spend a good deal of time on the site and end up placing an order. We set up 15 such users, each of whom hits the site 50 times, adding items to their carts and proceeding through checkout.


The Test Environment:

We load tested SoftSlate Commerce on a dual 3.6 GHz Dell 2850 with 2 GB memory and four drives in a RAID 10 array. The operating system was RedHat Enterprise 3.0. We used Tomcat 5.5 with the Sun JDK, version 1.5.0_04. All requests went through the Apache HTTP Server, and were relayed to Tomcat through the JK Apache - Tomcat Connector, version 1.2.10. The back end database used was MySQL version 4.1, and the database tables were set up as InnoDB tables for full transactional support and integrity.

To make sure the results applied even to stores with a great many products, the database for the test was loaded with data taken from a store with 13,144 products and 612 categories.


Results:

The results demonstrated quite clearly that SoftSlate Commerce demands relatively little from the server resources. Our original test plan of 15,000 page views in one hour caused the server's load average to peak at a safe 2.83. The memory demands from the Tomcat application server peaked at just 248 MB, with 3,776 independent user sessions at the peak.

Most importantly, the user experience was preserved even with this very high load. Across all 15,000 page views, the average response time from the server was a mere .603 seconds. The deviation of .402 seconds means that two-thirds of all requests took place in under one second. There were no failed requests encountered at all during the test.

A screen shot of the output of a top command taken half-way through the test accompanies this file. As it shows, MySQL was demanding more from the processor than Tomcat or Apache. This is likely due to the fact that the InnoDB engine was in effect, which provides support for database transactions and foreign key integrity. Undoubtedly, switching to the MyISAM engine would improve performance even more. 

The Apache JMeter log file produced from the test also accompanies this file. See log.jtl. 


Running the Test Plan:

The Apache JMeter .jmx test plan file accompanies this analysis, so you can run the exact same plan on your server, and review exactly what our test consisted of. NOTE: you will have to review the sample requests and make sure you change the product codes, category codes, and paths to suit your installation.


Questions:

For any questions about the load test methodology or results, don't hesitate to contact SoftSlate at info@softslate.com.