XINS - Feedback on performance tests

Introduction

This article will compare the performance of XINS Web Services vs JAX-WS RI and XFire (renamed as CFX).


The tests have been performed on September 20, 2007.

The test data and environment

The following version have been used:

  • XINS 2.1-beta3
  • CFX 2.0.1
  • NetBeans 6.0 Beta 1 (The version which has Glassfish V2 and Java EE, JAX-WS 2.1.2)

My environment:

  • Windows 2000
  • JDK 1.6.0_02
  • Pentium IV 3.0 GHz
  • 768 MB of memory
  • Software started: NetBeans 6.0 B1, Ant Commander, Outlook, Firefox, OpenOffice Tray icon

Tests:

  • The SOAP request have been created with SOAP UI 1.7.5 from the WSDL provided in the example.
  • The tests have been executed with Apache JMeter 2.1.1.
  • The test consist of 10 threads sending 10 requests (so 100 requests in total) with a ramp-up period of 1 second.
  • The test is executed once without taking the results into account.
  • I executed then 3 times the test, took for XFire and JAX-WS RI the best result and for XINS the worst result.
  • Between each test, a Clear all is performed in JMeter.
  • I used as result the average of the Aggregate Report (The averarge time that a request took).
  • The results are in milli seconds.
  • Axis 2 1.3 didn't work. I first had to swith to JDK 1.5 and to deploy under Tomcat 5.5, then SOAP UI worked but stock-axis2.jmx didn't. Tests performed by Sun Micorsystem suggest that Axis2 is slower than JAX-WS. Other tests suggest that Axis2 is faster than XFire.
  • Because of limited time I have decided not to benchmark JBoss WS or Spring Web Services.

XINS Tests:

  • cd demo\xins-project
  • xins run-myproject

CFX Tests:

  • cd samples\hello_world
  • ant server

JAX-WS RI Tests:

  • NetBeans File -> New Project -> Samples -> Web Services -> Calculator
  • Select CalculatorApp as main project and hit the run button (and wait)

The results

XINS SOAP5 ms
XFire (CFX)15 ms
JAX-WS RI152 ms
XINS performance

Analisys

  • There is a large difference factor between the different frameworks, meaning that if performance matters to you, you should consider using XINS.
  • XINS SOAP was 3 to 8 times faster than XFire. The best XINS SOAP result was 4 ms and the worst XFire result was 31 ms.
  • XINS was at INFO logging level which logs 3 times per request and slows down XINS (about 1 ms).
  • Using Stax XML parser and specific data binding does not determine the speed of the WS framework.
  • Try to code not with performance in mind but with code quality. Then benchmark and profile the application in the last beta stage.
  • At the end the performance depends more on the hardware, the network, the Servlet container and if used the database.

Pushing further

Now I'm at the point of thinking how to damage more XINS in order to perform slower. First thing is to choose another function. I decided to go for the SimpleTypes of the allinone. It contains 14 input parameters of different types (timestamp, binary, double, properties, list, ...) and returns also a few output parameters.
Worst result: 7 ms.

As it's not slow enough I decide to move to JDK 1.5 and Tomcat 5.5. (Still SimpleTypes with SOAP at INFO level)
Worst result: 3 ms.

In the following graph, I also added Axis 2 results got from the 10 times average call with SOAP UI (after 15 calls)

Here I'm almost out of ideas (except Thread.sleep() or Thread.wait() or JDK 1.3) to slow down XINS even further.

JAX-WS RI152 ms
XFire (CFX)15 ms
Axis2 (SOAP UI)11 ms
XINS SOAP5 ms
XINS SOAP, 14 parameters, JDK 1.5, Tomcat3 ms
XINS performance final result

Feedback

I have send e-mail to Dan Diephouse and Kohsuke Kawaguchi in order for them to have their say about the tests and the results. You can read their feedback in this document

Koshushe noticed a few points:

  • On his laptop (2.8GHz HT-enabled Pentium 4 machine with 2GB), his results for JAX-WS RI are 3 ms.
  • JMeter doesn't use HTTP keep-alive which can make a big difference for a trivial payload.
  • JMeter doesn't have a timer with good precision.
  • With a ramp up of 1 second, if the request time is less than 10 ms you will never have 2 concurent requests for 100 requests.