Thanks to Paul Findlay, we finally have a possible contender in the Java camp with Aalto.
This goes to show you how good library design and the D Programming Language come together to kick serious butt.
PS: I am looking for anyone to do comparisons with MSXML, RapidXML, etc. More native code help is needed. Send me email at scott aht dotnot daht org.
Popularity: 59%
Next up from Paul Findlay: Aalto. Aalto.java:
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamReader;
import java.io.*;
public class Aalto
{
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
long length = file.length();
byte[] bytes = new byte[(int)length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException(”Could not completely read file “+file.getName());
}
is.close();
return bytes;
}
public static void main (String args[]) throws Exception
{
int iterations = 2000;
XMLInputFactory xmlif = XMLInputFactory.newInstance();
xmlif.setProperty(XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, Boolean.FALSE);
xmlif.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, Boolean.FALSE);
xmlif.setProperty(XMLInputFactory.IS_COALESCING, Boolean.FALSE);
byte[] content = Aalto.getBytesFromFile(new File(args[0]));
ByteArrayInputStream bais = new ByteArrayInputStream(content);
for (int i = 0; i < 10; i++) {
long start = System.currentTimeMillis();
for (int j = 0; j < iterations; j++) {
XMLStreamReader xr = xmlif.createXMLStreamReader(bais);
while (xr.hasNext()) {
xr.next();
}
xr.close();
bais.reset();
}
long stop = System.currentTimeMillis();
double timer = (stop - start) / 1000.0;
double total = (content.length * iterations) / (timer * (1024 * 1024));
System.out.print(total);
System.out.println(” MB/s”);
}
}
}
How it was run:
Results:
Average for hamlet.xml: 147.22 MB/sec
Average for soap_mid.xml: 43.80 MB/sec
As noted on the website, Aalto does seem to be quite fast on the “fast path”. Impressive for a Java solution at this point.
Update: 2008-03-03 13:15 PST: Thanks to Paul Findlay for catching my misspelling of the aalto.jar in the java run command. These numbers posted are actually for the default Java6 StaX parser, and not Aalto. Re-running, I get:
Average for hamlet.xml: 147.85 MB/sec
Average for soap_mid.xml: 85.95 MB/sec
Much more impressive numbers from the Java camp. Graphs will be updated later today.
Popularity: 38%
Another benchmark from Paul Findlay, using Javolution. Here is Javolution.java:
import javolution.xml.stream.XMLInputFactory;
import javolution.xml.stream.XMLStreamReader;
import java.io.*;
public class Javolution
{
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
long length = file.length();
byte[] bytes = new byte[(int)length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException(”Could not completely read file “+file.getName());
}
is.close();
return bytes;
}
public static void main (String args[]) throws Exception
{
int iterations = 2000;
XMLInputFactory factory = XMLInputFactory.newInstance();
byte[] content = Javolution.getBytesFromFile(new File(args[0]));
ByteArrayInputStream bais = new ByteArrayInputStream(content);
for (int i = 0; i < 10; i++) {
long start = System.currentTimeMillis();
for (int j = 0; j < iterations; j++) {
XMLStreamReader xr = factory.createXMLStreamReader(bais);
while (xr.hasNext()) {
xr.next();
}
xr.close();
bais.reset();
}
long stop = System.currentTimeMillis();
double timer = (stop - start) / 1000.0;
double total = (content.length * iterations) / (timer * (1024 * 1024));
System.out.print(total);
System.out.println(” MB/s”);
}
}
}
Average for hamlet.xml: 51.16 MB/sec
Average for soap_mid.xml: 45.93 MB/sec
Most of the Java camp is starting to look the same.
Popularity: 31%
Thanks to Paul Findlay for submitting 3 new Java benchmarks, the first of which is for Woodstox. The file is Woodstox.java, listed here:
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamReader;
import org.codehaus.stax2.XMLInputFactory2;
import java.io.*;
public class Woodstox
{
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
long length = file.length();
byte[] bytes = new byte[(int)length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException(”Could not completely read file “+file.getName());
}
is.close();
return bytes;
}
public static void main (String args[]) throws Exception
{
int iterations = 2000;
XMLInputFactory2 xmlif = (XMLInputFactory2) XMLInputFactory2.newInstance();
xmlif.setProperty(XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, Boolean.FALSE);
xmlif.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, Boolean.FALSE);
xmlif.setProperty(XMLInputFactory.IS_COALESCING, Boolean.FALSE);
xmlif.configureForSpeed();
byte[] content = Woodstox.getBytesFromFile(new File(args[0]));
ByteArrayInputStream bais = new ByteArrayInputStream(content);
for (int i = 0; i < 10; i++) {
long start = System.currentTimeMillis();
for (int j = 0; j < iterations; j++) {
XMLStreamReader xr = xmlif.createXMLStreamReader(bais);
while (xr.hasNext()) {
xr.next();
}
xr.close();
bais.reset();
}
long stop = System.currentTimeMillis();
double timer = (stop - start) / 1000.0;
double total = (content.length * iterations) / (timer * (1024 * 1024));
System.out.print(total);
System.out.println(” MB/s”);
}
}
}
I built it and ran it with the following commands:
And the results:
Average for hamlet.xml: 79.76 MB/sec
Average for soap_mid.xml: 49.53 MB/sec
Popularity: 27%
I added Java DOM to the graphs. Building a tree in memory is not the fastest way to parse a doc, but it is the easiest way to modify the doc after parsing. Java 6 DOM shows off not too terribly bad in the parsing speed, but with all the allocation going on, RAM usage skyrockets, and the efficiency graph shows the pain.
This goes to show you how good library design and the D Programming Language come together to kick serious butt.
Popularity: 32%
I have added a Java 6 DOM to the benchmark, so I could compare the Tango DOM. I used Dom.java, listed here:
public class Dom {
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
long length = file.length();
byte[] bytes = new byte[(int)length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
is.close();
return bytes;
}
public static void main(String[] args) {
if (args.length <= 0) {
System.out.println("Usage: java Dom filename");
return;
}
try {
String document = args[0];
int iterations = 2000;
byte[] content = getBytesFromFile(new File(document));
ByteArrayInputStream bais = new ByteArrayInputStream(content);
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder parser = factory.newDocumentBuilder();
for (int i = 0; i < 10; i++) {
long start = System.currentTimeMillis();
for (int j = 0; j < iterations; j++) {
parser.parse(new InputSource(bais));
bais.reset();
}
long stop = System.currentTimeMillis();
double timer = (stop - start) / 1000.0;
double total = (content.length * iterations) / (timer * (1024 * 1024));
System.out.print(total);
System.out.println(" MB/s");
}
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Results on the quad core machine:
Hamlet average: 49.63
Soap_mid average: 29.22
Popularity: 25%
Speed master Kris made some changes to Tango’s xml libraries today, and increased the performance of the parser to over 500MB/second! The machine is still the quad core 2.66GHz Intel box running Linux with 4GB of RAM. This run reflects revision 3286 of Tango SVN.
I will only update the images here, I think you should now know how I obtained them…
While SAX is showing slower in speed than DOM in Tango (I hope that is as weird to read as it was for me to write), you can see that the RAM usage graph puts it back into perspective.
I also forgot to note that this quad core box is now capable of parsing XML at over 2GB/sec if all 4 cores are used. Impressive indeed.
Tango is an alternate standard library for the D Programming Language.
Popularity: 31%
I decided to post a graph of speed versus resource usage as an interesting view into the overhead of the various programs. Since all benchmarks maxxed out the CPU at 100%, and all cached the data to be parsed, so disk wasn’t being used, that leaves RAM as a measurement of resource usage. The following is a chart of the parsing speed divided by the memory usage. Of note was xmlpull and xmlsax using 688KB of memory, so their numbers actually increased, showing not only the speed, but the conservation of resources. The RAM numbers were taken from top while the program was running, and represent the “Resident Set” so as not to make Java look horribly bad.
Update: 2008-02-24 15:45 PST - I updated the graph to offset the RAM usage by subtracting the file size from the total RAM, so that as the files get larger, they won’t be penalized. To put it into other words, the closer you can keep RAM usage to the filesize, decreasing overhead, the more resource efficient your parser is. I bet you are thinking Tango was designed that way from the beginning right about now, aren’t you?
Popularity: 25%
Next is Java 6’s default SAX implementation, Xerces. The code used was Sax.java, listed here:
public class Sax extends DefaultHandler
{
public Sax ()
{
super();
}
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
long length = file.length();
byte[] bytes = new byte[(int)length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
is.close();
return bytes;
}
public static void main (String args[]) throws Exception
{
int iterations = 2000;
XMLReader xr = XMLReaderFactory.createXMLReader();
System.out.println(xr.getClass().getName());
Sax handler = new Sax();
xr.setContentHandler(handler);
xr.setErrorHandler(handler);
byte[] content = Sax.getBytesFromFile(new File("soap_mid.xml"));
ByteArrayInputStream bais = new ByteArrayInputStream(content);
for (int i = 0; i < 10; i++) {
long start = System.currentTimeMillis();
for (int j = 0; j < iterations; j++) {
xr.parse(new InputSource(bais));
bais.reset();
}
long stop = System.currentTimeMillis();
double timer = (stop - start) / 1000.0;
double total = (content.length * iterations) / (timer * (1024 * 1024));
System.out.print(total);
System.out.println(" MB/s");
}
}
}
Results for hamlet.xml and soap_mid.xml, respectively:
Average parsing speed: 79.02 and 39.83 MB/sec, respectively. Note that I did remove the DTD declaration from hamlet.xml for this benchmark, since it was erroring out trying to find play.dtd.
Ouput from java -version:
Popularity: 32%
Many thanks to Nietsnie who was kind enough to write up a libxml2 sax benchmark, and run it on his quad core 2.66GHz box running linux. I have updated other benchmarks to reflect using his machine as well, to keep all on the same playing field. test.c is the benchmark code used, listed here:
Results for hamlet.xml:
Average parsing speed: 78.3MB/sec.
Popularity: 26%
Here is the current summary of the benchmarks run so far in a graphical form:
I hope to add more (libxml2, Xerces-C, etc) in the future. If you have C++ chops, I am looking for someone to code up one for MSXML. I will also be adding some Java benchmarks in here as well.
Update 2008-02-23 20:57 PST - Since Nietsnie was kind enough to donate his machine time, I re-ran all the current benchmarks on his box, to be able to include the libxml2 sax numbers as apples to apples. The graph is now updated, and includes the speed (Megabytes per second). Thanks to Robert Fraser for catching that.
The current benchmarking machine is an Ubuntu box with 4GB RAM sporting a quad-core Intel chip at 2.66GHz. In other words, much faster than my machine.
Popularity: 29%
I hesitate to publish these numbers, as they are not direct apples to apples comparison. The reason is that the D Programming Language version 2.0’s std.xml is an xml parser, but one where you must know the schema beforehand, and register handlers for each element by name. I was unwilling/too lazy to write said handlers for the docs I was doing, so I found a method called check(), that according to the source code comments makes sure that a document is well-formed, and contains no bad characters. That’s as close as I am going to get to parsing these docs without code help from the community, so take this with a grain of salt or two. I am using DMD 2.011, using stdxml.d to benchmark, listed here:
import std.stdio;
import std.xml;
import std.perf;
void benchmark (int iterations, invariant char[] content) {
auto elapsed = new HighPerformanceCounter();
elapsed.start;
for (auto i=0; ++i < iterations;) {
check(content);
}
elapsed.stop;
float timer = elapsed.milliseconds / 1000.0;
auto total = (content.length * iterations) / (timer * (1024 * 1024));
writef(total);
writefln(" MB/s");
}
void main()
{
invariant char[] content = import ("hamlet.xml");
for (int i = 11; --i;)
benchmark (10, content);
}
You will note that iterations are way down compared to others, because I really couldn’t wait around all night for the results. Results are:
D:\d2>stdxml
1.65656 MB/s
1.68694 MB/s
1.67213 MB/s
1.68908 MB/s
1.67318 MB/s
1.68481 MB/s
1.68268 MB/s
1.68162 MB/s
1.68588 MB/s
1.67951 MB/s
D:\d2>stdxml
1.18841 MB/s
1.22127 MB/s
1.2201 MB/s
1.2201 MB/s
1.23065 MB/s
1.21894 MB/s
1.22829 MB/s
1.22829 MB/s
1.22829 MB/s
1.23065 MB/s
If someone from the phobos community wants to update the code run here, just leave a comment or send me mail privately. scott at you can guess where.
Update 2008-02-23 19:57 PST
Running on a quad core 2.66GHz box yielded:
Average for hamlet.xml: 6.51 MB/sec.
Average for soap_mid.xml: 4.39 MB/sec.
PS: I also wanted to note for any naysayers, that I left off -O -release and -inline because the phobos example actually runs SLOWER with any and/or all of these flags. I am not trying to slip anything by anyone here.
Popularity: 14%
Next is Tango’s SaxParser, a SAX API layered on top of PullParser for the D Programming Language. It passes parsing events through to a handler, push-style. I used the current SVN HEAD of Tango, which is current revision 3247, and compiled with DMD v1.024. I count the number of elements, attributes, and text nodes, along with their lengths, to attempt to compare to the benchmarks here. Apparently, Tango is beating them masterfully. soap_mid.xml is the same file (by size, and I suspect, origin) as their “soap2.xml”. And they have an extra 200MHz of CPU in their benchmark. The benchmark code used was xmlsax.d, listed here:
import tango.io.Stdout;
import tango.time.StopWatch;
import tango.text.xml.SaxParser;
void benchmark (int iterations, SaxParser!(char) parser, char[] content)
{
StopWatch elapsed;
elapsed.start;
for (auto i=0; ++i < iterations;)
{
parser.parse;
parser.reset;
}
Stdout.formatln ("{} MB/s", (content.length * iterations) / (elapsed.stop * (1024 * 1024)));
}
void main()
{
auto content = import ("hamlet.xml");
auto parser = new SaxParser!(char);
auto handler = new LengthHandler!(char);
parser.setSaxHandler(handler);
parser.setContent(content);
for (int i = 11; --i;)
benchmark (2000, parser, content);
}
private class LengthHandler(Ch = char) : SaxHandler!(Ch) {
public uint elm;
public uint att;
public uint txt;
public uint elmlen;
public uint attlen;
public uint txtlen;
public void startElement(Ch[] uri, Ch[] localName, Ch[] qName, Attribute!(Ch)[] atts) {
elm++;
elmlen += localName.length;
foreach (inout attr; atts) {
att++;
attlen += attr.localName.length;
}
}
public void characters(Ch[] ch) {
txt++;
txtlen += ch.length;
}
}
Results for hamlet.xml:
D:\d\tango\example\text>xmlsax
258.59 MB/s
259.45 MB/s
258.91 MB/s
258.70 MB/s
258.79 MB/s
259.27 MB/s
259.37 MB/s
259.94 MB/s
258.72 MB/s
258.64 MB/s
D:\d\tango\example\text>xmlsax
179.96 MB/s
180.15 MB/s
180.97 MB/s
179.67 MB/s
180.76 MB/s
180.46 MB/s
179.90 MB/s
178.61 MB/s
179.20 MB/s
180.47 MB/s
Average parsing speed: 180.02 MB/sec. Sax seems to do a bit better than DOM with the attributes, but still shows a significant overhead to PullParser.
Update 2008-02-23 19:57 PST
Running on a quad core 2.66GHz box yielded:
Average for hamlet.xml: 347.63 MB/sec.
Average for soap_mid.xml: 240.41MB/sec.
Popularity: 13%
Next is Tango’s Document, a DOM-ish parser built on top of PullParser fro the D Programming Language. It builds an in-memory tree of the document being parsed, which can then be easily navigated/edited in-memory. I used the current SVN HEAD of Tango, which is current revision 3247, and compiled with DMD v1.024. The benchmark code used was xmldom.d, listed here:
/*******************************************************************************
*******************************************************************************/
void bench (int iterations)
{
StopWatch elapsed;
auto doc = new Document!(char);
auto content = import (”hamlet.xml”);
elapsed.start;
for (auto i=0; ++i < iterations;)
doc.parse (content);
Stdout.formatln ("{} MB/s", (content.length * iterations) / (elapsed.stop * (1024 * 1024)));
}
/*******************************************************************************
*******************************************************************************/
void main()
{
for (int i=11; --i;)
bench (2000);
}
It was compiled using: jake xmldom.d -O -release -inline -J.
Resulting run was:
D:\d\tango\example\text>xmldom
240.39 MB/s
239.77 MB/s
239.34 MB/s
240.70 MB/s
242.35 MB/s
241.36 MB/s
241.66 MB/s
242.81 MB/s
241.96 MB/s
241.92 MB/s
Average of the resulting run: 241.23 MB/sec parsing. That is one speedy little DOM builder. Run for soap_mid.xml brought back:
D:\d\tango\example\text>xmldom
117.36 MB/s
118.34 MB/s
118.28 MB/s
118.38 MB/s
117.57 MB/s
118.28 MB/s
118.73 MB/s
118.22 MB/s
118.12 MB/s
118.63 MB/s
Average of the runs was 118.19 MB/sec parsing. Looks like a similar result to PullParser. Attributes must have a fairly high cost in this implementation.
Update 2008-02-23 19:57 PST
Running on a quad core 2.66GHz box yielded:
Average for hamlet.xml: 336.07 MB/sec.
Average for soap_mid.xml: 166.47MB/sec.
Popularity: 11%
First up, Tango’s tango.text.xml.PullParser. You instantiate the parser, start the parse, and then continue to ask for the next ‘node’. I used the current SVN HEAD of Tango, which at the time of writing was revision 3247, compiled with DMD v1.024. The benchmark code ran is xmlpull.d, and is listed here:
import tango.text.xml.PullParser;
void benchmark (int iterations)
{
StopWatch elapsed;
auto content = import (”hamlet.xml”);
auto parser = new PullParser!(char) (content);
elapsed.start;
for (auto i=0; ++i < iterations;)
{
while (parser.next) {}
parser.reset;
}
Stdout.formatln ("{} MB/s", (content.length * iterations) / (elapsed.stop * (1024 * 1024)));
}
void main()
{
for (int i = 11; --i;)
benchmark (2000);
}
It was compiled with the command: jake xmlpull.d -O -release -inline -J.. Results of the run:
D:\d\tango\example\text>xmlpull
316.82 MB/s
315.71 MB/s
315.49 MB/s
317.55 MB/s
316.77 MB/s
316.69 MB/s
316.64 MB/s
317.43 MB/s
316.16 MB/s
317.90 MB/s
Average of the resulting run: 316.72 MB/sec parsing. Replacing hamlet.xml with soap_mid.xml in the above code results in:
D:\d\tango\example\text>xmlpull
229.19 MB/s
227.78 MB/s
229.12 MB/s
229.71 MB/s
228.63 MB/s
229.40 MB/s
228.98 MB/s
229.21 MB/s
230.02 MB/s
228.56 MB/s
Average of the resulting run: 229.06. Lower than hamlet.xml, probably due to the attribute processing required, but also possibly the lack of whitespace.
Update 2008-02-23 19:57 PST
Running on a quad core 2.66GHz box yielded:
Average for hamlet.xml: 476.77 MB/sec.
Average for soap_mid.xml: 339.15MB/sec. Now we are talking some speed!!! This D Programming Language has some merit.
Popularity: 11%
In wanting to see how well the Tango XML parsers fair in the world, I have started this benchmarking post. I will post all of my results, as well as the code and files that achieve these results here, so this post will be living as I expand and update it.
First off, baseline equipment. I have a Thinkpad T60p with 2.0Ghz Intel T2500 CPU, 2GB RAM, and a fairly slow hard drive. All of my tests will cache the document to be parsed in memory to try and elminate the hard drive as a potential bottleneck.
Next up, the files. I will be starting with hamlet.xml and soap_mid.xml. hamlet.xml weighs in at 274KB, and contains no attributes at all, very element heavy, with a moderate amount of whitespace (enough to make the file readable). soap_mid.xml weighs in at 132KB, uses namespaces, and looks like it was barfed onto the street (not so human readable).
Now, the benchmark. I will be writing and posting the benchmarking code, but the gist is this: load up the file into memory to eliminate the hard drive as a bottleneck, execute 10 iterations of parsing the document enough times to constitute at least 100MB of data. I intend to use the fastest configuration of the parser as possible, not the safest, and will keep the code open to allow suggested improvements from the community.
Popularity: 5%
Tango has landed XML support in the tango.text.xml package. Current highlights include a pull parser, a DOM parser, and a SAX parser, as well as a budding XPath like package.
What makes these different you ask? Why another damn XML parser? Glad you asked. These components are intended to be high-speed, non-allocating tools that can be used at a server or appliance level with much less overhead than other solutions. For example, the SAX parser needs just a few KB of memory over and above the size of the content being parsed.
If you need a fast XML parser, check them out. I am still writing up my benchmarking output, so stay tuned for a post on that shortly.
What is Tango? Tango is an alternate standard library for the D Programming Language.
Popularity: 11%
