JatMark — Java Performance Benchmark

java -Xmx256m -jar JatMark.jar bunny.xml

Dear Hardware Reviewers,

While there are many hardware testing sites, me and many of my programmer friends find it difficult to make an informed choice based on your reviews. Why is that? Mostly because the benchmarks most of you use give little insight into Java performance on particular platform. Java is of course very well known for its server-side strenghts, but in recent times it is becoming increasingly popular in fields of technical/scientific computing as more and more people in physics/math/simulation research communities turn to Java for its portability, quick development time and easy debugging. Java is also a part of many university curriculums and most current to-be programmers will encounter it as their first language.

Unfortunately, while relative performance of different CPUs _and_ different Java virtual machines is usually known, it's still difficult to apply facts like "AMDs excell at FP" and "Intel is better at memory intensive operations" to Java, as implementation details of particular JVM pretty much decide how these strengths are actually utilized.

In order to rectify the situation somewhat, I'd like to propose a new benchmark, which -- while obviously can't be claimed to magically express Java performance in just one number -- still stresses JVM in sufficiently many areas to draw conclusions on its overall performance regarding computationally intensive processes.

The benchmark is easy to use and is based on the actual application I wrote for my PhD research, so it definitely cannot be claimed to be of "synthetic" variety.

What it does is to render a single image of moderately complex scene (~70K faces) using advanced global illumination algorithms. Specifically, it performs following steps (it's a bit technical, you may wish to skip this part):

1. Loads scene description and object dataset using XML and text file parsers.
2. Computes average vertex normals, which are used later for Phong interpolation.
3. Prepares ray traversal data structures to decrease the number of ray-object intersection tests (two-level hierarchical grid is used).
4. Emits global and caustic photons from area light source using Quasi Monte Carlo techniques (Halton sequence) and stores them in density-controlled photon maps based on KD-trees.
5. Converts KD-trees to heap representation and balances them.
6. Precalculates the irradiance estimates at photon hit points using convex hull "gift wrapping" algorithm.
7. Performs the first rendering pass which calculates irradiance values at important locations using final gatherings with 310 rays.
8. Performs the second rendering pass which is using distributed ray tracing (to compute soft shadow from area light source), irradiance interpolation (Ward's algorithm) and direct caustic estimation to generate final image.
9. Processes the image with tone mapping operator (simple gamma correction)
10. Stores the result in a PNG file. The image can be used to verify that the test succeeded.

All pixels are adaptively antialiased with maximally 41 rays.

If you need more technical details I would be happy to provide them, but what it all boils down to is that the renderer:

• Uses a lot of memory (around 128MB, but actually reserves 256MB to give garbage collector some elbow room), both in sense of constant creation and destruction of objects as well as extensive traversals of various, non-localized data structures (trees, vectors, lists etc).
• Is very floating point intensive, including trigonometrical operations and matrix math.
• Uses two- and three-dimensional arrays extensively.
• Runs for multiple minutes on modern hardware which should make it usable in the years to come, yet is not too time consuming (~10 minutes on AthlonXP 3200+).

JatMark number is simply the total number of seconds spent on the computation and on my machine (Athlon Thunderbird B 1GHz overclocked to 1.1GHz) is equal to 1599. Unfortunately, Java does not allow the measurement of actual CPU time used (as opposed to wall clock time), but most operating systems provide easy means to do it (e.g. "time" command in Linux and MacOS X).

The final, important issue to be discussed is the choice of JVM to be used in testing, as they obviously differ (sometimes significantly) in their performance. The first constraint here is that the benchmark requires version 1.4.0 or newer, as it uses ImageIO class package among other things. Another is that we would preferably see it running on as many systems as possible. So, Sun's 1.4.1 seems to be the most reasonable choice, even though IBM's 1.4.0 is around 25% faster. In any case, I guess it would be desirable to pair particular JatMark result with name of JVM used to obtain it, as JatMark is in fact the measurement of particular JVM's performance on given platform.

This is more or less it. JatMark is freely downloadable from

http://wfmh.org.pl/~thorgal/jatrac/JatMark.zip

The archive contains README file that explains how to run the benchmark, but should you have problems running it, please don't hesitate to ask me for help.

Best Regards,
Miloslaw Smyk