Benchmarking: Methodology

Posted: December 21st, 2008 under ActionScript 3.0 Benchmarks.

I know most of the die-hard critics our there are going to want to see my code, and that is perfectly acceptable. As I have already stated myself, many of the people trying to figure out this black box do a poor job. Just do not expect all of my code to be made public, if you want to grab the source it is available here. These are the core principles I implemented when conducting these benchmarks:

  • One test is never enough; it is not statistically relevant and other computer processes could interfere with your results.
  • Each trial needs to take long enough so that clocking speed does not affect your accuracy. Typically, I set my frame rate for 60 frames a second, which is true for these benchmarks. At 60 frames a second Flash is going to try to keep the frame rate at 0.016 seconds a frame (roughly). When Flash cannot get all of its calculations done in one frame, it drops frames until it completes is computations. Realistically you can only measure time in frame rate increments. Furthermore, I still believe that most computers out there can only measure time in millisecond increments (0.001).
  • Testing should be conducted after program is done initializing and give enough time in between tests for the CPU to recover and for Flash complete a garbage collection cycle. Otherwise, your tests can interfere with each other since they would be competing for the same resources.
  • Testing should be conducted in an appropriate environment. Conduct tests with the same settings you would give your applications, and running an application from the Flash editor is very different from running the application in a website or flash player.
  • Computer architectures are different so you need to test on more than just one machine. Even with Flash having a virtual machine, different computers and operating systems have varying speeds (at some level virtual machines have to bind to hardware). Regardless, relative speed of calculations is more important than absolute speed; absolute speed is only true for one type of machine, while relative speed should be true for almost everyone.

I have compiled the results from four different computers into my findings:

  • Computer A: Gateway Desktop, Intel Pentium 4 chip, 2.27 GHz, 1 GB RAM, Windows XP SP3
  • Computer B: Sony VAIO Laptop, Intel Duo T5750, 2×2.00 GHz, 3 GB RAM, Windows VISTA
  • Computer C: Dell Latitude Laptop, Intel Duo U2500, 2×1.20 GHz, 1 GB RAM, Windows XP Pro
  • Computer D: Dell Inspirion 531 Desktop, AMD Athlon 64 Duel Core, 2.81 GHz, 2.93 GB RAM, Windows XP SP3

I am sure that this is an imperfect sampling of computers, I did my best to vary the age and types of machines based on what I had available. I will only include results if they are significantly different from computer A, the complete results are available via link on the results page. Links accompany each benchmark if you want to run further tests yourself. When you run a benchmark you will see that I wait at least 5 seconds after the application is done loading to start testing, then wait at least 5 seconds between trials, and I run 15 trials carefully monitoring the standard deviation between trials to ensure other processes on my machine are not interfering with the results. While relative speed is more important than absolute speed, some tests have larger overhead than others, which can skew or obscure results. I will point out cases with substantial overhead in my analysis, as well as other imperfect conditions. Given the volume of the benchmarks, I have divided them into separate categories:

Continue to Operation Speed

This post is part of a series:
Introduction
Methodology <-- You are here
Operation Speed
Data Structure Speed
Graphics Speed
Results

Bookmark this on Delicious

2 Comments »

  • Comment by nil — September 15, 2010 @ 4:59 am

    1

    “At 60 frames a second Flash is going to try to keep the frame rate at 0.016 milliseconds a frame (roughly).”

    you mean 0.016 seconds. right?


  • Comment by Stephen Calender — September 30, 2010 @ 10:58 pm

    2

    Yes, obviously. Thank you for catching that mistake, it has been corrected.


RSS feed for comments on this post. TrackBack URL

Leave a comment