Day 55 of 60: Developer benchmarks (pt 1)

The last week has been quite busy with work that's not related to this project. Mindful that the 60 day time limit is almost up, and aware that I've not done any actual benchmarking of this workstation -- vis a vis "How does Solaris on this hardware compare against another OS on this system?", I've started doing some investigation.

Sun bill this machine as a developer workstation, so I thought I'd look at how speedy it is at carrying out tasks that developers do. I also thought it would be worthwhile carrying out a few performance benchmarks relating to a real-world application that I currently run on Solaris.

Comparison OS

The operating system I'm going to compare Solaris 10 against is FreeBSD 6.1. I'm very familiar with FreeBSD (more so than with Solaris) having used it on a daily basis for the past 13 years -- the IBM T42 laptop that I'm writing this on is running FreeBSD right now.

This familiarity with FreeBSD could give me an unfair advantage -- it's much easier for me to tune a FreeBSD system for performance than a Solaris system. To avoid this I'm not going to perform any special customisation of either OS, using essentially out-of-the-box configurations. I expect this to be relatively fair, as it's unlikely that a developer workstation is going to be extensively tweaked for maximum performance.

Developer time savings

As a developer, one of the things that chews up a chunk of my time is waiting for an application to recompile.

Accordingly, I'm going to look at how well C compilers run on both operating systems, with and without their optimisation settings enabled. At this point I'm not too interested in the speed of the generated code (I'll test that a little later), just how long it takes from running make to having an application built.

On Solaris I have two compilers available to me. The system installed gcc-3.4.3 in /usr/sfw/bin, and I have installed the Sun Studio 11 compiler.

FreeBSD also supports multiple compilers, available through the ports collection. I will just be using gcc for these tests, as I don't have experience with using icc, tendra, and others.

This will let me compare the run time of gcc on both platforms. It will also let me see how much faster (or slower) Sun's compiler is on Solaris when compared to gcc.

The two applications I recompile most frequently are Sendmail and Perl. I use both of them in my day job, and regularly dive in to the Sendmail code to determine the causes of behaviour that I'm seeing in production, or to add new features -- such as DTrace probe support.

I also use Perl extensively day to day, and automatically track changes made to Perl's code repository to build them locally.

Methodology

I will compile Sendmail and Perl on Solaris using the available compilers. Each compilation run will be timed using /usr/bin/time to give wall clock times. For these tests I'm interested in how long the developer has to spend twiddling their thumbs waiting for the application to be regenerated, not the number of microseconds spent in system calls.

There will be two tests of each compiler, with and without optimisation. Each test will consist of 20 'clean and rebuild' cycles, run in a loop. I won't do anything 'clever', like putting the compiled object files on a memory-backed filesystem for speed, since this is unlikely to happen in the real world. So all the usual bottlenecks of CPU, memory, and disk will apply.

For the non-optimized case I will use whatever the application suggests as the default command line.

For gcc that is probably going to be gcc or gcc -O. For the Sun Studio Compiler I expect that to just be cc

For the optimised case I will explicitly set the optimisation level. For gcc I will use -O2. I am aware that there are several other optimisation settings that can be enabled. I'm also aware that, historically, these have led to buggy code. That may not be the case any more, but I don't track gcc development very closely, and don't want to have to chase down bugs caused by the compiler.

For the Sun Studio Compiler I will use -fast. Much like gcc's -O2 option this is a meta-option that enables all the optimisation flags that Sun think are going to be most useful, or that act as a good starting point for further tuning.

The next step is to install FreeBSD...