Day 57 of 60: Developer benchmarks (pt 2)

As I explained on day 55, I've been comparing GCC and the Sun Studio compiler on Solaris, to GCC running on FreeBSD to see if there are any significant differences in the time taken to compile the applications, and if there is, whether that difference is reflected in the time taken by the applications to run. I used gcc 3.4.3 on both systems.

Testing Sendmail's build

For these results I used the source code for Sendmail 8.13.8, which has been recently released. The base build consisted of checking out the code from the repository, and then running the following command in a loop 20 times, in a script(1) session to capture the output.

/usr/bin/time sh Build -c

The -c option ensure that anything built by a previous build is removed.

gcc, no optimisation, Solaris and FreeBSD

This is the default build type for Sendmail. I had to make no changes to the default build on FreeBSD.

On Solaris it was necessary to first add /usr/sfw/bin and /usr/sfw/sbin to $PATH.

gcc, optimisation, Solaris and FreeBSD

On both operating systems I created a site.config.m4 file with the following entry:

define(`confOPTIMIZE', `-O2')

I then ran the 20 iteration build.

cc, no optimisation, Solaris

To use Sun's compiler it is necessary to explicitly specify the name of the compiler binary. You must also configure Sendmail to build with a different mechanism for determining dependencies.

To do this, I created a site.config.m4 that contains the following lines:

define(`confCC', `cc')
define(`confDEPEND_TYPE', `Solaris')

and then ran the 20 iteration build.

cc, optimisation, Solaris

This needs the same site.config.m4 file as in the non-optimised case. In addition, the following line must be added.

define(`confOPTIMIZE', `-fast')

I then ran the 20 iteration build.

Results

Here's the mean run time across the twenty sets of results for each test. All times are in seconds, and wall clock.

	gcc		cc
	-O*	-O2	None	-fast
Sendmail / Solaris	34.6	39.6	53.3	65.1
Sendmail / FreeBSD	29.8	35.7
* I'm aware that this is not 'no optimisation'. However, it's the default optimisation level if no other is requested when building Sendmail, so it seemed reasonable to leave it at the default.

That's quite a surprise. I thought FreeBSD might have the edge, but I wasn't expecting it to be quite so large. Nor was I expecting Sun's compiler to be so much slower than gcc. As ever, running the data through ministat provides helpful information.

x freebsd-gcc-sendmail.real
+ sol8-gcc-sendmail.real
* freebsd-gcc-O2-sendmail.real
% sol8-gcc-O2-sendmail.real
- sol8-cc-sendmail.real
@ sol8-cc-fast-sendmail.real
: = Mean
M = Median
+----------------------------------------------------------+
|x + * - @ |
|x + * - @ |
|x + * - @ |
|x + * - @ |
|x + * % - @ |
|x + * % - @ |
|x + * %% - @ @|
|x + * %% - @ @|
|x + * %% - @ @|
|x + ** %%% -- @ @|
|xx ++** %%% -- @ @|
|xx ++** %%% -- @ @|
|xx ++** %%% --- @ @|
|:| |
| M: |
| :| |
| |:| |
| :| |
| M:||
+----------------------------------------------------------+

	N	Min	Max	Median	Mean	Stddev
x	20	29.63	30.44	29.765	29.823	0.20227912
+	20	34.4	35.5	34.4	34.62	0.41498256
Difference at 99.5% confidence
4.797	+/-	0.342619
16.0849%	+/-	1.14884%
*	20	35.37	35.91	35.57	35.585	0.1455118
Difference at 99.5% confidence
5.762	+/-	0.184929
19.3207%	+/-	0.620088%
%	20	39.5	40.6	39.6	40.02	0.51971652
Difference at 99.5% confidence
10.197	+/-	0.413893
34.1917%	+/-	1.38783%
-	20	53.1	54.3	53.3	53.5	0.43889814
Difference at 99.5% confidence
23.677	+/-	0.358658
79.3917%	+/-	1.20262%
@	20	65	66.1	65.1	65.41	0.4897905
Difference at 99.5% confidence
35.587	+/-	0.393278
119.327%	+/-	1.31871%

This all suggests that, on this workload at least, FreeBSD+gcc outpaces Solaris+gcc, both of which are significantly faster than Solaris+Sun Studio cc.

Granted, the differences between gcc on both platforms is, in real terms, quite small. A difference of a few seconds every compilation is not going to impinge on my productivity. However, the larger differences, between Sun's compiler and gcc are, in my opinion, more significant. Over a week that sees perhaps 10 recompilations a day that's an extra 19 minutes spent waiting for the compiler. Across a month that's approximately an hour and a quarter of developer time wasted.

Of course, there are ways to improve this that don't involve changing compilers. For example, Sendmail's build infrastructure makes no attempt to compile multiple source files in parallel. I've seen this bring benefits even on single processor machines (where the amount of time spent in IO is sufficiently large that two concurrent compilers don't step on each other's shoes, as while one is compiling the other is frequently waiting for IO to complete). On a multiprocessor system, such as the one that I'm using, compiling multiple source files simultaneously would yield significant benefits, at the cost of a more complicated build system.

Also, of course, these results only reflect performance compiling Sendmail, a reasonably large C application. Compiler performance on other applications, or other languages (C++ say) may be significantly different.

To test at least part of that hypothesis my next round of results are going to look at the time taken to build and test Perl 5.8.8.