Day 58 of 60: Developer benchmarks (pt 3)

Yesterday I looked at performance compiling Sendmail on Solaris and FreeBSD using Sun's compiler (on Solaris) and gcc (on both systems).

In the tests gcc come out handily ahead, with gcc on FreeBSD being 16% faster than gcc on Solaris with low optimisation options, and 12% faster than gcc on Solaris with optimisation turned on. Sun's compiler was over twice as slow as gcc on either system.

Today I've been looking at the time taken to compile Perl on both systems, using both compilers, with and without optimisation.

Building Perl

For these results I used the source code for Perl 5.8.8, the most recent release of Perl designed for production use. I imported the source code in to my Subversion repository and built from that. Here's the Perl 5.8.8 code.

Perl's build is slightly different from Sendmail's. There's a configure step, then a build step, then an explicit test step. The explicit test step is useful -- partly as a way of validating that the built code works, but also as a crude benchmark. The tests are run under the new binary, so it's a quick-and-dirty way of determining how the speed of Perl built with different compilers and different optimisation options varies.

The base build consisted of checking out the code, and then running the following commands in a loop 20 times, in a script(1) session to capture the output.

sh Configure -des
/usr/bin/time make
/usr/bin/time make test
make distclean

This generates separate timings for the build and test step, which I report on later. The -des option to Configure tells it to use the defaults for this platform and compiler, and to be relatively silent about what it's doing. The make distclean step cleans out all the generated code prior to starting the next run.

A note on the gcc builds

There's a problem building Perl with gcc on Solaris. Part way through the build a fatal error

perlio.c: In function `PerlIOStdio_invalidate_fileno':
perlio.c:2899: error: structure has no member named `_file'

occurs. Some sleuthing showed me that Alan Burlison had reported this, and that change 27722 fixed it. Accordingly I've applied that patch to the source code I'm building.

gcc, no optimisation, Solaris and FreeBSD

As the base build, but with -Dcc=gcc added to the Configure command line to specify the use of gcc.

On Solaris it was necessary to first add /usr/sfw/bin to $PATH.

gcc, optimisation, Solaris and FreeBSD

As the gcc build, but with -Doptimize='-O2' added to the Configure command line.

cc, no optimisation, Solaris

As the base build, with no additional parameters.

cc, optimisation, Solaris

As the base build, but with -Doptimize='-fast' added to the Configure command line.

Build ("make") results

Here’s the mean run time across the twenty sets of results for each test. All times are in seconds, wall clock, and rounded to one decimal place.

	gcc		cc
	-O*	-O2	None	-fast
Perl build / Solaris	89.7	104.3	148.2	194.0
Perl build / FreeBSD	79.1	95.0
* I'm aware that this is not 'no optimisation'. However, it's the default optimisation level if no other is requested when building Perl, so it seemed reasonable to leave it at the default.

Based on the previous results compiling Sendmail this is pretty much what I expected. gcc is considerably faster than Sun's compiler, and gcc on FreeBSD is faster than gcc on Solaris. The relative speed differences are also similar, with Sun's compiler taking a veritable age to do the job.

What was interesting to see was the consistency of the times taken by the various compilers. This is most obvious viewed in a ministat plot and associated statistics. In particular, note the small standard deviation for each result set.

x freebsd-gcc-perl.build.real
+ sol8-gcc-perl.build.real
* freebsd-gcc-O2-perl.build.real
% sol8-gcc-O2-perl.build.real
- sol8-cc-perl.build.real
@ sol8-cc-fast-perl.build.real
: = Mean
M = Median
+----------------------------------------------------------+
|x % @|
|x % @|
|x * % @|
|x * % @|
|x * % @|
|x * % @|
|x * % @|
|x * % @|
|x + * % - @|
|x + * % - @|
|x + * % - @|
|x + * % - @|
|x ++ * % - @|
|x ++ * % -- @|
|x ++ * % -- @|
|x ++ * % -- @|
|x ++ * % -- @|
|x ++ * % -- @|
|x ++ * % -- @|
|x ++*** % --- @|
|: |
| :| |
| : |
| : |
| :| |
| :|
+----------------------------------------------------------+

	N	Min	Max	Median	Mean	Stddev
x	20	78.62	79.53	79.115	79.1145	0.26009057
+	20	89.4	90.2	89.65	89.675	0.2531382
Difference at 99.5% confidence
10.5605	+/-	0.269357
13.3484%	+/-	0.340465%
*	20	93.23	97.66	94.94	95.027	0.82294785
Difference at 99.5% confidence
15.9125	+/-	0.640528
20.1133%	+/-	0.809622%
%	20	104.1	104.6	104.25	104.28	0.13992479
Difference at 99.5% confidence
25.1655	+/-	0.219187
31.809%	+/-	0.277051%
-	20	146.5	150.4	148	148.195	1.1623366
Difference at 99.5% confidence
69.0805	+/-	0.883962
87.3171%	+/-	1.11732%
@	20	193.7	194.2	193.95	193.965	0.14608937
Difference at 99.5% confidence
114.851	+/-	0.221391
145.17%	+/-	0.279837%

Test ("make test") results

The Perl tests provide an opportunity to see if there's any performance improvement in the binaries that the compilers generate. This test is little more than indicative, as the tests generally consist of running hundreds of small applications, so the overhead of starting new processes is likely to dwarf any performance benefits in the binaries. However, this may show something interesting.

The results are:

	gcc		cc
	-O	-O2	None	-fast
Perl test / Solaris	304.8	296.2	302.6	306.1
Perl test / FreeBSD	293.4	287.8

Now that's downright interesting.

On FreeBSD, the code produced by gcc is faster than the code produced by gcc on Solaris, irrespective of optimisation. In fact, the optimised code produced by gcc on Solaris is slower than the unoptimised code produced by gcc on FreeBSD. This may be down to gcc producing inferior code on Solaris than on FreeBSD, or it may be down to FreeBSD being faster.

Given that Sun's compiler produces slower code than gcc on FreeBSD, and that the optimised code produced by gcc on Solaris is faster than the optimised code produced by Sun's compiler on Solaris I suspect that gcc may not be to blame here, and FreeBSD is, at least in this test, quite a bit faster than Solaris. Keep in mind, too, just how much longer the Sun compiler takes to produce a binary that runs slower.

The ministat results paint an interesting picture. Note that the standard deviation for these results is larger than for the compilation results, probably reflecting the extra IO overhead.

x freebsd-gcc-O2-perl-test.real
+ freebsd-gcc-perl.test.real
* sol8-gcc-O2-perl.test.real
% sol8-cc-perl.test.real
- sol8-gcc-O2-perl.test.real
@ sol8-cc-fast-perl.test.real
: = Mean
M = Median
+----------------------------------------------------------+
| ++ + - -- % % @ |
| x ++ + - -- %% % @@@ @ |
|xxxx x +++-x- --- %% %% @ @@@ @ |
|xxxxxxx xxx x ++++---- ---- + %%%%%%@ @@@@@@@@ @|
||___M__:______| |
| |_M:__| |
| |_:M_| |
| |_M:__| |
| |_:M_| |
| |___:___| |
+----------------------------------------------------------+

	N	Min	Max	Median	Mean	Stddev
x	20	284.52	294.78	286.555	287.807	3.3765197
+	20	292.22	299.21	292.985	293.393	1.482303
Difference at 99.5% confidence
5.586	+/-	2.73673
1.94088%	+/-	0.95089%
*	20	293.9	297.8	296.65	296.19	1.2961075
Difference at 99.5% confidence
8.383	+/-	2.68416
2.91272%	+/-	0.932626%
%	20	300.4	305.9	302.45	302.585	1.4401297
Difference at 99.5% confidence
14.778	+/-	2.7243
5.13469%	+/-	0.946571%
-	20	293.9	297.8	296.65	296.19	1.2961075
Difference at 99.5% confidence
8.383	+/-	2.68416
2.91272%	+/-	0.932626%
@	20	303.2	311.9	305.95	306.085	1.7986179
Difference at 99.5% confidence
18.278	+/-	2.83924
6.35078%	+/-	0.986508%

I'm not sure yet where the speedup comes from. As I say, this test involves executing a large number of files, and it may that FreeBSD is better optimised for this. My next set of tests will run a resource intensive Perl application on both operating systems with a total of 6 versions of the Perl binary, corresponding to the different compiler choices and optimisation options, to see if any intrinsic difference in performance between the two systems can be identified.