DTrace
DTrace is remarkably useful. The ability to trivially trace what an application is doing with a very low overhead completely changes how you go about investigating production issues.
The fact that it's so easy to implement static probes in a C application is also a huge benefit. Problems fighting with Sendmail's build infrastructure aside, I was surprised just how easy it was to add instrumentation and immediately see results from it.
With DTrace implementations appearing on (at least) FreeBSD and Mac OS X, and with other open source projects, such as Postgres adding support for DTrace I think it stands to become an increasingly useful part of a developers toolset.
Zones
Zone management on Solaris is nice. Certainly the bits of it that I've had to deal with have been.
FreeBSD's jails are similar, but you don't get tools to easily manage them out of the box (no equivalents to
zonecfg and zoneadm. There are a number of third party equivalents though -- FreshPorts has a useful list of jail admin utilities.What zones do have over and above jails is resource control. As well as being able to assign zones a certain amount of disk space and networking QoS guarantees (which you can also do with FreeBSD jails), zones also allow you to apply CPU and memory resource limits on a per-zone basis -- something that I don't believe FreeBSD supports at the moment.
At this point you start to get close to the full virtualisation environments of a tool such as VMWare or Xen. I think it remains to be seen at the moment whether one or other approach will become dominant. Certainly the Solaris/FreeBSD approach is probably simplest if you want to virtualise multiple copies of the same operating system on a single hardware instance.
However, with Solaris now supporting x86 and amd64 systems, the question is now a little more complicated. Different operating systems have different strengths, as the results from the last few days have shown. A VMWare approach, allowing you to run multiple concurrent different operating systems on the same hardware instance may prove to be more useful in the long run, even if the administration overhead is perhaps a little higher. Time will tell.
Solaris annoyances
There are a couple of things that have been particular thorns in my side while I've been working on this project.
Integration
I think my main gripe for the moment is that, although the installation proceeds smoothly, and you go from DVD image to working X system in a relatively short space of time, there's still an amount of customisation that needs to be done under the hood. There's not quite sufficient integration between the two worlds.
I can get over my dislike of GNOME, at least enough to use the Java Desktop as a development environment. What's missing is better coupling between the installation and the desktop. For example, the questions during install that specify keyboard type and layout don't propogate their answers through to the desktop. Further, the GNOME tools for setting these options don't work -- you have to edit
xorg.conf by hand.I don't mind having to do this level of post-install customisation. But it's not documented anywhere obvious, and it makes the integration that has been carried out look half-hearted.
This is also reflected in the two different windowing environments that you can log in to. There are menu items in the CDE, such as administrative tools for managing user accounts, that are not present in the GNOME desktop. As a result, neither of them feel like the "correct" desktop to use.
Third party software packaging
As a developer who spends a lot of time working with open source software, it's important to be able to quickly download and install new open source applications and libraries. The ability to rapidly explore new tools and projects to see if they're a fit for requirements helps keep my productivity high. Of course, to ease the maintenance headache these really need to be packaged.
I recounted my issues with Blastwave and pkgsrc earlier. Suffice to say that Solaris would benefit from having much more open source software easily available. Blastwave does a good job, but I prefer the much more open approach taken by pkgsrc, where the barrier to contributing is much lower, and it's easier to poke around, rebuild applications from source if necessary, and verify that what you get is what you expect.
Sun, please just donate a couple of Solaris 10 systems to the pkgsrc developers and be done with it.
What didn't get done?
A little over 60 days ago I wrote:
Over the next 60 days I’m going to:
- Receive a system from Sun
- Install both FreeBSD and Solaris 10 on it
- Use Solaris zones to create several virtual servers that will send e-mail back and forth
- Instrument the open source version of Sendmail with dtrace probes, and make those code changes public
- Learn D, the dtrace scripting language, and use it write useful probe scripts for use with Sendmail
- Benchmark Solaris 10 and FreeBSD carrying out the same operations (e.g., building applications like Sendmail and Perl from source)
- Document the whole thing here
I'm quite pleased that I managed to hit all those targets, even if it's been a bit of a squeeze to fit everything in over the last few days.
I also wrote:
If I get the time I also plan to build on some pre-existing work to add dtrace probes to Perl. I’m not as familiar with the Perl codebase as I am with the Sendmail codebase though, so that’s not a target I’m going to guarantee hitting.
Unfortunately that's not a target I hit -- there were just too many other things going in for me to be able to get the time to do that.
Future work
If Sun let me keep the machine there are three main areas I'd like to look at.
First, continue the work adding DTrace probes to Sendmail. The probes that I've added have already proven to be useful, but the work could use a little cleanup, and better documentation. There are also many other places where probes could usefully be added to Sendmail, and I think there need to be a few ground rules laid to ensure that the probe names are consistent.
Also, they might help expose other bugs.
Second, do some more digging in to the various compiler optimisation options for cc and gcc. I was very surprised at quite how badly Sun's compiler did in the tests of the last few days, and it may be the case that a particular set of optimisation options can help. I'd also like to see whether making the tests multi-process or multi-threaded is of any particular benefit on Solaris.
Third, add DTrace instrumentation to Perl. Now that I've got some experience with DTrace I have a better idea of how this work might proceed and still have a relatively low impact on the speed of the Perl interpreter.
Great wrap up.
ReplyDelete