raison d’être

It started when I read a number of posts at Jonathan Schwartz's blog (in order: here, here, here, here, and here).

Jonathan is Sun's CEO (although he wasn't at the time he started this series). The essence of it is that Sun are so stoked about their new hardware that:

So... here's an invitation to developers and customers that don't want to move to Solaris, want to stay on GNU/Linux, but still want to take advantage of Niagara's (or our Galaxy system's) energy efficiency - click here, we'll send you a Niagara or Galaxy system, free. Write a thorough*, public review (good or bad - we just care about the fidelity/integrity of what's written - to repeat, it can be a good review, or a poor review), we'll let you keep the system. Free.

That sounds like a good deal to me. So I started thinking about how I might take advantage of this offer.



I do a reasonable amount of open source development. Historically a lot of that has been on the FreeBSD OS. However, my day job is all Solaris (Solaris 8 as it happens, with a transition to Solaris 10 somewhere down the pipe). Having a Solaris 10 box at home will give me experience with Solaris 10, and some of its spiffy features. I'm thinking particularly of dtrace, ZFS, and zones. And since FreeBSD works on (some of) Sun's hardware I'll be able to do a dual install to compare and contrast the two systems.

That's not really enough though. Frankly, fiddling with operating systems just for the sake of it isn't especially enthralling, so I started thinking about some of the technologies in Solaris that look especially enticing. For me, that's the previously mentioned dtrace, ZFS, and zones. They're technologies that I've been reading a lot about over the last 6-8 months, but I haven't had a chance to get any real hands-on experience with them.

A large part of my day job also involves two goliaths of the open source world; Sendmail and Perl, and I spend a chunk of my day engineering systems that use both of these tools, and analysing production systems to see where the bottlenecks are so that they can be removed (or, at least, pushed to other parts of the system). This is technically quite challenging.

dtrace is supposed to solve all that. Reading blogs written by various dtrace developers I'm becoming increasingly interested in the technology, but I haven't had a chance to use it in anger yet. And FreeBSD is gaining dtrace support as well, which means it's going to see wider adoption outside of Sun shops, and hopefully spur on advances in systems troubleshooting.

Although dtrace (apparently -- remember, I have no first hand experience of it yet) lets you instrument unmodified systems in ways far more interesting than truss(8), it seems to me that the real benefit comes of having applications that have been modified to support what dtrace calls "probes".

These probes allow the application to say when events happen that are interesting from the application's viewpoint, not the operating systems. For example, truss(8) can tell you that the application opened a file. But that's more interesting information if you know that it's a Sendmail queue file. More interesting still if you can use dtrace probes to "decorate" that information with extra metadata maintained by the application. Or being able to map a send(2) call to a specific point in the SMTP conversation.

You can truss a Perl application and see that it's allocating memory, making system calls, and so on. But you can't easily map that information back to the Perl source code of the running application. How much more useful would it be when troubleshooting a program that's allocating a lot of memory if you can trace the malloc() call back to the line in the Perl program that triggered it. There's been some work on adding dtrace probes to Perl already, but the work seems to have stopped.

So a plan began to form.

Over the next 60 days I'm going to:

• Receive a system from Sun


• Install both FreeBSD and Solaris 10 on it


• Use Solaris zones to create several virtual servers that will send e-mail back and forth


• Instrument the open source version of Sendmail with dtrace probes, and make those code changes public


• Learn D, the dtrace scripting language, and use it write useful probe scripts for use with Sendmail


• Benchmark Solaris 10 and FreeBSD carrying out the same operations (e.g., building applications like Sendmail and Perl from source)


• Document the whole thing here

If I get the time I also plan to build on some pre-existing work to add dtrace probes to Perl. I'm not as familiar with the Perl codebase as I am with the Sendmail codebase though, so that's not a target I'm going to guarantee hitting.

The most recent e-mail from Sun said:

scheduled for shipment on July 05, 2006

so hopefully everything will kick off tomorrow.