Day 10 of 60: First probes added to Sendmail

Following Monday's info dump about queues, I've spent some time over the last few days reading the DTrace documentation in detail. In particular, the Solaris Dynamic Tracing Guide. This is the DTrace handbook, with a great deal of information about how to use DTrace.

It also contains the information about how to add custom DTrace probes to user applications. I was a bit surprised when I first read that, as it's only a couple of pages long.

It turns out that adding DTrace probes really is that simple...



The first thing I did was write a definition file for the probes. This was commit 1057. If you look at provider.d you'll see that I've defined two probes for the moment.

The first is going to fire when Sendmail starts reading a queue directory looking for items to deliver. The second will fire when Sendmail finishes reading the queue directory.

With the provider defined I needed to add the probe points.

At the moment sendmail/queue.c:gatherq() looks like being the function to instrument. Accordingly, I've added DTRACE_PROBE() calls at the apropriate points.

The other piece of the puzzle is for provider.d to be processed by dtrace(1) to generate provider.o. This then needs to be added to the files that are linked in to the Sendmail binary in order for the probes to work.

You can see the work I had to do to the Sendmail build process as part of the same commit, 1058, in the changes to sendmail/Makefile.m4. That looks like it will need some cleanup, and for consistency I should be wrapping all these changes in

#ifdef _FFR_DTRACE
...
#endif

checks. But that would complicate things at this point, where I'm trying to do the bare minimum to get things to work.

I did bump in to one issue. If you review change 1058 you'll see that I had to remove the static declaration from the definition for gatherq(). Without that change things fail at link time -- provider.o can't see the gatherq symbol. I'm not completely happy about this, so more investigation needs to happen there.

And does it work? Well, everything compiles, which is a good start. The next step is to configure the test zones that I created earlier to use the new Sendmail binary. Then testing can start in earnest.