Day 26 of 60: Instrumenting Sendmail queue file creation (pt 1)

I've (finally) got Sendmail built, zones configured, DTrace working for functions declared static, and a mechanism for creating test SMTP sessions.

So it's time to start putting this together, instrumenting Sendmail, and seeing whether or not I can use this to prove (or disprove) some common advice given when configuring Sendmail.

First, I'm going to look at queue directories.



More specifically, how many of them do you need?

I've written previously about Sendmail queues.

Conventional wisdom is that if you're going to be queueing a lot of messages with Sendmail then it makes sense to make sure you have multiple queue directories. There are two benefits to this.

The first is that, at file creation time, it helps guard against any inefficiencies in the filesystem that you're using with respect to directories that contain a large number of files.

This is especially useful if you're going to put these queue directories across multiple disks, since the IO load is going to be split across those disks, and disk controllers.

The second is that, at queue processing time, each queue runner can process a smaller part of the queue, and so work more efficiently.

Right now I'm testing the first of these assumptions -- does having multiple queue directories make an appreciable difference to the amount of time it takes to write a message to the queue?

But first, a disclaimer.

Disclaimer: I do not suggest that you use any results shown here to make decisions about your own production system(s). Rather, use the methods shown here to generate data about your systems, and make a decision accordingly.

There are a number of things about these tests that mean that they're not directly applicable to the real-world.

1. Everything's running off one disk, and hence one disk controller.
2. I haven't gone out of my way to tune the Solaris install.
3. The SMTP traffic is being generated by the same machine that's receiving it.
4. The test machine is running a number of services that you wouldn't expect to see on a real mail server. For instance, a GNOME desktop, web browser, and so on.

However, for this test that's not necessarily important. More specifically, the numbers that I get out of the test aren't as important as the how the numbers compare to each other. It's the relative differences (if any) that are going to be interesting, not the actual differences.

I've made a few changes for this test. First, the Sendmail configuration file that I'll be using has been changed to expect directories under /var/spool/mqueue/qdir*. This makes it easier to test multiple queues -- in the single queue directory case I'll create /var/spool/mqueue/qdir00. In the multiple queue case I'll create qdir01, qdir02, qdir03 and so on. This is change 1107.

There are two new DTrace Sendmail probes, which fire when the code that creates a new entry in the queue starts and stops. This is change 1108.

I've also changed the Sendmail startup script that's used in the zone. The default script included a -q option, which means that Sendmail will periodically process the queue. The new script removes that, which means that the queue will only be processed when explicitly requested, and removes a possible source of 'jitter' in the results. This is change 1109.

The next post will go through the DTrace script to instrument this, and run the tests to get some results.