Day 27 of 60: Instrumenting Sendmail queue file creation (pt 2)

It's time to run an instrumented Sendmail, throw some messages at it, and see how it performs. Specifically, does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue?



To do this, I need to instrument Sendmail's queue creation functions, and track the amount of time it takes for Sendmail to create a new entry in the queue, while also tracking the total number of entries created.

I can then take these figures, group them in 'buckets' of 'n' messages each, and determine the average amount of time taken to create each message.

The DTrace script

This is implemented in queue-create.d. It tracks the time when queue creation starts, and again when it ends. This time is averaged, divided by 1,000 (to give uSec, not nanosec) and assigned to @time.

This shows a wrinkle that I discovered that wasn't clearly documented -- you can't run DTrace in a non-global zone. It must be run in the global zone, and you must use a zonename DTrace variable in your predicates if you want to restrict the tracing to a particular zone.

Preparing Sendmail

The tests are going to be run against Sendmail in the relay-zone. Before each test Sendmail was prepared to make sure that the tests run in a consistent environment.

First, Sendmail was stopped. Then the mailqueue was cleared out:

# rm -rf /var/spool/mqueue/*

Recall that Sendmail has been configured to use /var/spool/mqueue/qdir* as the queue directory. So for these tests I then re-created that directory:

# mkdir /var/spool/mqueue/qdir00

Sendmail is then started again.

Choosing a message size

One of the parameters that must be set when testing using smtp-source is the size of the message that smtp-source is going to generate.

To determine a "sensible" value for this I looked at the messages in my primary e-mail "Inbox". There are currently 2,379 messages in there, with a total size of 32,997,794 bytes (that includes headers and body). Doing the math gives an average message size of 13,870 bytes. So that's what I'll use in these tests.

Running the tests

To actually run the tests I started three sessions.

The first session is connected to relay-zone, and is the one that starts/stops Sendmail.

The second session is connected to the global zone, and runs:

# queue-create.d relay-zone 1000 > /tmp/results.n

The first argument is the name of the zone to instrument, the second is how many messages to put in each bucket. The results are being saved to /tmp/results. where is the test run number. /tmp is swap-backed.

The third session is connected to the global zone, and runs:

% smtp-source -d -l 13870 -m 30000 -s 1 \
-t nik@relay-zone relay-zone

Those arguments mean:

-d

Send the messages over a single connection, instead of making a new connection for each message.

-l 13870

Sets the message size to 13,870 bytes

-m 30000

Send a total of 30,000 messages

-s 1

Use a total of one connection to send the message.

-t nik@relay-zone

Sets the recipient address

relay-zone

The name of the host to send to

Some of those arguments are things to experiment with. For example, instead of using a single connection to send all the messages (the -s parameter), what happens if 10 connections are opened? Or what effect does having to make a new connection time do? Those are options that I'll vary in future runs.

Right now I'm going to run four sets of tests. Each test will consist of 5 runs, and after each run Sendmail will be stopped, the queue directories will be removed and recreated, and Sendmail will be started again.

The tests will see how Sendmail's queue creation performance differs with 1, 5, 10, and 20 queue directories. The next post will have results.