Day 28 of 60: Instrumenting Sendmail queue file creation (pt 4)

Yesterday I looked at the effect of multiple queue directories when processing messages over a single connection.

Today I've been looking at how multiple queue directories can help when processing concurrent connections.

The methodology was identical to the previous tests. The only change was to the smtp-source(1) command line. The previous tests were run with -s 1, indicating one concurrent connection. These tests were run with -s 10, to force 10 concurrent connections.



Test 5 - 1 queue directory



View raw data

ministat showed that all but the fourth run had no difference at 99.5%. The fourth run did, but very small.

Difference at 99.5% confidence
-2308.38 +/- 2274.66
-4.08402% +/- 4.02435%


So I'll continue to use the first run for the remaining figures.

It was obvious that these tests were taking considerably longer to run than the originals, but it wasn't clear quite how much slower they were until I compared the results from test1 with these results. The plot's not very interesting, but the differences are. The first result is from 1 queue / 1 connection, the second is from 1 queue / 10 connections.









#NMinMaxMedianMeanStddev
x61953139811431143.770588.056496
+6043499680815592956522.3674354.2006


Difference at 99.5% confidence
55378.6 +/- 1722.91
4841.76% +/- 150.634%


Clearly, having a single queue directory when processing many concurrent connections is a liability, compared to the single connection case, being almost 50 times slower.

This is especially apparent if I plot these results and overlay the results from test1.

Plots from tests 5 and 1, result set 1

In this plot the row of squares at the very bottom, almost bumping in to the X axis, is the data from test 1. The data from test 5 is the much more widely spaced points towards the top of the graph.

The "Slopes" table shows the data from test 5 and test 1 respectively. As you can see, not only is there much more 'jitter' in the results from test 5 when compared like this, the cost per directory entry (the slope of the trend line) is much steeper too.

Test 6 - 5 queue directories



View raw data

These tests felt subjectively much faster when I ran them. The ministat results bear this out. These ministat results compare all the runs from this test, showing no significant differences between the runs.















#NMinMaxMedianMeanStddev
x6096451415811747.511781.0671156.3799
+6095921372211701.511771.41068.0186
No difference proven at 99.5% confidence
*6094921609311637.511719.6331127.3325
No difference proven at 99.5% confidence
%6098421487711934.511897971.37713
No difference proven at 99.5% confidence
-609832136301175611595.317903.07967
No difference proven at 99.5% confidence


This plot compares the first run from test 5 with the first run from this test.

x test5-run1
+ test6-run1
: = Mean
M = Median
+----------------------------------------------------------+
| + |
| + |
| + |
| ++ |
| ++ |
| ++ |
| +++ |
| +++ |
| +++ |
| +++ |
| +++ x |
| ++++ x |
| ++++ xx |
| ++++ xx |
| ++++ xxx xx |
|+++++ xxxxxxxx x |
|+++++ x xxxxxxxx x |
|+++++ xx xxxxxxxx xxx |
|+++++ x xxx xxxxxxxxxxxx x|
| |:| |___M:___| |
+----------------------------------------------------------+








#NMinMaxMedianMeanStddev
x6043499680815592956522.3674354.2006
+6096451415811747.511781.0671156.3799

Difference at 99.5% confidence
-44741.3 +/- 1797.18
-79.1568% +/- 3.17959%


It's apparent that adding even 4 additional queue directories has a significant positive benefit on the amount of time taken to queue messages.

This chart makes the differences even more apparent. The green plot points are from test 5, the red plot points from test 6. The figures above show that test 6's standard deviation is less than that of test 5, and this is obvious from this chart in the reduced 'jitter' between the points.

Plot from tests 5 and 6, result set 1

Note that although this is a big improvement from test 5, these results are still almost 10 times slower than the results from the first 4 tests. Using ministat to compare the results from test 1 and test 6, test 6 is 930.02% +/- 40.1145% slower.

Test 7 - 10 queue directories



View raw data

Having seen the significant effect that adding 4 additional queue directories made, I was hoping to see a greater benefit with a total of 10 directories.

Again, running these tests they felt subjectively faster, although not significantly faster than test 6.

The ministat results confirm that the test runs are equivalent, with no obvious outlier data points, so I set about comparing tests 6 and 7. ministat indicated a small improvement.

x test6-run1
+ test7-run1
: = Mean
M = Median
+----------------------------------------------------------+
| + + |
| + + + x |
| + + ++ + + ++ x x x x xx x |
|+ + + ++ ++++ +x++++x++x x+xx xx xxx xx x x |
|+ + ++++++++++ ++++ ++++++++++xx+xx +x+xxxxxxx +xx x xxx|
| |__________:__________| |
| |_________:_________| |
+----------------------------------------------------------+









#NMinMaxMedianMeanStddev
x6096451415811747.511781.0671156.3799
+6079671318497829841.61070.5916

Difference at 99.5% confidence
-1939.47 +/- 628.644
-16.4626% +/- 5.33605%


This improvement is also obvious if the results from test 6 and test 7 are charted and overlaid, like so:

Plot from tests 6 and 7, result set 1

Green points are from test 6, red points are from test 7. 10 queue directories is clearly superior to 5 in this test, although by nothing like the margin of difference going from 1 to 5 queue directories.

Test 8 - 20 queue directories



View raw data

This final test involved 20 queue directories. The ministat results show no significant differences between the results of each run, so I compared run 1 from this test with run 1 from test 7.

This showed no significant differences.







#NMinMaxMedianMeanStddev
x6079671318497829841.61070.5916
+6077001285096319691.68331099.0576
No difference proven at 99.5% confidence


This suggests that there is no significant benefit, when speaking of the time required to create a new queue entry, going from 10 to 20 queue directories. At least in the environment that I'm conducting the tests.

The chart bears this out. This shows results from test 7 (green) and test 8 (red) overlaid.

Plot from tests 7 and 8, result set 1

I suspect this is because when the number of queue directories equals the number of simultaneous connections other factors start contributing with more weight.

Conclusion



Even with a single disk, multiple queue directories have a dramatic effect on the time taken to queue messages as soon as Sendmail is processing concurrent connections.

That effect appears to tail off as the number of queue directories approaches the number of concurrent connections.

Further work



These results pose some questions that I intend to investigate and answer in the next post.

First, what is responsible for the dramatic slow down in the single-queue case (test 4)? I suspect that the process of creating a new directory entry is serialised in the filesystem, and with 1 directory and 10 connections all the processes meet at this bottleneck. I should be able to use DTrace to instrument the single queue case in more detail, and get metrics on exactly which part of the process is taking the most time.

Second, with these results the benefit of multiple queue directories tails off as the number of concurrent connections approaches the number of queue directories. This can be tested (e.g., 20 connections with 5, 10, and 20 queue directories, 30 connections with 5, 10, and 20 queue directories, and so on), and I'll do that to see if anything interesting falls out of the data.

4 comments:

  1. Since you clearly need yet another variable in the mix, I'd be interested in what difference ufs vs zfs makes on the queue performance.

    ReplyDelete
  2. Heh. It's on the TODO list, as I'd like to get some experience with ZFS. Not sure that I'll get the chance to do so in the next 30 days though. A lot depends on how rapidly I can get through this round of data gathering and reporting.

    ReplyDelete
  3. [...] As expected, given the results from yesterday this shows the dramatic difference when using multiple queue directories. [...]

    ReplyDelete
  4. [...] Back on day 28 I looked at the effect of multiple queue directories with concurrent senders. [...]

    ReplyDelete