First, the new HeloName option.
When Sendmail needs to deliver mail to a different server, it connects to the remote server, waits for it to send the greeting banner, and then sends:
(or EHLO the.host.name, but I'm keeping this simple). Sendmail uses the local, fully qualified hostname of the host when it does this. So if you host is called foo.example.com Sendmail will send HELO foo.example.com.
What happens next depends on how the remote system is configured. It is very typical for the remote site to do a DNS lookup on the provided hostname, to make sure that it exists. This is a common anti-spam measure. If the given hostname doesn't exist then the rest of the connection will be rejected.
Further, it's also common to do the reverse lookup. So if foo.example.com resolves to the IP address 126.96.36.199, the remote site will then look up 188.8.131.52.in-addr.arpa, and verify that that resolves back to foo.example.com. Again, if it doesn't, the connection is refused. This is a less common anti-spam technique, but it's still found in regular use at hundreds of thousands of sites around the world.
You've probably worked out how this can cause problems. It's common, especially in corporate environments, for hosts to have multiple hostnames, especially if they are internet facing hosts. There'll be one name by which the host is known internally (and is only published in the internal, corporate DNS), and one name (and associated IP address) by which the host is known externally.
This can lead to situations where the mail server connects to the remote server, and sends:
my.internal.name is not published in the public DNS, so the remote server rejects the rest of the connection.
It may be even more complicated than that -- for example, the internet facing mail relay might be behind a firewall that carries out Network Address Translation (NAT) -- in which case, even if the server is configured to use the correct hostname in the HELO step, that hostname might not resolve back to the right IP address, and again, the connection may be refused.
Prior to the HeloName option being added to Sendmail you could use the confDOMAIN_NAME configuration option to specify the full domain name to use.
This works, but has other side-effects. In particular, the value of confDOMAIN_NAME appears in Received: headers, and if you set it obscures the actual hostname, which can make analysis of messages through the Received: headers more difficult.
This is the situation I ran in to at $work. We have many Internet facing mail servers relaying mail to the outside world. Each server knows its own internal hostname, but has no knowledge of the hostname it presents to the outside world, nor does it know the IP address that the packets it sends will finally have (all the servers are behind NAT devices).
I considered configuring each server with this information, but decided that that would be a bad idea. It would have meant tightly coupling our configuration information with configuration information maintained by an entirely separate group (the Networks Team). Should they decide to renumber a NAT'd IP address they would need to ensure that we were informed, and that the change was co-ordinated.
That's extra work, extra complexity, and extra opportunity for things to go wrong.
Each server's IP address is already in a large A record, mail.example.com. I considered adjusting confDOMAIN_NAME and related settings so that each outbound mail relay considered itself to be mail.example.com, but rejected that idea on the previously mentioned issues relating to Received: lines and the extra complexity it introduces when debugging.
So, plan C. Specifically, I wrote a small patch for Sendmail that introduces a new configuration option, HeloName. If set, Sendmail will use the value of this option when it sends HELO/EHLO, instead of the fully qualified hostname (or confDOMAIN_NAME setting).
Our Sendmail configuration files now contain this directive:
This is sufficient for all our outbound mail relays to HELO with the same name. This name contains all their IP addresses. And the Received: headers still contain the correct internal hostnames, to make troubleshooting easier.
If we'd been using a closed source MTA this wouldn't have been possible.