Most of the spam I see has been sent by servers forging or otherwise obscuring their server identity. RFC2505 states that the server identity and sender address are easily forged. Of these, it is easiest to identify server forgery. Very little, if any, of the personal email has a forged server identity. Unfortunately, legitimate bulk and automated email often shows signs of server identity. If you deliver either of these types of email this article will provide information of fixing the situation.
The rules here apply email originating from the Internet only. Mail User Agents submitting email are expected to violate these rules. MUAs should use an authenticated encrypted connection to the Submission port (576). Relay servers should not apply these rules to connections originating form the local network.
Characteristics of a Valid Identity
A legitimate email server has a number of simple characteristics.
- It has a static addresss with a DNS PTR record specifying the host’s FQDN (Fully Qualified Domain Name). This should have at least three domain components: hostname, sub-domain, and the TLD (top level domain). For example mail.example.com instead of example.com.
- It has an A record for its FQDN returning its IP address.
- It provides a Helo name that is a FQDN when it issues a EHLO or HELO command.
- It has a working postmaster address.
- It may have an SPF record allowing it to send email. Most servers should be able to use ‘”v=spf1 a -all”. When validating the HELO name, I treat softfail and neutral results a fail response.
For IPv4 the creation of the PTR record requires co-operation of the IP address provider. They may either delegate the PTR records, or configure the record themselves. Large organizations are likely to provide their own IP addresses. For IPv6, it should be much easier to get the DNS delegated making it simpler to maintain proper data.
Characteristics of a Forgery
The included statistics are based on my logs for the last six months. Only the last attempt from each IP address is considered. Statistics based on the name in the HELO command are likely understated due to spam reduction delays prior to this command.
There are several clear indication of forgery. A legitimate Internet server should never show them, and if so can easily be reconfigured to correct the problem. The most common forgeries I encounter are:
- The IP address is listed in the zen.spamhaus.org DNS blacklist. The IP address is either dynamic, or is being used to send Spam. This applies to 77.6% of sending addresses.
- Use of an unqualified hostname as the Helo name. This appears to be use of a hostname lookup on a host without a defined domain. This applies to 22.9% of sending addresses.
- Use of my FQDN or IP address as the Helo name. This appears to be an attempt to convince my server the email is arriving from a local source. This applies to 5.9% of sending addresses.
- Use of a domain literal as the Helo name. This is the proper format for presenting an IP address. However, a legitimate email server should be able to provide an FQDN. This applies to 1.1% of sending addresses.
- Use of an IP address as the Helo name. This is just plain incorrect, and should not be accepted from any source. This applies to 0.4% of sending addresses.
- Use of an underscore in the Helo name. This may be a valid Windows network name, but is prohibited by the RFCs.
- The Helo name is a second level domain for a large domains such as gmail.com, hotmail.com, and yahoo.com this is certainly a forgery. For incorrectly configured legitimate servers, the servers DNS hostname will almost always be a sub-domain of the presented name. For the top 21 bogus domains this applies to 3.6% of sending addresses.
- The Helo name is prohibited by SPF. SPF for a server is very easy to get right. This applies to 2.2% of sending addresses. An additional 30.9% don’t supply a FQDN and will be excluded by prior rules.
- One of the host names listed by the PTR records for the IP address is prohibited by SPF. SPF for a server is very easy to get right, and the host names are difficult to forge. This applies to 0.2% of sending addresses.
Where possible I have included three percentages. These are based on the last connection from an IP address. Due to changes in my configuration which results in spambots disconnecting, the latest values may be inaccurate for values based on Helo data. Statistics were generated when I first drafted this article. The three values are:
- Percent of addresses in the last six months that triggered this rule.
- Percent of addresses in the last six months that triggered this rule, and had the message accepted.
- Percent of addresses in the last three months that triggered this rule, and had the message accepted. This value is likely overstated as many spambots disconnect before providing a HELO command.
There are additional cases which are often failed by legitimate senders. These are a symptom of DNS or server misconfiguration.
- rDNS verify fails for the IP address. ( 60% / 0.24% / 0.34% )
- rDSN verify fails for the Helo name. ( >44% / 1.61% / 2.15% )
These rules could be used to block some of these forgeries. They will block some legitimate mail, most of which will be from automated mailers. I use temporary errors when blocking on these rules.
- DNS lookups for rDNS looks fail. Until recently this was a rare occurrence, but since the botnet takedowns began have occurred frequently. This may be a result in DNS changes for dynamic address ranges. I defer the connection in this case. A review of the IP addresses failing indicates they were not legitimate senders.
- There is no PTR record for the IP address. (34.5% / 0.2% / 0.04%)
- Use of a Helo name invalid at the second level (e.g. example.ca). Very few legitimate senders fail this test. Those that do usually have a PTR record which returns a sub-domain of the HELO FQDN.
- Use of an invalid Helo name. Unfortunately, in my data, this is more likely to indicate bulk or automated mail than an invalid sender. Most have valid rDNS data and may share the same second level domain. More that 1% of legitmate mail fails this test.
- There is no PTR record for the IP address, and the IP address does not match the A record of the Helo name, if present for the FQDN. These hosts are missing from DNS. Almost no legitimate servers fail both these tests, especially if the IP match is done using a /24 mask. Legitimate sites do get their DNS slightly off in some case.
Verifying a DNS Configuration
On Windows 7 the DNS lookup tool is nslookup which is run from the command line. For Linux replace
nslookup -type=ANY with
host -a. (If the host command is missing install the
C:\Users\you>nslookup -type=ANY mail.example.com 184.108.40.206. Server: google-public-dns-b.google.com Address: 220.127.116.11 Non-authoritative answer: mail.example.com ??? unknown type 99 ??? mail.example.com text = "v=spf1 a -all" mail.example.com internet address = 192.0.2.2 mail.example.com AAAA IPv6 address = 2001:DB8:1122:aabb::11
The unknown type 99 message is an SPF record which Windows does not yet recognize. Windows also incorrect formats the output for the TXT (text) record. Host will show both the SPF and TXT records which should have the same content if they are present. The SPF data should be a single string. If the text reads
"v=spf1" "a" "-all", the SPF data is broken as it will be read as
For each address check the DNS PTR data by doing a lookup. This example shows a possibly broken PTR. The pointer should have returned
mail.example.com. It is valid if
smpt1.example.com points to
192.0.2.6. In that case the mail server should use
smtp1.example.com in its banner message and HELO commands.
C:\Users\bill>nslookup -type=PTR 192.0.2.6 18.104.22.168. Server: google-public-dns-b.google.com Address: 22.214.171.124 Non-authoritative answer: 126.96.36.199.in-addr.arpa name = smtp1.example.com
Verifying a Server’s Public Identity
Other than verifying the configuration there are various methods which can be used to verify your server’s public identity. Simply using telnet to connect to the server should give you the public identity. Enter the command
quit to exit. You should see the name you connected to in the
221 messages. The 220 message may disclose information about the server software.
telnet mail.example.com smtp Trying 192.0.2.11... Connected to mail.example.com. Escape character is '^]'. 220 mail.example.com ESMTP Server Ready Sun, 13 Feb 2011 10:33:31 -0500 quit 221 mail.example.com closing connection Connection closed by foreign host.
Each mail server handling a message adds a Received header. These can be seen by examining the full headers of the message. You can verify the headers by sending the email to an external address such as a gmail account, or by examining a bounce message. A bounce message
If a servers public identity matches the rDNS its rDNS name it may not be reported in the Received header. Otherwise it is usually reported. The following headers from a phishing mail all show a Helo name that does not match.
Received: from mail2.abc.gov.ar ([192.0.2.5] helo=mail.abc.gov.ar) by mail.example.com with esmtp (Exim 4.71) (envelope-from ) id 1Pj5HO-00021k-0h for firstname.lastname@example.org; Sat, 29 Jan 2011 02:32:04 -0500 Received: from mail.abc.gov.ar (localhost.localdomain [127.0.0.1]) by localhost.abc.gov.ar (Postfix) with ESMTP id 47A4B6B84C8; Sat, 29 Jan 2011 04:27:32 -0300 (ART) Received: from mail.abc.gov.ar (loopback.ed.gba.gov.ar [127.0.0.1])by mail.abc.gov.ar (Postfix) with ESMTP id 2A51B6B84B2;Sat, 29 Jan 2011 04:27:32 -0300 (ART)
This header shows a received header where the Helo name and host rDNS names match. (This format is from Exim, other software may use different formats, or fail to do DNS lookups.)
Received: from xsmtp07.mail2web.com ([188.8.131.52]) by mail.example.com with esmtp (Exim 4.71) id 1PogeV-0006H9-RT for email@example.com; Sun, 13 Feb 2011 13:26:36 -0500
If you have an email address on the server you want to test, there are services which will verify your mail for you. These include:
- mailto:firstname.lastname@example.org returns the IP address of your server.
- http://postmaster.aol.com/cgi-bin/plugh/rdns.pl does a pointer lookup for your IP using AOLs nameservers.
- http://www.intodns.com/ does extensive DNS checks including verifying MX and rDNS data.
- http://www.email-unlimited.com/tools/verify-email.aspx displays a log of the verification attempt. The 220 message can be used to verify the public identity of a server.
- mailto:email@example.com which does extensive testing on your email reporting most verifiable conditions. It does not verify rDNS, but does report the HELO hostname. Combine with intoDNS for a full check.
- Anti-Spam Recommendations for SMTP MTAs
- Messaging Anti-Abuse Working Group
- Email Sender & Provider Coalition resources
- SMTP Service Extension for Command Pipelining
- Domain Administrators Operations Guide (Note: Applicable to static IP addresses. You may need your IP address provider to change the PTR records. )
- Sender Policy Framework