100

Why do I need an intermediate SMTP server to send mail? Why can’t my client (Outlook, Thunderbird) send messages directly to the recipient’s SMTP domain?

For example, if I have to send an email to [email protected] with my Gmail account, I send it to the smtp.gmail.com server; and then this server will send my message to the MX server of example.com.

1

6 Answers 6

123

It is technically possible to send an email directly to the recipient's SMTP server from your computer.

Looking at it from a historical basis, if the remote SMTP server is down, you want a system to automatically handle it and keep retrying it— hence you have an SMTP server. Similarly, in the old days, not all mail servers were connected all the time—long distance links were expensive, so mail would be queued and sent when a link was established.

Moving on to where Internet is cheap, it's still useful to have mechanisms to retry sending email if a server is unavailable, and it's not ideal for this functionality to be written into the MUA (Mail user agent/end user mail program). These functions fit into an MTA (Mail server/SMTP server).

But it gets worse—spammers. Most emails (way more than 80%) are spam. So mail providers do whatever they can to reduce this problem—and a large number of techniques make assumptions about the way email is delivered—the following are important considerations:

  1. Greylisting: Some providers will automatically drop a mail connection if the sender and recipient have not communicated before, and expect them to try a second time—because spammers often don't, while an SMTP server is always supposed to. This reduces spam volumes by about 80%. It sucks to have to do this though.

  2. Reputation: It is a lot more likely that someone sending email through a reputable, known SMTP server is legit than a fly-by-night server. To get a feel for reputation, providers do a number of things:

    1. Block dynamic / client addresses (Not 100%, but large chunks of the Internet have been mapped out).

    2. Look that reverse DNS matches forward DNS: Not very hard to do, but shows some level of accountability and knowledge of best practices—and something a lot of client address blocks don't have.

    3. Reputation: When communicating with other SMTP servers, a lot of providers keep track of the amount of spam and volumes of email sent and can reduce the amount of spam by limiting connections and keeping an eye on these parameters. (There are lots of ways this is done, not all of them obvious, but which require a known sender).

    4. SPF and DKIM: These mechanisms tie DNS resources to the domain name to make forging mail harder, and would be difficult (but not necessarily impossible to deploy if the mail program (MUA) is responsible for outgoing mail. (Added to make this answer more complete as it's already been accepted. Credit for it should go to posters below as it slipped my mind, but is, nonetheless, very valid)

There are probably other minor concerns, but these would be the major ones.

7
  • 19
    Don't forget such not exactly minor things as SPF (whitelisting of hosts allowed to send mail for a domain) and DKIM (digitally signing messages at domain level) – especially the latter is only possible with a dedicated relay. Nov 27, 2015 at 8:11
  • @grawity Definitely worth mentioneing, but why isn't DKIM "possible" without a dedicated relay? DKIM selectors are not bound to the sending application or IP address. If your mail client can sign the message with a published key it's as valid as any other signatory. Nov 27, 2015 at 10:49
  • 2
    @ManuH: As per metrics in the fine answer, normal mail compromises 1/5 of the mail volume. As per metrics on my server, normal mail compromises 1/20 of the mail volume. It is a terrific trade off.
    – dotancohen
    Nov 27, 2015 at 15:24
  • 2
    @manuh: Greylisting works by closing the connction before the email is sent - it listens only until it receives the sender and recipient - which are in th header. Also, some greylist systems will ommediatelly. accept all email from an smtp server which has a history of retrying delivery. Sadly, its very effective.
    – davidgo
    Nov 27, 2015 at 16:33
  • 4
    Could add that in "the good ol' days" mail was often sent from one SMTP server to another, then another, then another before reaching it's destination. This usually worked fine, but for example during the rtm-worm attack, one of the computer down was one of the essential mail-relays, so emails with warning, solutions and fixes to the worm, could take as much as 48 hours to reach their recipients. Nov 28, 2015 at 11:07
36

Why do I need an intermediate SMTP server to send mail? Why can’t my client (Outlook, Thunderbird) send messages directly to the recipient’s SMTP domain?

In 1991 — and most of the early 1990s and even earlier — you might be able to do what you describe. But the reality in 2015 is, while one can technically send an email to anyone from any machine that has a mail service installed on it, the world of SPAM has made that method effectively useless.

When you use a “real” SMTP service, things are set like PTR records, SPF records and even DomainKeys that are all established for one purpose and one purpose only: to vouch that the SMTP that is sending the message is legit. And if it isn’t? Filter the message into a SPAM folder or the “great abyss” of deletion. Here is a breakdown of what each of those items are:

  • PTR (Pointer Record/Reverse DNS Record): Server level verification. As explained here, a PTR record is used to map a network interface (IP) to a host name. Meaning if you have address 123.456.789.0 on your SMTP server sending emails for smtp.example.com then an appropriate PTR record for that would be smtp.example.com. Seems too simple, but it works since the only one who can really set a PTR record is the owner of the IP address and it can only be set on their hardware. So it acts as a verification point towards who owns/runs/manages that IP address.

  • SPF (Sender Policy Framework): Hostname DNS entry level verification. An SPF record—as explained here—is basically a DNS record set by the domain name holder that provides a list of IP addresses and hostnames of servers that are allowed to send emails for that domain name. That again is another verification step that ensures that only the true domain name owner for an SMTP server can send mails. So let’s say a server with the IP address of 123.456.789.9 is sending emails for example.com. We already know that smtp.example.com uses 123.456.789.0, but an SPF record entry for example.com can state, “Hey! 123.456.789.9 is a good server! He’s legit! Respect his emails!”

  • DKIM (DomainKeys Identified Mail): Email message level verification. As explained here and on Wikipedia, “DKIM is an email validation system designed to detect email spoofing by providing a mechanism to allow receiving mail exchangers to check that incoming mail from a domain is authorized by that domain’s administrators and that the email (including attachments) has not been modified during transport.” By using cryptographic hashes, DKIM verifies that the mail itself was not filtered or tampered with during transit. This also serves as yet another verification point in the “Are you legit or are you SPAM?” chain.

So in the end, a public-facing SMTP server worth anything will have at least two of these items (PTR and SPF) set to verify that the SMTP server and related email are legit. Not everyone uses DKIM, but it is another layer of validation that is becoming increasingly popular nowadays as SPAMmers become more tenacious in their efforts to send SPAM.

15

Most residential ISPs block TCP port 25 (SMTP) in order to prevent you from participating in a spam network. If your PC becomes infected, your PC can start spewing spam at the behest of someone else.

4
  • You write "Most residential ISPs block TCP port 25 (SMTP)" <-- could you elaborate on what that means. Do you mean they won't let you make an outgoing connection to an SMTP server on port 25? Or do you mean they won't let you receive a connection on port 25?
    – barlop
    Nov 28, 2015 at 2:07
  • 2
    @barlop the former — they block outgoing connections on 25 from residential links to machines other than their own mail servers (or indeed to anywhere, because they might use 587 or 465 for their own servers). However, it's somewhat of an exaggeration to say that most ISPs do it.
    – hobbs
    Nov 28, 2015 at 4:08
  • 2
    @hobbs - My experience (and its a fair part of my job) is different. While a lot of ISP's will block traffic leaving their network with a target of port 25 (which forces port 25 traffic through their mail servers), the same is generally not true for port 587 or 465 - and indeed this makes sense. Port 587 and 465 generally REQUIRE authentication, and blocking and are specifically MUA to MTA rather then MTA-MTA - Blocking these ports would generate HUGE backlash as many companies require it so as allow roaming, accountability and not breaking SPF.
    – davidgo
    Nov 28, 2015 at 19:44
  • 3
    @hobbs, I never wrote that most ISPs do this; what I wrote is that most residential ISPs do this. For instance, AT&T, Comcast, TWC, Verizon, etc. do this for their residential customers, but they do not do this for their business customers.
    – Ron Maupin
    Nov 29, 2015 at 19:17
8

The other answers are all excellent, and spam does have a lot to do with it.

But there actually is a simpler, more generic, answer: features. Sending email through SMTP is actually a very complex undertaking. Even without spam, you wouldn't want to implement the whole feature set of the SMTP protocol in every email client; you are better off with a dedicated piece of software (sendmail, postfix etc. are the big ones in the *nix world, Exchange in the Windows world).

For example, even at the most basic, a "real" SMTP server has to at least be able to resolve MX records. Then it has to negotiate features (mostly TLS, but there are other features, too). It has to manage queues for retrying, generate non-delivery reports, etc.

And that's just the basic, must-have, functionality without which the server wouldn't even work. It doesn't even include things such as address rewriting, mailertables. Not to mention the dozen or so other protocols that sendmail et al support, such as UUCP.

The SMTP implementation in Outlook, Thunderbird etc. is very minimal - at best, roughly the equivalent of using a smart host on sendmail, if that.

Related, but a separate issue: email is a very security-sensitive topic, and you would want to have one or a very few centrally-managed servers handling it, instead of potentially hundreds or thousands of individual ones on each desktop.

1
  • This is a good point. It isn't just about the actual features for queueing and so forth though: the availability of the server makes a difference to some of those features. If there is a problem and you shut your laptop down it can't retry until next turned on - the mail server is likely to be available 24/7 though so it is in a much better position to manage a queue of messages. Once you've submitted your message to the server by SMTP your mail client doesn't need to stay online to ensure delivery. Nov 30, 2015 at 10:25
5

Why do I need an intermediate SMTP server to send mail? Why can’t my client (Outlook, Thunderbird) send messages directly to the recipient’s SMTP domain?

You could create an email program that did this, and I have no doubt others have done (or attempted) it before too.

You'd essentially be writing a tool that is both a MUA (mail user agent) and MTA (mail transfer agent) in one.

The reason that this is traditionally separated into different tools, with the MTA residing "server-side", is that an MTA that sends mail through the open internet is considerably more complex to write and configure, and also that it benefits from residing on a reliable "always on" server.

An MTA has to:

  • Look up and connect to servers it doesn't trust, or which may misbehave, and deal with error conditions in a sensible way that doesn't lose mail.

  • Deal with servers that are down, and route to alternate servers or queue the mail for later re-trying. This runs best on a server process that is "always connected" to the internet. It also implies that the mail transfer agent needs its own storage areas for mail that is queued.

  • Deal with a range of different server capabilities, adjusting behaviour according to the capabilities of the receiving server.

  • Report back to the user about error conditions or when mail is undeliverable, so that mail is not just lost.

  • Have excellent security practices and be very security conscious.

  • Ideally, reside on a reliable, always-connected server with a stable IP address and a reverse DNS entry, ie an internet connection suitable for public-facing servers. This helps other systems not detect mail sent as spam.

Given these requirements it makes sense to house the SMTP server on a public-facing always-on server somewhere, and to try and use a tool suited to doing that particular job.

1

Another thing to consider is receiving returned email. At a minimum, all outgoing email has a FROM address where a response can be sent (unknown user, vacation reply, etc). In order for the return address to resolve, an MX record must exist that points to the return inbox location. Unless you are sending email from a computer with a static IP address that is always on, you are going to need a server to handle these inbound messages. This is typically (but not always) handled by the same service.

GMail, Outlook 365, and Yahoo Mail are examples of email services that are used by individuals who are sending email. For commercial email sending, there are services like MailChimp, Marketo, and Eloqua that are very good at sending mass email for a company and handling things like bounces, throttling, and deliverability.

See: https://en.m.wikipedia.org/wiki/Bounce_address

9
  • I don't understand why I need a static IP to get my reply... The reply should be delivered to my MX server (ex. Gmail) not to my computer. Am I right?
    – Tobia
    Nov 27, 2015 at 19:05
  • Yes, you are correct. I guess my point is that an inbox usually exists on a server somewhere in order for outgoing email to be sent. Logically, it makes sense for that server to handle outgoing email as well. Otherwise you would lose things like having a "sent" email folder.
    – dana
    Nov 27, 2015 at 19:10
  • Mmh it is reasonable. But I'm free to send a message with Gmail with an unknown "from" or "replyto" address unsing its smtp server...
    – Tobia
    Nov 27, 2015 at 19:14
  • 1
    If you use GMail, you have to use smtp authentication. So, the FROM address is set to your @gmail.com address. Otherwise, you would be able to use their service for spoofing.
    – dana
    Nov 27, 2015 at 19:30
  • 2
    These days, a lot of users couldn't care less about the bounces, BUT a setup that will not accept bounces is usually considered a probable spam source. Dec 1, 2015 at 10:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .