Email Standards – basic and lesser known
What was SMTP, SPF, DKIM, DMARC, SRS and ARC again?
SMTP — Simple Mail Transfer Protocol
The “Simple Mail Transfer Protocol” (SMTP, see Wikipedia) is the standard mail transfer protocol. It is used by users to “submit” new emails to their local email server and for servers to “relay” the users' emails to the destination server. These two roles were historically handled by the same servers (meaning both “hardware” and “software”). Nowadays these roles are usually handled by different servers listening on different ports and – strictly speaking – slightly different protocols.
The “traditional” and “relaying” role(s) are covered by many RFCs describing the core protocol and quite some more RFCs defining extensions. The core RFCs (all titled “Simple Mail Transfer Protocol”) seem to be RFC 788 (1981), RFC 821 (1982), RFC 2821 (2001) and RFC 5321 (2008). RFC 821 is part of STD 10. It is strange, that the newer RFCs have not been elevated to STD 10.
The “message submission” role is described in another set of RFCs with the title “Message Submission for Mail” (RFC 2476 (1998), RFC 4409 (2006) and RFC 6409 (2011) which is STD 72) but the used protocol is still SMTP! The RFC (and the extensions, of course) “only” change the rules how the server can and should modify messages before delivering them, how to authenticate users and more stuff like that.
The following assumes a “relaying server” (aka. Mail Transfer Agent – MTA):
Given an recipient address like
firstname.lastname@example.org the sender will lookup
MX records for the domain part of the email address:
> dig +short gmail.com MX 5 gmail-smtp-in.l.google.com. 10 alt1.gmail-smtp-in.l.google.com. 20 alt2.gmail-smtp-in.l.google.com. 30 alt3.gmail-smtp-in.l.google.com. 40 alt4.gmail-smtp-in.l.google.com.
The number in the
MX record is a priority (i.e. lower is better). So the
connection will be made to the server with the priority 5. If there is no
MX record then a direct connection to domain
gmail.com will be tried. This
is called “implicit MX record”.
The connection to the destination MTA (i.e. relaying role) will use port 25.
MX MX record does not support different port numbers.
A minimal SMTP session looks like this:
> dig +short example.org MX 10 smtp.example.org > telnet smtp.example.org 25 S: 220 smtp.example.org ESMTP software-id C: HELO mx1.example.net S: 250 Hello mx1.example.net C: MAIL FROM:<email@example.com> S: 250 ok C: RCPT TO:<firstname.lastname@example.org> S: 250 ok C: DATA S: 354 go ahead C: From: Alice <email@example.com> C: To: Bob <firstname.lastname@example.org> C: Subject: simple SMTP test C: C: It Works! CU later! C: . S: 250 OK C: QUIT S: 221 bye Connection closed by foreign host.
Although this will not work for the big email providers like Gmail and
Outlook.com. These will require additional headers like
will still reject the message due to missing other stuff like missing
records (aka “reverse DNS”, rDNS). Google talks about that here: Prevent
mail to Gmail users from being blocked or sent to spam
SPF — Sender Policy Framework
The “Sender Policy Framework” (SPF, see Wikipedia) is an email
authentication standard. The receiving MTA can verify that the sending MTA
is actually allowed to send emails for the domain given in the
envelope address within the SMTP connection. In some situations also the
domains in the
EHLO greeting handshake are checked. Since the the
receiver only sees the IP address of the sender the available tests resolve
around different kinds of DNS tests. The policy itself is also in DNS.
The RFCs for SPF are RFC 4408 (2006) and RFC 7208 (2014) having the title “Sender Policy Framework (SPF) for Authorizing Use of Domains in E-Mail, Version 1”.
Given an alleged sender
email@example.com in the
MAIL FROM: envelope the
receiver will lookup a
TXT DNS record which starts with
> dig +short gmail.com TXT "v=spf1 redirect=_spf.google.com"
redirect (wrapping some long lines) returns:
> dig +short _spf.google.com TXT "v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"
Digging into one
include returns (also wrapped and slightly reformatted):
> dig +short TXT _netblocks.google.com "v=spf1 ip4:220.127.116.11/24 ip4:18.104.22.168/19 ip4:22.214.171.124/20 ip4:126.96.36.199/20 ip4:188.8.131.52/18 ip4:184.108.40.206/16 ip4:220.127.116.11/21 ip4:18.104.22.168/16 ip4:22.214.171.124/17 ip4:126.96.36.199/19 ip4:188.8.131.52/19 ~all"
_netblocks3.google.com are structurally similar.
It is interesting that Google uses
~all which means “Soft Fail”.
Some points about SPF:
SPF is about origin, not content! So it’s not – by itself – a spam / spoofing / phishing protection since it only checks the
MAIL FROM:envelope address and not the
From:header in the email. This envelope address is not presented to the user. But see DMARC which does “align” both parts.
SPF makes it impossible to forward emails using forward rules in the server.1 properly:
Why exactly? A traditional forward without SPF worked like this:
System A sends an email to B:
MAIL FROM:<firstname.lastname@example.org> RCPT TO:<email@example.com>
System B forwards the email to system C:
MAIL FROM:<firstname.lastname@example.org> RCPT TO:<email@example.com>
System C accepts the mail and the easy / good case is closed. But if system C cannot deliver the mail later on then C sends a “bounce message” back to Alice.
With SPF activated (but without DMARC, see later) system C will reject the email, because the B is not allowed to send emails on behalf of A. But B could do this:
MAIL FROM:<firstname.lastname@example.org> RCPT TO:<email@example.com>
Now C accepts the message because even with SPF. But C must send a bounce message back it will address that to B/Bob and not back to Alice. But that bounce will most likely also fail if it is forwarded to C again. Alice will never know that the message didn’t make it Bob won’t know also.
With SPF and DMARC enabled the forward from B to C will fail because the envelope address
firstname.lastname@example.org not “align” with the header
From: email@example.com the message itself.
See Sender Rewriting Scheme for possible mitigations.
Mailing lists have the same problem, since they used to forward mails on behalf of the original sender to many recipients.
Nevertheless – a small site might want to deploy SPF records in order to improve their reputation with the big email receivers like Gmail and Outlook.com.
DKIM — DomainKeys Identified Mail
“DomainKeys Identified Mail” (DKIM, see Wikipedia) is also an email authentication standard. The sending MTA signs selected headers and the body of the email using a private key. The receiving MTA fetches the corresponding public key from DNS and then validates the integrity of the mail.
The RFCs for DKIM are RFC 4871 (2007) and RFC 6376 (2011) both titled “DomainKeys Identified Mail (DKIM) Signatures”. The later RFC is STD 76. Note that RFC 4870 (2007) is a precursor of DKIM with a slightly different title.
A DKIM signature header from Gmail look like this:
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:from:to:cc:subject:date; bh=ZIe1W6Yx/RQkrEdFbGsZMX4VV/Ib0+g8JIgM1R8S5i4=; b=LfDJmO3KmEEu8Ui1bngqK9T8ja3L2jc0Jwmbbi4j7MYWoiGXuq2MYyUVmeQawtohXc hDzC1K+MTOg09J3bxwVOqsPr8nTjz8dbj/Y/0TjGKy+X+NZ5CVIYaRmyPDkxVsNX1H3b VtGqPd5nE02c5CySe046d9j2hL72eAiTc9IHK6wy1V1K7UMKjbUNdkUsPcMUBIZnztzr YChVZ3s0hy9ijifHIqlOSSezkBMKYzD5sB9+7skZIjgk3VYGXkojZaZ6bqjI2U4XtVCo WNGKyj3Bohkw5qHtli618hbK1hOCC9/m8OD5wpZeeZJ6g03SsjkF7Ar+zCjDKmy4Ml3C 9QUw==
b the actual signature,
bh is the body hash,
h the list of signed
d is the “signing domain identifier” and
s the “selector”.
s=20210112 the public key can be fetched (wrapped):
> dig +short 20210112._domainkey.gmail.com TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq8JxVBM LHZRj1WvIMSHApRY3DraE/EiFiR6IMAlDq9GAnrVy0tDQyBND1G8+1fy5RwssQ9DgfNe7r ImwxabWfWxJ1LSmo/DzEdOHOJNQiP/nw7MdmGu+R9hEvBeGRQ" "Amn1jkO46KIw/p2lGvmPSe3+AVD+XyaXZ4vJGTZKFUCnoctAVUyHjSDT7KnEsaiND2rVs DvyisJUAH+EyRfmHSBwfJVHAdJ9oD8cn9NjIun/EHLSIwhCxXmLJlaJeNAFtcGeD2aRGbH aS7M6aTFP+qk4f2ucRx31cyCxbu50CDVfU+d4JkIDNBFDiV+MIpaDFXIf11bGoS08oBBQi yPXgX0wIDAQAB"
Some random selectors:
Some points about DKIM:
DKIM is about (partial) content, not origin! So any host can send a signed message in a “replay attack”. If the
To:Header is not covered by the signature then the message can be sent to anyone. If the
loption limits the length of data covered by the body hash then a valid signed “prefix” message can be prepended to additional body data.
The signing domain (
d=) is not checked against the the user visible
evil.comcan sign a message claiming to be from
Forwarding (as mailing lists do) is not possible: They change the
To:header which is often covered by the signature. They also often add footers with information about the mailing list and unsubscribe links. They might also rewrite the
From:header to point back to the mailing list and
Subject:to include tags like
More about DKIM and Mailing List can be found in RFC 6377 (“DomainKeys Identified Mail (DKIM) and Mailing Lists”, 2011)
While both DKIM and SPF perform email authentication they cover disjoint topics. Therefore both are – per se – not enough to reduce spam / spoofing / phishing. But both can impede legit email flows like forwarding and mailing lists. And like SPF – a small site might want to deploy DKIM in order to improve their reputation with the big email receivers like Gmail and Outlook.com.
DMARC — Domain-based Message Authentication, Reporting, and Conformance
“Domain-based Message Authentication, Reporting, and Conformance” (DMARC, see Wikipedia) is the third email authentication method.
The RFC for DMARC is RFC 7489 (2015) titled “Domain-based Message Authentication, Reporting, and Conformance (DMARC)”.
DMARC relies on either SPF or DKIM or on both and adds some stuff on top:
“Identifier Alignment”: For DKIM the domain in the
From:header of the email and the
d=option in the DKIM signature must match. For SPF the domain from the
MAIL FROM:envelope part must match the domain in the
From:header. In both cases there is some wiggle room for subdomains. This Alignment ties the information presented to the user to the technical declarations and as such benefits the user directly.
Policy: The sender domain can state what the receiver should do when neither SPF nor DKIM succeeded: Accept the message nevertheless, quarantine it or reject it.
Reporting: The sender domain can request “aggregate” reports or “forensic” reports for validation failures.
So DMARC really connects several loose parts of SPF and DKIM.
Of course DMARC also stores the information in a DNS
> dig +short _dmarc.gmail.com TXT "v=DMARC1; p=none; sp=quarantine; rua=mailto:firstname.lastname@example.org"
p specifies the policy for the domain (here: please accept even invalid
sp is the policy for subdomains (mark as spam).
the address to which aggregate feedback reports should be sent. The standard
defines more fields but
gmail.com uses the defaults for these.
Regarding the address for aggregate feedback report (
rua): Since the
sender’s domain (
gmail.com) does not match with the one in the
google.com) another DNS record is required to accept these mails:
> dig +short TXT gmail.com._report._dmarc.google.com "v=DMARC1"
This means: I myself cannot send the reports for my toy domain to my gmail account because Google will not setup the DNS.
Regarding email forwarding and mailing lists: DMARC mostly makes things worse:
With SPF the alignment prevents forwarding quite directly because the
MAIL FROM:envelope address must match the
From:header in the body. So the relay can neither insert its own address nor the original value in the
MAIL FROMaddress. See the example above in SPF.
With DKIM many headers and the body must not be altered. Additional headers can be used in for some things. See “Actually, DMARC works fine with mailing lists” for some details. But that means that both the mailing list software and all email clients must support the new headers.
Some years have gone by since the introduction of DMARC. So this might have happened by now. Hopefully. But this does not cover all forwarding scenarios. RFC 7960 “Interoperability Issues between Domain-based Message Authentication, Reporting, and Conformance (DMARC) and Indirect Email Flows” (2016) lists some more and also some mitigations.
Some points about DMARC:
If both SPF and DKIM are enabled then DMARC will fail only if both checks fail for an email.
The policy (
sp) cannot be configured for SPF and DKIM separately, but reporting of failures can!
Overall it’s the same as with SPF and DKIM: A small mailer should setup also DMARC in order to play nice with the big ones.
SRS — Sender Rewriting Scheme
“Sender Rewriting Scheme” (SRS, see Wikipedia) is a method to
prevent SPF failures when emails are forwarded. It works by rewriting
MAIL FROM: envelope address.
Contrary to the other standards this is not an RFC but a PDF from the libsrs2 homepage.
SRS works like this: System A sends an email to system B:
MAIL FROM:<email@example.com> RCPT TO:<firstname.lastname@example.org> From: Alice <email@example.com>
System B checks SPF, DKIM and DMARC and all checks pass. But Bob had configured a forward rule to Charlie so B forwards the mail to C:
MAIL FROM:<SRS0=HHH=TTfirstname.lastname@example.org> RCPT TO:<email@example.com> From: Alice <firstname.lastname@example.org>
It is similar to the example from SPF because the
MAIL FROM: envelope
address is rewritten to match the forwarders domain. If C accepts the message
but wants to send a bounce message later on it will send it to B. B can
validate the address using the cryptographic truncated hash
HHH and the
TT, reconstruct the original address and forward the
bounce back to Alice. The hash and the timestamp prevent spammers from abusing
SRS also contains an
SRS1= variant which used when multiple forwards happen:
In that case subsequent forwarders don’t extend the address ad infinitum but
only twice altogether.
However: This works only as long as DMARC is not enabled. Since SRS does not
From: header the alignment fails. Wikipedia says:
Please note : Using SRS protocol fails the SPF Alignment check for your DMARC record, and it’s by design. Your DMARC record can still pass with a DKIM check. – Wikipedia – Sender Rewriting Scheme
So why bother with SRS at all? And doesn’t that imply, that any Alice must setup DKIM and no forwarder breaks the signature? Both seem unlikely even considered each on its own.
Some more links to read up more details:
It’s interesting, that Microsoft support SRS.
Avoid changing the envelope sender when forwarding email to Gmail.
[…]If you never forward spam to Gmail, you can change the envelope sender to your domain.
ARC — Authenticated Received Chain
“Authenticated Received Chain” (ARC, see Wikipedia) is also an email authentication method. It tries to address the forwarding issues of the other methods by allowing to override the DMARC policy with a local policy again.
The RFC for ARC is RFC 8617 (2019), titled "(The Authenticated Received Chain (ARC) Protocol)".
ARC builds upon DMARC and – transitively – on DKIM and SPF. The receiving MTA performs the DKIM, SPF and DMARC validation as usual, records the validation results, signs these and puts the records and the signature into headers. The signature also covers any existing ARC headers.
When the email is forwarded the next MTA will apply DKIM, SPF and DMARC again. SPF will fail in almost all situations. DKIM might fail if the forwarder modified the email or some signed headers. If both checks fail or the DMARC alignment fails a strict DMARC policy could call for rejection of the message. A receiver supporting ARC can overrule that policy if the ARC chain and the recorded result are valid and the receiver trusts the forwarder.
[arc-spec - Resources][http://arc-spec.org/?page_id=79] shows supporters for ARC. Google supports it with Gmail, G Suit and Google Groups. Microsoft is shown with limited support for Outlook 365.
Example headers of mail from GMX to Gmail:
ARC-Seal: i=1; a=rsa-sha256; t=1665601609; cv=none; d=google.com; s=arc-20160816; b=h85LEorEEyiTnn2V4dzJQTfRsAPLn6yk7e7HBKKuneFGcbpueyDFhA+eIeq0C9lXWn K5vZHtEmvioImYHbBFIFSrO5NkR4xooHP7ugkWA0ceBN6+i/gisVHqVCanK2rmxxYDbA n7lEfu5T999SraOGFIQwejjHgHdMVgiZOQdAkXPNJVralUuKJcQR6IXyliVUaagG/5CK mLHILGQj6/fstHFn1J/REvsBq5ymk1LK6iufu7Rg4VUqO0953yAp36opB3sE8nfNEHvk lAdTC4HX/soZtwVUc8NTyDAOY5AlfeBt9GrwONHJTGY+qnr3yqT0lVMOTQXvdoq2eRrL cg6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=subject:from:to:user-agent:mime-version:date:message-id :dkim-signature; bh=6ebdA/otT+4IxXlxuGcD9WM90CrApxfN29I3kxuTSSQ=; b=G7sivSbWm0FYSO50iwxA4WVjeqRzuT8J3LDCM4uTzO4e4aNFUcKDfIcMYibmR6y2Dx I/rMY1Rkpxftmqkod3HuJ27h8x4m4bS/zdIk92J0VhIZK4YncokcLA/jRoTVsa4NnDCM D8gWWfD9joP9OhBHHZpzUCOwFiRuAsnOC4ZwpentGazt7KVeVleWpZEiu/tFbJvUdszS +IPHbQvXkXUAc5dPovdDpvj9rkvRbMFgH0RJ7Jx+/sYV0whGnWyEVHXWi5gK5l6xjasi 19FL3MC+4Q36NzOInBsxmYF9P+gDWn3+fKnFJjGsCov+O+ytkuFMkPcBF4g+R+Igvam0 G0Kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass email@example.com header.s=badeba3b8450 header.b=c+GkZtfO; spf=pass (google.com: domain of XXX@gmx.de designates 184.108.40.206 as permitted sender) smtp.mailfrom=XXX@gmx.de; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de
The public key can be fetched like a DKIM key (line wrapped):
> dig +short arc-20160816._domainkey.google.com TXT "k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Lztpxs7yUxQEsbD FhjMc9kZVZu5P/COYEUIX4B39IL4SXAbv4viIlT9E6F6iZmTh1go7+9WQLywwgwjXMJx/D z0RgMoPeyp5NRy4l320DPYibNqVMWa5" "iQ2WiImQC0en1O9uhLLvzaSZJ03fvGmCo9jM o0GwKzLNe14xMgn/px2L5N/3IKlKX4bqUAJTUt8L993ZlWzvgMnSFSt8B+euSKSrtAiopd y4r1yO4eN5goBASrGW0eLQc1lYouNvCrcTQpos4/GEAqiGzpqueJLmBfOO4clNvVvpPkvQ s2BHw9I9LmIjaMxTNGxkGBRaP3utDiKXXqu1K+LRzl0HCNSdQIDAQAB"
Some random ARC selectors:
- Mail Transfer Agent
- Domain Name System
- Administrative Management Domain
- indirect mailflow
- A mailflow with intermediate hops like forwarding, mailing list, … See RFC 7960
Pressing the “Forward To” button in the mail client is still possible, but that creates a new mail. ↩︎