Email Standards – basic and lesser known

SMTP SPF DKIM DMARC SRS ARC

What was SMTP, SPF, DKIM, DMARC, SRS and ARC again?

SMTP — Simple Mail Transfer Protocol

The “Simple Mail Transfer Protocol” (SMTP, see Wikipedia) is the standard mail transfer protocol. It is used by users to “submit” new emails to their local email server and for servers to “relay” the users' emails to the destination server. These two roles were historically handled by the same servers (meaning both “hardware” and “software”). Nowadays these roles are usually handled by different servers listening on different ports and – strictly speaking – slightly different protocols.

The “traditional” and “relaying” role(s) are covered by many RFCs describing the core protocol and quite some more RFCs defining extensions. The core RFCs (all titled “Simple Mail Transfer Protocol”) seem to be RFC 788 (1981), RFC 821 (1982), RFC 2821 (2001) and RFC 5321 (2008). RFC 821 is part of STD 10. It is strange, that the newer RFCs have not been elevated to STD 10.

The “message submission” role is described in another set of RFCs with the title “Message Submission for Mail” (RFC 2476 (1998), RFC 4409 (2006) and RFC 6409 (2011) which is STD 72) but the used protocol is still SMTP! The RFC (and the extensions, of course) “only” change the rules how the server can and should modify messages before delivering them, how to authenticate users and more stuff like that.

The following assumes a “relaying server” (aka. Mail Transfer Agent – MTA):

Given an recipient address like example@gmail.com the sender will lookup MX records for the domain part of the email address:

> dig +short gmail.com MX
5 gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
20 alt2.gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.
40 alt4.gmail-smtp-in.l.google.com.

The number in the MX record is a priority (i.e. lower is better). So the connection will be made to the server with the priority 5. If there is no MX record then a direct connection to domain gmail.com will be tried. This is called “implicit MX record”.

The connection to the destination MTA (i.e. relaying role) will use port 25. The MX MX record does not support different port numbers.

A minimal SMTP session looks like this:

> dig +short example.org MX
10 smtp.example.org
> telnet smtp.example.org 25
S:	220 smtp.example.org ESMTP software-id
C:	HELO mx1.example.net
S:	250 Hello mx1.example.net
C:	MAIL FROM:<alice@example.net>
S:	250 ok
C:	RCPT TO:<bob@example.org>
S:	250 ok
C:	DATA
S:	354 go ahead
C:	From: Alice <alice@example.net>
C:	To: Bob <bob@example.org>
C:	Subject: simple SMTP test
C:
C:	It Works! CU later!
C:	.
S:	250 OK
C:	QUIT
S:	221 bye
Connection closed by foreign host.

Although this will not work for the big email providers like Gmail and Outlook.com. These will require additional headers like Message-ID: and will still reject the message due to missing other stuff like missing PTR records (aka “reverse DNS”, rDNS). Google talks about that here: Prevent mail to Gmail users from being blocked or sent to spam

SPF — Sender Policy Framework

The “Sender Policy Framework” (SPF, see Wikipedia) is an email authentication standard. The receiving MTA can verify that the sending MTA is actually allowed to send emails for the domain given in the MAIL FROM: envelope address within the SMTP connection. In some situations also the domains in the HELO / EHLO greeting handshake are checked. Since the the receiver only sees the IP address of the sender the available tests resolve around different kinds of DNS tests. The policy itself is also in DNS.

The RFCs for SPF are RFC 4408 (2006) and RFC 7208 (2014) having the title “Sender Policy Framework (SPF) for Authorizing Use of Domains in E-Mail, Version 1”.

Given an alleged sender example@gmail.com in the MAIL FROM: envelope the receiver will lookup a TXT DNS record which starts with v=spf1:

> dig +short gmail.com TXT
"v=spf1 redirect=_spf.google.com"

Following the redirect (wrapping some long lines) returns:

> dig +short _spf.google.com TXT
"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com
 include:_netblocks3.google.com ~all"

Digging into one include returns (also wrapped and slightly reformatted):

> dig +short TXT _netblocks.google.com
"v=spf1
ip4:35.190.247.0/24 ip4:64.233.160.0/19 ip4:66.102.0.0/20  ip4:66.249.80.0/20
ip4:72.14.192.0/18  ip4:74.125.0.0/16   ip4:108.177.8.0/21 ip4:173.194.0.0/16
ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19
~all"

_netblocks2.google.com and _netblocks3.google.com are structurally similar.

It is interesting that Google uses ~all which means “Soft Fail”.

Some points about SPF:

Nevertheless – a small site might want to deploy SPF records in order to improve their reputation with the big email receivers like Gmail and Outlook.com.

DKIM — DomainKeys Identified Mail

“DomainKeys Identified Mail” (DKIM, see Wikipedia) is also an email authentication standard. The sending MTA signs selected headers and the body of the email using a private key. The receiving MTA fetches the corresponding public key from DNS and then validates the integrity of the mail.

The RFCs for DKIM are RFC 4871 (2007) and RFC 6376 (2011) both titled “DomainKeys Identified Mail (DKIM) Signatures”. The later RFC is STD 76. Note that RFC 4870 (2007) is a precursor of DKIM with a slightly different title.

A DKIM signature header from Gmail look like this:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112;
 h=content-transfer-encoding:subject:from:to:content-language
  :user-agent:mime-version:date:message-id:from:to:cc:subject:date;
 bh=ZIe1W6Yx/RQkrEdFbGsZMX4VV/Ib0+g8JIgM1R8S5i4=;
 b=LfDJmO3KmEEu8Ui1bngqK9T8ja3L2jc0Jwmbbi4j7MYWoiGXuq2MYyUVmeQawtohXc
  hDzC1K+MTOg09J3bxwVOqsPr8nTjz8dbj/Y/0TjGKy+X+NZ5CVIYaRmyPDkxVsNX1H3b
  VtGqPd5nE02c5CySe046d9j2hL72eAiTc9IHK6wy1V1K7UMKjbUNdkUsPcMUBIZnztzr
  YChVZ3s0hy9ijifHIqlOSSezkBMKYzD5sB9+7skZIjgk3VYGXkojZaZ6bqjI2U4XtVCo
  WNGKyj3Bohkw5qHtli618hbK1hOCC9/m8OD5wpZeeZJ6g03SsjkF7Ar+zCjDKmy4Ml3C
  9QUw==

b the actual signature, bh is the body hash, h the list of signed headers. d is the “signing domain identifier” and s the “selector”.

Using d=gmail.com and s=20210112 the public key can be fetched (wrapped):

> dig +short 20210112._domainkey.gmail.com TXT
"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq8JxVBM
LHZRj1WvIMSHApRY3DraE/EiFiR6IMAlDq9GAnrVy0tDQyBND1G8+1fy5RwssQ9DgfNe7r
ImwxabWfWxJ1LSmo/DzEdOHOJNQiP/nw7MdmGu+R9hEvBeGRQ"
"Amn1jkO46KIw/p2lGvmPSe3+AVD+XyaXZ4vJGTZKFUCnoctAVUyHjSDT7KnEsaiND2rVs
DvyisJUAH+EyRfmHSBwfJVHAdJ9oD8cn9NjIun/EHLSIwhCxXmLJlaJeNAFtcGeD2aRGbH
aS7M6aTFP+qk4f2ucRx31cyCxbu50CDVfU+d4JkIDNBFDiV+MIpaDFXIf11bGoS08oBBQi
yPXgX0wIDAQAB"

Some random selectors:

Domain Selector
gmail.com 20210112
amazon.com pog7p5txl4ruqhixze35ey6av7ivmsea
amazonses.com 6gbrjpgwjskckoa6a5zn6fwqkn67xbtw
protonmail.com protonmail3
gmx.net badeba3b8450
web.de s29768273
mailings.web.de 1662700821

Some points about DKIM:

While both DKIM and SPF perform email authentication they cover disjoint topics. Therefore both are – per se – not enough to reduce spam / spoofing / phishing. But both can impede legit email flows like forwarding and mailing lists. And like SPF – a small site might want to deploy DKIM in order to improve their reputation with the big email receivers like Gmail and Outlook.com.

DMARC — Domain-based Message Authentication, Reporting, and Conformance

“Domain-based Message Authentication, Reporting, and Conformance” (DMARC, see Wikipedia) is the third email authentication method.

The RFC for DMARC is RFC 7489 (2015) titled “Domain-based Message Authentication, Reporting, and Conformance (DMARC)”.

DMARC relies on either SPF or DKIM or on both and adds some stuff on top:

So DMARC really connects several loose parts of SPF and DKIM.

Of course DMARC also stores the information in a DNS TXT record:

> dig +short _dmarc.gmail.com TXT
"v=DMARC1; p=none; sp=quarantine; rua=mailto:mailauth-reports@google.com"

p specifies the policy for the domain (here: please accept even invalid messages), sp is the policy for subdomains (mark as spam). rua specifies the address to which aggregate feedback reports should be sent. The standard defines more fields but gmail.com uses the defaults for these.

Regarding the address for aggregate feedback report (rua): Since the sender’s domain (gmail.com) does not match with the one in the rua field (google.com) another DNS record is required to accept these mails:

> dig +short TXT gmail.com._report._dmarc.google.com
"v=DMARC1"

This means: I myself cannot send the reports for my toy domain to my gmail account because Google will not setup the DNS.

Regarding email forwarding and mailing lists: DMARC mostly makes things worse:

Some points about DMARC:

Overall it’s the same as with SPF and DKIM: A small mailer should setup also DMARC in order to play nice with the big ones.

SRS — Sender Rewriting Scheme

“Sender Rewriting Scheme” (SRS, see Wikipedia) is a method to prevent SPF failures when emails are forwarded. It works by rewriting the MAIL FROM: envelope address.

Contrary to the other standards this is not an RFC but a PDF from the libsrs2 homepage.

SRS works like this: System A sends an email to system B:

MAIL FROM:<alice@a.examle.net>
RCPT TO:<bob@b.example.org>
From: Alice <alice@a.example.net>

System B checks SPF, DKIM and DMARC and all checks pass. But Bob had configured a forward rule to Charlie so B forwards the mail to C:

MAIL FROM:<SRS0=HHH=TT=a.example.net=alice@b.example.org>
RCPT TO:<charlie@c.example.edu>
From: Alice <alice@a.example.net>

It is similar to the example from SPF because the MAIL FROM: envelope address is rewritten to match the forwarders domain. If C accepts the message but wants to send a bounce message later on it will send it to B. B can validate the address using the cryptographic truncated hash HHH and the truncated timestamp TT, reconstruct the original address and forward the bounce back to Alice. The hash and the timestamp prevent spammers from abusing the system.

SRS also contains an SRS1= variant which used when multiple forwards happen: In that case subsequent forwarders don’t extend the address ad infinitum but only twice altogether.

However: This works only as long as DMARC is not enabled. Since SRS does not change the From: header the alignment fails. Wikipedia says:

Please note : Using SRS protocol fails the SPF Alignment check for your DMARC record, and it’s by design. Your DMARC record can still pass with a DKIM check. – Wikipedia – Sender Rewriting Scheme

So why bother with SRS at all? And doesn’t that imply, that any Alice must setup DKIM and no forwarder breaks the signature? Both seem unlikely even considered each on its own.

Some more links to read up more details:

ARC — Authenticated Received Chain

“Authenticated Received Chain” (ARC, see Wikipedia) is also an email authentication method. It tries to address the forwarding issues of the other methods by allowing to override the DMARC policy with a local policy again.

The RFC for ARC is RFC 8617 (2019), titled "(The Authenticated Received Chain (ARC) Protocol)".

ARC builds upon DMARC and – transitively – on DKIM and SPF. The receiving MTA performs the DKIM, SPF and DMARC validation as usual, records the validation results, signs these and puts the records and the signature into headers. The signature also covers any existing ARC headers.

When the email is forwarded the next MTA will apply DKIM, SPF and DMARC again. SPF will fail in almost all situations. DKIM might fail if the forwarder modified the email or some signed headers. If both checks fail or the DMARC alignment fails a strict DMARC policy could call for rejection of the message. A receiver supporting ARC can overrule that policy if the ARC chain and the recorded result are valid and the receiver trusts the forwarder.

[arc-spec - Resources][http://arc-spec.org/?page_id=79] shows supporters for ARC. Google supports it with Gmail, G Suit and Google Groups. Microsoft is shown with limited support for Outlook 365.

Example headers of mail from GMX to Gmail:

ARC-Seal: i=1; a=rsa-sha256; t=1665601609; cv=none;
	d=google.com; s=arc-20160816;
	b=h85LEorEEyiTnn2V4dzJQTfRsAPLn6yk7e7HBKKuneFGcbpueyDFhA+eIeq0C9lXWn
	 K5vZHtEmvioImYHbBFIFSrO5NkR4xooHP7ugkWA0ceBN6+i/gisVHqVCanK2rmxxYDbA
	 n7lEfu5T999SraOGFIQwejjHgHdMVgiZOQdAkXPNJVralUuKJcQR6IXyliVUaagG/5CK
	 mLHILGQj6/fstHFn1J/REvsBq5ymk1LK6iufu7Rg4VUqO0953yAp36opB3sE8nfNEHvk
	 lAdTC4HX/soZtwVUc8NTyDAOY5AlfeBt9GrwONHJTGY+qnr3yqT0lVMOTQXvdoq2eRrL
	 cg6w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;
    s=arc-20160816;	h=subject:from:to:user-agent:mime-version:date:message-id
	 :dkim-signature;
	bh=6ebdA/otT+4IxXlxuGcD9WM90CrApxfN29I3kxuTSSQ=;
	b=G7sivSbWm0FYSO50iwxA4WVjeqRzuT8J3LDCM4uTzO4e4aNFUcKDfIcMYibmR6y2Dx
	 I/rMY1Rkpxftmqkod3HuJ27h8x4m4bS/zdIk92J0VhIZK4YncokcLA/jRoTVsa4NnDCM
	 D8gWWfD9joP9OhBHHZpzUCOwFiRuAsnOC4ZwpentGazt7KVeVleWpZEiu/tFbJvUdszS
	 +IPHbQvXkXUAc5dPovdDpvj9rkvRbMFgH0RJ7Jx+/sYV0whGnWyEVHXWi5gK5l6xjasi
	 19FL3MC+4Q36NzOInBsxmYF9P+gDWn3+fKnFJjGsCov+O+ytkuFMkPcBF4g+R+Igvam0
	 G0Kw==
ARC-Authentication-Results: i=1; mx.google.com;
	dkim=pass header.i=@gmx.net header.s=badeba3b8450 header.b=c+GkZtfO;
	spf=pass (google.com: domain of XXX@gmx.de designates 212.227.15.18 as
	 permitted sender) smtp.mailfrom=XXX@gmx.de;
	dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de

The public key can be fetched like a DKIM key (line wrapped):

> dig +short arc-20160816._domainkey.google.com TXT
"k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Lztpxs7yUxQEsbD
FhjMc9kZVZu5P/COYEUIX4B39IL4SXAbv4viIlT9E6F6iZmTh1go7+9WQLywwgwjXMJx/D
z0RgMoPeyp5NRy4l320DPYibNqVMWa5" "iQ2WiImQC0en1O9uhLLvzaSZJ03fvGmCo9jM
o0GwKzLNe14xMgn/px2L5N/3IKlKX4bqUAJTUt8L993ZlWzvgMnSFSt8B+euSKSrtAiopd
y4r1yO4eN5goBASrGW0eLQc1lYouNvCrcTQpos4/GEAqiGzpqueJLmBfOO4clNvVvpPkvQ
s2BHw9I9LmIjaMxTNGxkGBRaP3utDiKXXqu1K+LRzl0HCNSdQIDAQAB"

Some random ARC selectors:

Domain Selector
google.com arc-20160816
microsoft.com arcselector9901

Glossary

MTA
Mail Transfer Agent
DNS
Domain Name System
ADMD
Administrative Management Domain
indirect mailflow
A mailflow with intermediate hops like forwarding, mailing list, … See RFC 7960

  1. Pressing the “Forward To” button in the mail client is still possible, but that creates a new mail. ↩︎