Email Standards – basic and lesser known

What was SMTP, SPF, DKIM, DMARC, SRS and ARC again?

SMTP — Simple Mail Transfer Protocol

The “Simple Mail Transfer Protocol” (SMTP, see Wikipedia) is the standard mail transfer protocol. It is used by users to “submit” new emails to their local email server and for servers to “relay” the users' emails to the destination server. These two roles were historically handled by the same servers (meaning both “hardware” and “software”). Nowadays these roles are usually handled by different servers listening on different ports and – strictly speaking – slightly different protocols.

The “traditional” and “relaying” role(s) are covered by many RFCs describing the core protocol and quite some more RFCs defining extensions. The core RFCs (all titled “Simple Mail Transfer Protocol”) seem to be RFC 788 (1981), RFC 821 (1982), RFC 2821 (2001) and RFC 5321 (2008). RFC 821 is part of STD 10. It is strange, that the newer RFCs have not been elevated to STD 10.

The “message submission” role is described in another set of RFCs with the title “Message Submission for Mail” (RFC 2476 (1998), RFC 4409 (2006) and RFC 6409 (2011) which is STD 72) but the used protocol is still SMTP! The RFC (and the extensions, of course) “only” change the rules how the server can and should modify messages before delivering them, how to authenticate users and more stuff like that.

The following assumes a “relaying server” (aka. Mail Transfer Agent – MTA):

Given an recipient address like example@gmail.com the sender will lookup MX records for the domain part of the email address:

> dig +short gmail.com MX
5 gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
20 alt2.gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.
40 alt4.gmail-smtp-in.l.google.com.

The number in the MX record is a priority (i.e. lower is better). So the connection will be made to the server with the priority 5. If there is no MX record then a direct connection to domain gmail.com will be tried. This is called “implicit MX record”.

The connection to the destination MTA (i.e. relaying role) will use port 25. The MX MX record does not support different port numbers.

A minimal SMTP session looks like this:

> dig +short example.org MX
10 smtp.example.org
> telnet smtp.example.org 25
S:	220 smtp.example.org ESMTP software-id
C:	HELO mx1.example.net
S:	250 Hello mx1.example.net
C:	MAIL FROM:<alice@example.net>
S:	250 ok
C:	RCPT TO:<bob@example.org>
S:	250 ok
C:	DATA
S:	354 go ahead
C:	From: Alice <alice@example.net>
C:	To: Bob <bob@example.org>
C:	Subject: simple SMTP test
C:
C:	It Works! CU later!
C:	.
S:	250 OK
C:	QUIT
S:	221 bye
Connection closed by foreign host.

Although this will not work for the big email providers like Gmail and Outlook.com. These will require additional headers like Message-ID: and will still reject the message due to missing other stuff like missing PTR records (aka “reverse DNS”, rDNS). Google talks about that here: Prevent mail to Gmail users from being blocked or sent to spam

SPF — Sender Policy Framework

The “Sender Policy Framework” (SPF, see Wikipedia) is an email authentication standard. The receiving MTA can verify that the sending MTA is actually allowed to send emails for the domain given in the MAIL FROM: envelope address within the SMTP connection. In some situations also the domains in the HELO / EHLO greeting handshake are checked. Since the the receiver only sees the IP address of the sender the available tests resolve around different kinds of DNS tests. The policy itself is also in DNS.

The RFCs for SPF are RFC 4408 (2006) and RFC 7208 (2014) having the title “Sender Policy Framework (SPF) for Authorizing Use of Domains in E-Mail, Version 1”.

Given an alleged sender example@gmail.com in the MAIL FROM: envelope the receiver will lookup a TXT DNS record which starts with v=spf1:

> dig +short gmail.com TXT
"v=spf1 redirect=_spf.google.com"

Following the redirect (wrapping some long lines) returns:

> dig +short _spf.google.com TXT
"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com
 include:_netblocks3.google.com ~all"

Digging into one include returns (also wrapped and slightly reformatted):

> dig +short TXT _netblocks.google.com
"v=spf1
ip4:35.190.247.0/24 ip4:64.233.160.0/19 ip4:66.102.0.0/20  ip4:66.249.80.0/20
ip4:72.14.192.0/18  ip4:74.125.0.0/16   ip4:108.177.8.0/21 ip4:173.194.0.0/16
ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19
~all"

_netblocks2.google.com and _netblocks3.google.com are structurally similar.

It is interesting that Google uses ~all which means “Soft Fail”.

Some points about SPF:

• SPF is about origin, not content! So it’s not – by itself – a spam / spoofing / phishing protection since it only checks the MAIL FROM: envelope address and not the From: header in the email. This envelope address is not presented to the user. But see DMARC which does “align” both parts.

• SPF makes it impossible to forward emails using forward rules in the server.¹ properly:

  Why exactly? A traditional forward without SPF worked like this:

  System A sends an email to B:

   MAIL FROM:<alice@a.examle.net>
   RCPT TO:<bob@b.example.org>
  

  System B forwards the email to system C:

   MAIL FROM:<alice@a.examle.net>
   RCPT TO:<charlie@c.example.edu>
  

  System C accepts the mail and the easy / good case is closed. But if system C cannot deliver the mail later on then C sends a “bounce message” back to Alice.

  With SPF activated (but without DMARC, see later) system C will reject the email, because the B is not allowed to send emails on behalf of A. But B could do this:

   MAIL FROM:<bob@b.example.org>
   RCPT TO:<charlie@c.example.edu>
  

  Now C accepts the message because even with SPF. But C must send a bounce message back it will address that to B/Bob and not back to Alice. But that bounce will most likely also fail if it is forwarded to C again. Alice will never know that the message didn’t make it Bob won’t know also.

  With SPF and DMARC enabled the forward from B to C will fail because the envelope address bob@b.example.org does not “align” with the header From: alice@a.example.net in the message itself.

  See Sender Rewriting Scheme for possible mitigations.

• Mailing lists have the same problem, since they used to forward mails on behalf of the original sender to many recipients.

Nevertheless – a small site might want to deploy SPF records in order to improve their reputation with the big email receivers like Gmail and Outlook.com.

DKIM — DomainKeys Identified Mail

“DomainKeys Identified Mail” (DKIM, see Wikipedia) is also an email authentication standard. The sending MTA signs selected headers and the body of the email using a private key. The receiving MTA fetches the corresponding public key from DNS and then validates the integrity of the mail.

The RFCs for DKIM are RFC 4871 (2007) and RFC 6376 (2011) both titled “DomainKeys Identified Mail (DKIM) Signatures”. The later RFC is STD 76. Note that RFC 4870 (2007) is a precursor of DKIM with a slightly different title.

A DKIM signature header from Gmail look like this:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112;
 h=content-transfer-encoding:subject:from:to:content-language
  :user-agent:mime-version:date:message-id:from:to:cc:subject:date;
 bh=ZIe1W6Yx/RQkrEdFbGsZMX4VV/Ib0+g8JIgM1R8S5i4=;
 b=LfDJmO3KmEEu8Ui1bngqK9T8ja3L2jc0Jwmbbi4j7MYWoiGXuq2MYyUVmeQawtohXc
  hDzC1K+MTOg09J3bxwVOqsPr8nTjz8dbj/Y/0TjGKy+X+NZ5CVIYaRmyPDkxVsNX1H3b
  VtGqPd5nE02c5CySe046d9j2hL72eAiTc9IHK6wy1V1K7UMKjbUNdkUsPcMUBIZnztzr
  YChVZ3s0hy9ijifHIqlOSSezkBMKYzD5sB9+7skZIjgk3VYGXkojZaZ6bqjI2U4XtVCo
  WNGKyj3Bohkw5qHtli618hbK1hOCC9/m8OD5wpZeeZJ6g03SsjkF7Ar+zCjDKmy4Ml3C
  9QUw==

b the actual signature, bh is the body hash, h the list of signed headers. d is the “signing domain identifier” and s the “selector”.

Using d=gmail.com and s=20210112 the public key can be fetched (wrapped):

> dig +short 20210112._domainkey.gmail.com TXT
"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq8JxVBM
LHZRj1WvIMSHApRY3DraE/EiFiR6IMAlDq9GAnrVy0tDQyBND1G8+1fy5RwssQ9DgfNe7r
ImwxabWfWxJ1LSmo/DzEdOHOJNQiP/nw7MdmGu+R9hEvBeGRQ"
"Amn1jkO46KIw/p2lGvmPSe3+AVD+XyaXZ4vJGTZKFUCnoctAVUyHjSDT7KnEsaiND2rVs
DvyisJUAH+EyRfmHSBwfJVHAdJ9oD8cn9NjIun/EHLSIwhCxXmLJlaJeNAFtcGeD2aRGbH
aS7M6aTFP+qk4f2ucRx31cyCxbu50CDVfU+d4JkIDNBFDiV+MIpaDFXIf11bGoS08oBBQi
yPXgX0wIDAQAB"

Some random selectors:

Some points about DKIM:

• DKIM is about (partial) content, not origin! So any host can send a signed message in a “replay attack”. If the To: Header is not covered by the signature then the message can be sent to anyone. If the l option limits the length of data covered by the body hash then a valid signed “prefix” message can be prepended to additional body data.

• The signing domain (d=) is not checked against the the user visible From: header. So evil.com can sign a message claiming to be from trustworthy@example.com.

• Forwarding emails without modification is possible as long as DMARC is not enabled also because DKIM – like SPF – does not “align” the envelope MAIL FROM: address to the From: header.

• Forwarding (as mailing lists do) is not possible: They change the To: header which is often covered by the signature. They also often add footers with information about the mailing list and unsubscribe links. They might also rewrite the From: header to point back to the mailing list and Subject: to include tags like [listname].

• More about DKIM and Mailing List can be found in RFC 6377 (“DomainKeys Identified Mail (DKIM) and Mailing Lists”, 2011)

While both DKIM and SPF perform email authentication they cover disjoint topics. Therefore both are – per se – not enough to reduce spam / spoofing / phishing. But both can impede legit email flows like forwarding and mailing lists. And like SPF – a small site might want to deploy DKIM in order to improve their reputation with the big email receivers like Gmail and Outlook.com.

DMARC — Domain-based Message Authentication, Reporting, and Conformance

“Domain-based Message Authentication, Reporting, and Conformance” (DMARC, see Wikipedia) is the third email authentication method.

The RFC for DMARC is RFC 7489 (2015) titled “Domain-based Message Authentication, Reporting, and Conformance (DMARC)".

DMARC relies on either SPF or DKIM or on both and adds some stuff on top:

• “Identifier Alignment”: For DKIM the domain in the From: header of the email and the d= option in the DKIM signature must match. For SPF the domain from the MAIL FROM: envelope part must match the domain in the From: header. In both cases there is some wiggle room for subdomains. This Alignment ties the information presented to the user to the technical declarations and as such benefits the user directly.

• Policy: The sender domain can state what the receiver should do when neither SPF nor DKIM succeeded: Accept the message nevertheless, quarantine it or reject it.

• Reporting: The sender domain can request “aggregate” reports or “forensic” reports for validation failures.

So DMARC really connects several loose parts of SPF and DKIM.

Of course DMARC also stores the information in a DNS TXT record:

> dig +short _dmarc.gmail.com TXT
"v=DMARC1; p=none; sp=quarantine; rua=mailto:mailauth-reports@google.com"

p specifies the policy for the domain (here: please accept even invalid messages), sp is the policy for subdomains (mark as spam). rua specifies the address to which aggregate feedback reports should be sent. The standard defines more fields but gmail.com uses the defaults for these.

Regarding the address for aggregate feedback report (rua): Since the sender’s domain (gmail.com) does not match with the one in the rua field (google.com) another DNS record is required to accept these mails:

> dig +short TXT gmail.com._report._dmarc.google.com
"v=DMARC1"

This means: I myself cannot send the reports for my toy domain to my gmail account because Google will not setup the DNS.

Regarding email forwarding and mailing lists: DMARC mostly makes things worse:

• With SPF the alignment prevents forwarding quite directly because the MAIL FROM: envelope address must match the From: header in the body. So the relay can neither insert its own address nor the original value in the MAIL FROM address. See the example above in SPF.

• With DKIM many headers and the body must not be altered. Additional headers can be used in for some things. See “Actually, DMARC works fine with mailing lists” for some details. But that means that both the mailing list software and all email clients must support the new headers.

  Some years have gone by since the introduction of DMARC. So this might have happened by now. Hopefully. But this does not cover all forwarding scenarios. RFC 7960 “Interoperability Issues between Domain-based Message Authentication, Reporting, and Conformance (DMARC) and Indirect Email Flows” (2016) lists some more and also some mitigations.

Some points about DMARC:

• If both SPF and DKIM are enabled then DMARC will fail only if both checks fail for an email.

• The policy (p and sp) cannot be configured for SPF and DKIM separately, but reporting of failures can!

Overall it’s the same as with SPF and DKIM: A small mailer should setup also DMARC in order to play nice with the big ones.

SRS — Sender Rewriting Scheme

“Sender Rewriting Scheme” (SRS, see Wikipedia) is a method to prevent SPF failures when emails are forwarded. It works by rewriting the MAIL FROM: envelope address.

Contrary to the other standards this is not an RFC but a PDF from the libsrs2 homepage.

SRS works like this: System A sends an email to system B:

MAIL FROM:<alice@a.examle.net>
RCPT TO:<bob@b.example.org>
From: Alice <alice@a.example.net>

System B checks SPF, DKIM and DMARC and all checks pass. But Bob had configured a forward rule to Charlie so B forwards the mail to C:

MAIL FROM:<SRS0=HHH=TT=a.example.net=alice@b.example.org>
RCPT TO:<charlie@c.example.edu>
From: Alice <alice@a.example.net>

It is similar to the example from SPF because the MAIL FROM: envelope address is rewritten to match the forwarders domain. If C accepts the message but wants to send a bounce message later on it will send it to B. B can validate the address using the cryptographic truncated hash HHH and the truncated timestamp TT, reconstruct the original address and forward the bounce back to Alice. The hash and the timestamp prevent spammers from abusing the system.

SRS also contains an SRS1= variant which used when multiple forwards happen: In that case subsequent forwarders don’t extend the address ad infinitum but only twice altogether.

However: This works only as long as DMARC is not enabled. Since SRS does not change the From: header the alignment fails. Wikipedia says:

Please note : Using SRS protocol fails the SPF Alignment check for your DMARC record, and it’s by design. Your DMARC record can still pass with a DKIM check. – Wikipedia – Sender Rewriting Scheme

So why bother with SRS at all? And doesn’t that imply, that any Alice must setup DKIM and no forwarder breaks the signature? Both seem unlikely even considered each on its own.

Some more links to read up more details:

• SRS Support in Outlook 365

  It’s interesting, that Microsoft support SRS.

• Google: Best practices for forwarding email to Gmail:

  Avoid changing the envelope sender when forwarding email to Gmail.
  […]If you never forward spam to Gmail, you can change the envelope sender to your domain.

• SRS support for Postfix: PostSRSd

• Serverfault: Is SRS rewriting absolutely necessary for a forwarding mailserver?

• Serverfault: Gmail rejects forwarded mail with DMARC but I AM using SRS

ARC — Authenticated Received Chain

“Authenticated Received Chain” (ARC, see Wikipedia) is also an email authentication method. It tries to address the forwarding issues of the other methods by allowing to override the DMARC policy with a local policy again.

The RFC for ARC is RFC 8617 (2019), titled "(The Authenticated Received Chain (ARC) Protocol)".

ARC builds upon DMARC and – transitively – on DKIM and SPF. The receiving MTA performs the DKIM, SPF and DMARC validation as usual, records the validation results, signs these and puts the records and the signature into headers. The signature also covers any existing ARC headers.

When the email is forwarded the next MTA will apply DKIM, SPF and DMARC again. SPF will fail in almost all situations. DKIM might fail if the forwarder modified the email or some signed headers. If both checks fail or the DMARC alignment fails a strict DMARC policy could call for rejection of the message. A receiver supporting ARC can overrule that policy if the ARC chain and the recorded result are valid and the receiver trusts the forwarder.

[arc-spec - Resources][http://arc-spec.org/?page_id=79] shows supporters for ARC. Google supports it with Gmail, G Suit and Google Groups. Microsoft is shown with limited support for Outlook 365.

Example headers of mail from GMX to Gmail:

ARC-Seal: i=1; a=rsa-sha256; t=1665601609; cv=none;
	d=google.com; s=arc-20160816;
	b=h85LEorEEyiTnn2V4dzJQTfRsAPLn6yk7e7HBKKuneFGcbpueyDFhA+eIeq0C9lXWn
	 K5vZHtEmvioImYHbBFIFSrO5NkR4xooHP7ugkWA0ceBN6+i/gisVHqVCanK2rmxxYDbA
	 n7lEfu5T999SraOGFIQwejjHgHdMVgiZOQdAkXPNJVralUuKJcQR6IXyliVUaagG/5CK
	 mLHILGQj6/fstHFn1J/REvsBq5ymk1LK6iufu7Rg4VUqO0953yAp36opB3sE8nfNEHvk
	 lAdTC4HX/soZtwVUc8NTyDAOY5AlfeBt9GrwONHJTGY+qnr3yqT0lVMOTQXvdoq2eRrL
	 cg6w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;
    s=arc-20160816;	h=subject:from:to:user-agent:mime-version:date:message-id
	 :dkim-signature;
	bh=6ebdA/otT+4IxXlxuGcD9WM90CrApxfN29I3kxuTSSQ=;
	b=G7sivSbWm0FYSO50iwxA4WVjeqRzuT8J3LDCM4uTzO4e4aNFUcKDfIcMYibmR6y2Dx
	 I/rMY1Rkpxftmqkod3HuJ27h8x4m4bS/zdIk92J0VhIZK4YncokcLA/jRoTVsa4NnDCM
	 D8gWWfD9joP9OhBHHZpzUCOwFiRuAsnOC4ZwpentGazt7KVeVleWpZEiu/tFbJvUdszS
	 +IPHbQvXkXUAc5dPovdDpvj9rkvRbMFgH0RJ7Jx+/sYV0whGnWyEVHXWi5gK5l6xjasi
	 19FL3MC+4Q36NzOInBsxmYF9P+gDWn3+fKnFJjGsCov+O+ytkuFMkPcBF4g+R+Igvam0
	 G0Kw==
ARC-Authentication-Results: i=1; mx.google.com;
	dkim=pass header.i=@gmx.net header.s=badeba3b8450 header.b=c+GkZtfO;
	spf=pass (google.com: domain of XXX@gmx.de designates 212.227.15.18 as
	 permitted sender) smtp.mailfrom=XXX@gmx.de;
	dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de

The public key can be fetched like a DKIM key (line wrapped):

> dig +short arc-20160816._domainkey.google.com TXT
"k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Lztpxs7yUxQEsbD
FhjMc9kZVZu5P/COYEUIX4B39IL4SXAbv4viIlT9E6F6iZmTh1go7+9WQLywwgwjXMJx/D
z0RgMoPeyp5NRy4l320DPYibNqVMWa5" "iQ2WiImQC0en1O9uhLLvzaSZJ03fvGmCo9jM
o0GwKzLNe14xMgn/px2L5N/3IKlKX4bqUAJTUt8L993ZlWzvgMnSFSt8B+euSKSrtAiopd
y4r1yO4eN5goBASrGW0eLQc1lYouNvCrcTQpos4/GEAqiGzpqueJLmBfOO4clNvVvpPkvQ
s2BHw9I9LmIjaMxTNGxkGBRaP3utDiKXXqu1K+LRzl0HCNSdQIDAQAB"

Some random ARC selectors:

Glossary

MTA

Mail Transfer Agent

DNS

Domain Name System

ADMD

Administrative Management Domain

indirect mailflow

A mailflow with intermediate hops like forwarding, mailing list, … See RFC 7960