Wednesday, July 11, 2012

Understanding Email Headers, Part III - The Received Header, or How'd It Get Here?

(If you haven't read Part 1 - "What Are They, and Where Can I See Them?" and Part 2 - "The Basics" yet, you probably should.  We'll wait...)

Tonight, we're going to talk about the Received header.  RFC 5322, Internet Message Format, establishes an entire class of "trace" headers, but does not define them beyond their most basic format.  The reason for that is simple; we are not limited to any single delivery mechanism for Internet messages, nor does a single mail server always use the same delivery mechanism.  It's true that most of us receive Internet messages via Simple Mail Transfer Protocol (SMTP), but the RFC 5322 authors left the specific format of "trace" headers to the discretion of other protocol standards. Most of us use SMTP these days, so we'll look at the definition of Received in RFC 5321, Simple Mail Transfer Protocol.

The Received header simply says what system got the message, what system (name & IP address) they got it from, the date/time, and (optionally) the addressee for whom the message is intended.  You may also see indicators of what SMTP server software was in use at each step along the way.  Each SMTP server that touches a message prepends its Received header to those already present (if any).  So, a typical Received header might look like this:

Received: from mail02.connect.vmware.com ([209.167.231.113]) by SNT0-MC3-F18.Snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4900);
Wed, 11 Jul 2012 16:35:43 -0700

or this (note the presence of the "for whom" clause):

Received: from mercure3.sct.gouv.qc.ca ([142.213.66.19] helo=mercure.sct.gouv.qc.ca)
by magus.postgresql.org with esmtp (Exim 4.72)
(envelope-from <eric.fournier@cspq.gouv.qc.ca>)
id 1Rfv4l-0005Vc-Jx
for pgsql-admin@postgresql.org; Wed, 28 Dec 2011 15:05:57 +0000

For many of today's Internet email services, connectivity is so plentiful that users of Hotmail, Gmail et al. may only see one or two Received headers in a typical email message.  In the corporate world, however, such is not the case; many enterprises only send/receive Internet email through a particular SMTP relay (or relays), so you may see a number of Received headers.  If you do, the topmost is the most recent (and should reflect ultimate delivery.)  For instance, here are the Received headers from an email message sent from within ibm.com to hotmail.com:

Received: from e33.co.us.ibm.com ([32.97.110.151]) by COL0-MC1-F36.Col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4900);
Tue, 10 Jul 2012 20:40:25 -0700
Received: from /spool/local
by e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted
for <asdfqwerasdf@hotmail.com> from <asdfqwerwen@us.ibm.com>;
Tue, 10 Jul 2012 21:40:24 -0600
Received: from d03dlp02.boulder.ibm.com (9.17.202.178)
by e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted;
Tue, 10 Jul 2012 21:40:22 -0600
Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106])
by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id A425E3E4004E
for <
asdfqwerasdf@hotmail.com>; Wed, 11 Jul 2012 03:40:21 +0000 (WET)
Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167])
by d03relay04.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q6B3dt7X244944
for <
asdfqwerasdf@hotmail.com>; Tue, 10 Jul 2012 21:40:06 -0600
Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1])
by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q6B3denJ030524
for <
asdfqwerasdf@hotmail.com>; Tue, 10 Jul 2012 21:39:40 -0600
Received: from wtfbes02.edc.lotus.com (wtfbes02.lotus.com [9.32.140.208])
by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q6B3ddop030485
for <
asdfqwerasdf@hotmail.com>; Tue, 10 Jul 2012 21:39:39 -0600

Wow, that message went through quite a path!  Careful scrutiny of these Received headers can yield important information, so let's take a look. 

  • There's only one Internet-routable IP address in these headers; that must be an Internet SMTP relay for ibm.com
  • Note that the outbound SMTP relay added "from <asdfqwerwen@us.ibm.com>" to its Received header.  That's perfectly legal, and is often seen when a single mail relay handles messages to/from several domains.
  • One more point about that Internet SMTP relay - it's either a dual-homed system or parked in a NATted DMZ. It has an internal address of 192.168.1.133 (as seen in the third Received header) and an external address of 32.97.110.151 (as seen in the first Received header).
  • Looking at the hostnames and descriptions, it looks like there's an anti-virus appliance in the SMTP path - d03av01.boulder.ibm.com
    • Apparently, that anti-virus appliance isolates inbound messages until they're scanned - note that it accepted the message with an "AVin" service, but sent it to itself (via loopback [127.0.0.1]) for further delivery by the "AVout" service.
  • The first SMTP server to handle the message (the last Received header) was "wtfbes02" - that strongly implies a Blackberry Enterprise Server (BES).  (Yes, this message was sent from a Blackberry.)
  • The admins of d03dlp02 need to fix their system timezone; the RFCs specifically discourage the use of +0000 by systems outside that timezone.
  • We can identify three different SMTP server software packages used along the way - Microsoft SMTPSVC, sendmail ("8.13.8" and "8.14.4" are sendmail version numbers), and Postfix.
  • From the first Received header to the last, only 1 minutes' time was required to deliver this message.

That last observation is probably the most frequently use of Received headers in troubleshooting - finding the point at which a "late message" was delayed.  Yes, the sender is SUPPOSED to receive a "message delivery delayed" message if the mail path breaks down, but that doesn't always happen.  If you're ever asked "Why did this take so long?  They sent it yesterday morning!", you can take a quick look at the Received headers and determine where the logjam occurred.

Another important point to remember is that "from the same person" does NOT necessarily mean "traversed the same mail path."  For instance, mail sent from my Blackberry takes a different path than does mail sent from my Lotus Notes client, even though they carry the same From header when they hit your inbox.  Received headers tell the REAL behind-the-scenes story.

Next up - all those miscellaneous headers...

2 comments:

Sam Sawatzky said...

Great stuff, Wes!

I hope you can cover the X- spam/antivirus stuff; that stuff can seem like witchcraft when trying to figure out why a message was marked as spam (or not).

If you are taking suggestions, the DKIM and SPF stuff would be a nice peripheral topic.

John Michle said...

Some facts and other points given here are quite considerable and to the point as well, would be so far better idea to look for more of these kind for efficient results.

Field Service Management Software