Repository navigation
smtplib send_message should add Date header if it is missing, per RFC5322 #73065
Description
Activity
HenningvonBargen commented
on Dec 5, 2016 HenningvonBargenmannequinMannequinAuthorI'm using CPython 2.7 with the smtplib and email modules to send emails with SMTP.
Today, one of our clients complained that the email sent is not RFC 5322 compliant because the required Date header is missing. The RFC states in section 3.6.:"The only required header fields are the origination date field and
the originator address field(s). All other header fields are
syntactically optional."Our program has been sending millions of email message this way and this is the first complaint.
I guess that the library doesn't add the header field automatically.
That is correct. Most SMTP gateways add the header on submission if it is missing. At least a few other MUA programs do not automatically add the Date header, they let the SMTP server do it. I have one person who sends me email that saw this same problem (I had my server set to reject messages without the date header), and they weren't using smtplib to send. (I think the sender used Exchange, though I don't remember for sure.)
HenningvonBargen commented
on Dec 6, 2016 HenningvonBargenmannequinMannequinAuthorI can give a little more information.
First, I created a very simple stand-alone test script (for Python >= 2.6):#!/bin/env python
# -- coding: utf-8 --import smtplib
# Adjust these!
HOST = "smtp.nowhere.local" PORT = 25 from_name = u"Test Python smtplib" from_addr = u"valid.address@nowhere.local" to_name = u"Python Test Recipient" to_addr = u"some.address@yourcompany.com" subject = "Test Nr. 1" smtp = smtplib.SMTP(HOST, PORT, timeout=10) smtp.set_debuglevel(1) smtp.sendmail(from_addr, [to_addr], subject) smtp.quit()
This script shows the same behavior, which shows that the problem is not related to my program, but it is indeed a problem with smtplib.py.
Unfortunately, I could only test this with Python 2.6, because I'm not allowed to install Python 2.7 on that machine; OTOH I have to run it on that machine to have access to the mail server.
The observations are:
-
The message is sent (I sent it to one of my email addresses).
-
Spam Assassin flags the message with additional header lines: X-Amavis-Alert: BAD HEADER SECTION, Missing required header field: "Date"
X-Spam-Status: No, score=-2.36 ... -
In some cases the message is silently dropped, because the missing date header caues a negative score; which in turn may be bad enough to classify it as spam, depending on the content.
-
The SMTP server used here is Postfix.
-
When I receive the message, a Date header is present (obviously it has been inserted during the message's journey through the internet).
Personal opinion:
I think classifying a message as spam just because a practically useless header field is missing is bad behavior of Spam Assaassin.
Nevertheless, the standard library should try to conform to RFC 5322.
It is good practise to be forgiving while reading but pedantic while writing.-
As I implied but did not say explicitly, this is the expected behavior of smtplib. You are responsible for adding any headers to the message that you want smtplib to send. In particular, the 'sendmail' method takes a string to send, and smtplib does not modify it except for cr/lf transformation. smtplib does not itself understand RFC5322 message syntax.
We can, however, add a Date header in the new send_message method of the python3 email library, because it accepts a Message object, and smtplib can use the knowledge the Message object encapsulates to check for the Date header and add one if it is missing. That's why I've left this issue open. I have now adjusted versions accordingly (ie: this is not a bug in python2.7, it is an enhancement request for Python3). Sorry I wasn't clear about this earlier.
Hmm. Actually, we can argue that it is an RFC compliance issue, as you have suggested, and change it in 3.6 as well, since it isn't likely to break anyone's working code. So I'll put 3.6 in the versions unless someone objects to that logic. But even after this is changed in python3, the smtplib sendmail method will not add a Date header, only the send_message method.
- changed the title
[-]smtplib RFC 5322 date header missing[/-][+]smtplib send_message should add Date header if it is missing, per RFC5322[/+]on Dec 6, 2016 elafontaine commented
on Dec 6, 2016 elafontainemannequinMannequinHi, Not sure this is where the comment goes...
I work with the smtplib and email libraries. I understand Henning von Bargen when he say that we should have a way to support RFC 5322 without asking the user to understand how to support it. The issue is that the SMTP protocol is NOT the protocol that format the e-mail. SMTP is the protocol that identify the "from", the "to", start encryption and finally transfert the message. The actual e-mail content is all passed inside the SMTP "DATA" Command. I strongly believe that an email should not be modified by a SMTP library.
the discussion should be focused on trying to make it available to the user WITHOUT changing the current behavior of email.message class.
In other words, I disagree to change the SMTPlib module and suggest that it's how you construct your email in the first place that should consider it;
class MessageRfc5322(email.message.Message):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
if self.get('Date', None) is None:
self.add_header('Date', email.utils.formatdate())msg = email.message_from_string(string_message, MessageRfc5322)
But, that's my opinion as someone who uses the smtplib and email library but also need to support rfc822 clients...
I tend to agree with Eric Lafontaine, looking at the quote Henning von Bargen posted the originator address field is also required, but yet we don't explicitly check its presence in the code, but rely on the SMTP server to error out.
The sendmail function will never modify the RFC822+ content. send_message, however, already does several manipulations of the message headers to make sending email simpler. Practicality (make it easy to send messages without knowing the details of the SMTP/RFC5322 rules) beats purity (an SMTP library should not modify the content of the DATA) in this case, especially since smtplib *does* provide the purity version if you prefer to work that way.
That is, 'sendmail' is pure SMTP, while send_message is a practical enhancement that provides additional services to the caller, and can (and does and probably should) do checks for RFC required headers.
If someone wants to do a purity refactor (to which I would not object, and would in fact encourage), the RFC5322-aware code could be factored out into one or more functions or Message object methods in the email library that smtplib would call from the send_message method.
My visualization is that the email library should allow you to construct a valid email in whatever order you want (adding the Date header late in the process, for example), but should support validating the email before it is sent. One way to do this would be to have the SMTP policy do the unambiguous fixups such as date headers, and raising errors for the rest (probably only if the strict flag is on, at this point) when the message is serialized.
Note that we previously fixed send_message to add a Resent-Date header if there are Resent- headers and no Resent-Date, so the precedent is already set (that is, smptlib send_message is already "not pure" :)
I've chatted a bit with David about this feature. Here are some thoughts:
- check what SMTP standard says about some validation rules
- add validate method, probably into email package
elafontaine commented
on Dec 16, 2016 elafontainemannequinMannequinHi all,
Thanks for the enlightment. I never figured that there was a send_message function XD. Never needed it and it's true that the example in the email library use sendmail and not send_message.
https://docs.python.org/2/library/smtplib.html#smtplib.SMTP.sendmail
https://docs.python.org/3.5/library/smtplib.html#smtplib.SMTP.send_messageThis function is fairly recent (python 3.2) from what I see.
Reading the documentation of the python 3.5 send_message function :
"[...] If from_addr is None or to_addrs is None, send_message fills those arguments with addresses extracted from the headers of msg as specified in RFC 5322: from_addr is set to the Sender field if it is present, and otherwise to the From field. to_addrs combines the values (if any) of the To, Cc, and Bcc fields from msg. "As we're already using this function for convenience of the RFC 5322, then I agree to add it. We should also modify the doc & comment inside the code to make it clear that date is added if absent and following RFC 5322. (I've looked at the source and the send_message only mention RFC2822 in the comments, no RFC 5322).
Finally, why would we want to add a validate fonction to the email library? What would it do ? validate that we respect a certain RFC? Who other than SMTPlib would use it? I would like to understand the reasonning behind it.
Again, all this are opinions to let the discussion continue :).
For now, what I see we need to do (this bullet point list is intended to be expanded with what you think we need to do):
- implement a patch for the code to add a missing "Date" field if it doesn't exist
- Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate
- Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).
- Fix it on all Python3 versions? It should have been supported since 3.2 right?
As it's my first time trying to contribute... I still don't know how to do so...
Regards,
Eric Lafontaine
eric.lafontaine1@gmail.com <= if you can help me outside of this discussion to contribute, it would be my pleasure.Sure the comments can be updated. Some of them elsewhere have been already.
The reason for the email library to have a validation function is that it has an 'SMTP' policy that is designed to produce valid SMTP messages when the message is serialized. (It also has an HTTP policy that is designed to produce valid HTTP header blocks...though I'm sure there are bugs there as well). These policies are relatively new, even newer than the send_message method of smtplib.
If this gets done it time it could be fixed in 3.5...Larry is planning the final non-security-fix 3.5 release some time after 3.6.0 final goes out the door, as is our tradition. After that it could only go into what will very shortly be the maintenance release (3.6) and the next feature release (3.7).
To contribute, create a diff against the tip of the default branch and post it here. (We will be switching to github "soon", but posting a patch here will always work). See docs.python.org/devguide for more details on contributing.
11 remaining items
I'm happy to comment on issues as far as mentoring goes, and yes email is my primary area of responsibility for CPython. However, I don't have much free time, so getting to reviews is proving to be problematic. There's a chance a might have some time this weekend, but no promises :)
elafontaine commented
on Jan 17, 2017 elafontainemannequinMannequinHi all,
The IETF didn't answer yet :(.
I'll await your news regarding this patch ("issue_28879_V4.patch").
I would like to have feedback if I need to change something.
Thanks a lot in advance,
Eric Lafontaineelafontaine commented
on Jan 18, 2017 elafontainemannequinMannequinHi all,
I've received an answer from the IETF Pete Resnick. The answer does say however that there is no guaranteed way of getting the right headers if the message doesn't respect the standard;
"[...]
In this case, if the trace fields were not present, you would not be able to clearly distinguish. Certainly you know which blocks the Resent-Date and Resent-From belong to, and in this case you know which blocks the Resent-Message-ID belong to (since there are two of them and only two Resent-* blocks), and you know which block the Resent-Bcc belongs to (because it comes between the first Resent-Date and Resent-Message-ID), but you can't tell which block the Resent-To: belongs to. And if the either Resent-Message-ID was missing, you would be unable to tell where the Resent-Bcc or Resent-Message-ID belongs. This is simply a weakness in the standard.However, the trace fields should exist, and that should divide the Resent-* blocks. As it says in 3.6:
header fields SHOULD NOT be reordered when a message is transported
or transformed. More importantly, the trace header fields and resent
header fields MUST NOT be reordered, and SHOULD be kept in blocks
prepended to the message.I hope that helps.
pr
Pete Resnick http://www.qualcomm.com/~presnick/
Qualcomm Technologies, Inc. - +1 (858)651-4478
"
In other words, if we were to take an e-mail that would have more than one Resent- headers, there should be traces in between the resent block. If there are no traces for a Resent- block, and we can detect there are 2 blocks in what is supposed to be a block , we should raise an error.
However, if we were to do the implementation and make it to detect "blocks" of resents (with the position of the resent for example). We could use the first block as the one we should be using and see if there are 2 Resent-Dates or 2 Resent-From in that block only.
What do you think David? I would like to know :).
P.s. I can forward the e-mail to does who want to have it :).
Regards,
Eric LafontaineI think this constitutes the heuristic I was talking about in that comment, that will get it right 99+% of the time. Strict mode should raise an error, but strict is not the default in the email package.
I probably won't have time to do any review for a while yet, I'm afraid.
elafontaine commented
on Jan 23, 2017 elafontainemannequinMannequinHi,
I've implemented the heuristic, but it's messy with the issue of this ticket.
I'm going to do some clean-up and separate the issue from the heuristic and post them separated.
Date issue ;
Test Case
Documentation
ImplementationHeuristic of Resent ;
Test Case
Documentation
ImplementationRegards,
Eric Lafontaineelafontaine commented
on Jan 31, 2017 elafontainemannequinMannequinHi all,
Here you go :).
Regards,
Eric Lafontaineelafontaine commented
on Jan 31, 2017 elafontainemannequinMannequinResent-heuristic
elafontaine commented
on Feb 18, 2017 elafontainemannequinMannequinHi,
Could someone put this ticket on "waiting for review"?
Regards,
Eric Lafontaineelafontaine commented
on Jul 1, 2017 elafontainemannequinMannequinHi All,
Should I try to make this a github PR instead to accelerate the review?
Regards,
Eric LafontaineYes. There's a chance someone else will review it if you do that. I'm still not likely to have time for a while, myself, but ping me again in a month if I haven't gotten to it.
jenstroeger commented
on Sep 8, 2018 jenstroegermannequinMannequinAny updates on this? Looks like the proposed change has not been merged into mainstream yet?
I’m having problems with Google rejecting emails:
(555, b'5.5.2 Syntax error, goodbye. r10-v6sm7321838qtj.41 - gsmtp', '…')and using IETF’s message linter (https://tools.ietf.org/tools/msglint/) I get the following:
ERROR: missing mandatory header 'date' lines 1-7 ERROR: missing mandatory header 'return-path' lines 1-7amongst a few others.
Reacted by Strubbl, Felix N, Dani and Antonio Spadaro
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs