[DFW.pm] SpamMeNot design considerations - unicode? sysread vs. read? parsing? security?

[John Fields]

I was thinking over this....  I think it will be instructive to put an email's raw transmission on the projector. Will take 5 
minutes and everyone will go... THAT's ALL IT DOES? ;)

So on the good 'ol days side, the RFCs originally specify things in ASCII.  They then rolled in MIME to encode character sets, 
images, or anything OTHER than default (the charset now known as) us-ascii. So that is why Dovecot, the IMAP server, saves and 
serves to RFC spec - because it doesn't have to peek inside the MIME package to read the headers in the envelope.

So that means there is a clear separation between the envelope and the contents.  The envelope can now be encoded in UTF8 (I need to 
find the RFC where they loosened the us-ascii grip) but that is totally a separate issue from the MIME packages.  To be clear:

* The "body" of the email, everything the user sees in the message window, COULD be a MIME package encoded in a weird charset.
* Attachments are just MIME packages with a file pack'd encoding.
* The "headers" however, which is what we will be parsing including TO, FROM, SUBJECT, REPLY-TO, CC, BCC... those things must be in 
the universally accepted encoding or else the package won't even transit. That is where we will need to be UTF8 aware to future proof.

*$* Be advised, old codeset spammenot.com isn't UTF8 aware RIGHT NOW and it hasn't caused a single dropped email... that I know of. :)

 >-------
It will be interesting to perfbench the different read methods! Good catch on the EOF thing.



On 03/19/2013 07:34 PM, Tommy Butler wrote:
> If anyone's watched the git repo for the SpamMeNot project as introduced and discussed in our meetings for the last 2 months...
>
> You may have noted some things.  You may have noted that it has become both a functional daemon and it has a functional Catalyst 
> backend.  Profit!
>
>
>       Questions
>
> But while hacking on the codebase recently I've started to ask myself some questions.  Questions like:
>
>   * I've started this out with full unicode support for both incoming streams and what will eventually be passed to the message
>     storage mechanism (maildir), but do I really need unicode?  It's already probably base64 when it comes in right?  But what
>     about headers with unicode characters?  D*o(*main names can have *ü*nicode chars nowad*a(~*ys!
>   * The original codebase literally reads until *EOF*; this can't be good!  Someone could send me an email as big as my server's
>     hard drive!  In the newest code you'll see that I decided to *read()* one utf8-encoded char at a time, and limit the maximum
>     size of each header and the email's message content.  It has allowed for effective header parsing, but it's tedious and
>     probably dreadfully inefficient.  Wouldn't it be better to *read STDIN, $buffer, $max_msg_size* and if there's anything left
>     -- kick it back?  That's still DoS resistant, and much faster.  I could do post-processing and parse the headers/message after
>     the fact.
>
>
>       *The First SpamMeNot RFC
>       *
>
> _*What do you mongers think?*_ (That's the RFC)
>
> While you think about it, I'm going to *'git branch'* the code (again) and try:
>
>   * *sysread*-ing using a*:unix:encoding(UTF-8)* IO layer stack into a buffer that is *$max_msg_size* in one go (there's an
>     inherent bug in this -- can you identify it?)
>   * parsing/splitting the message and its headers after the fact (and use a trusty CPAN module to do this for me, because as you
>     know this makes one's coolness grow larger)
>   * ...we'll take it from there.
>
>
>       Things to consider
>
>   * You can always downgrade unicode strings later, but you can't always go the other way after you've saved a file with the wrong
>     encoding.  Isn't unicode the /"right"/ thing to do?
>   * Dovecot seems to store files with ASCII encoding (i.e.- no UTF-8 necessarily) --
>
>     [/var/vmail/tommybutler.me/ace/cur]
>     # file -i 1363734898.M59401P5961.peedoo,S=6676,W=6785:2,
>     1363734898.M59401P5961.peedoo,S=6676,W=6785:2,: _*message/rfc822; charset=us-ascii*_
>
>
> --Tommy Butler
>
>
> _______________________________________________
> Dfw-pm mailing list
> Dfw-pm at pm.org
> http://mail.pm.org/mailman/listinfo/dfw-pm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/dfw-pm/attachments/20130319/d1a06e48/attachment-0001.html>