[tpm] how to do pop3s for gmail

[Rob Janes]

The Q at the bottom is about changing the IO::Handle api to accomodate
IO::Socket::SSL's pending method.

I wrote a perl to capture gmail, first using imap via
Mail::IMAPClient, then pop3 via Net::POP3.  with pop3 the difficulty
is that Net::POP3 doesn't do ssl, for pop3s, which is all gmail
supports.

There's a nice hack, Net::POP3::SSLWrapper, that shows how to convince
Net::POP3 to do pop3s.  I call it a nice hack because it makes perl
look like ruby, and I didn't know you could do that.  Trouble is, it
locally overrides @ISA, and I wanted to same code to work for pop3 and
pop3s, and one sub to connect, and another sub to copy.  The override
for @Net::POP3::ISA has to be in effect for all code that uses the
$pop handle.

But, that's a minor problem.  Other minor problems are that gmail
messes up the headers sometimes, so as to put extra blank lines in the
middle, and that's just not rfc822-kosher.  Minor problems, easy to
fix.

The more difficult problem, is that the download would get stuck in
the middle.  I tracked the problem down to Net::Cmd, in the getline
method ...

    my $select_ret = select($rout = $rin, undef, undef, $timeout);
    if ($select_ret > 0) {
      unless (sysread($cmd, $buf = "", 1024)) {
        carp(ref($cmd) . ": Unexpected EOF on command channel")
          if $cmd->debug;
        $cmd->close;
        return undef;
      }

Net::Cmd is a package you stick on top of your IO::Handle thing to
layer in command line protocols like pop3, smtp or imap.  It has
methods like getline, CRLF munging, and read_until_dot.

This snippet would get stuck in the middle of a message, and after the
select timeout the remainder of the message would get strewn about in
the response to the next few retrieval commands.

Turns out the IO::Socket::SSL handle had data to read in a buffer, but
the actual socket had nothing.  The last little bit of the message was
still in the SSL buffer.  Since the socket had nothing, the select
call blocked until the timeout.  My clue was a comment in
Mail::IMAPClient, which gets the ssl stuff right, but it's
relationship with the socket is has-a, not is-a.  Until I saw that
code I was chasing a red hering from using openssl - openssl s_client
-connect pop.gmail.com:995 -crlf.  Without -ign_eof, every time you
type in a line that starts with an "R", like "RETR 5", openssl
renegotiates the ssl connection, which sends the connection into
limbo, after showing an unhelpful message "RENEGOTIATING" when
"GOODBYE" would have been more appropriate.

The fix is a hack to Net::Cmd,

my $select_ret = UNIVERSAL::can($cmd, 'pending') && $cmd->pending ? 1
: select($rout = $rin, undef, undef, $timeout);

I don't like the hack, but I couldn't figure out any other way to
override the behaviour of Net::Cmd::getline.

It works, but there's a general problem here, that of an IO::xxx
compression or encryption stream that buffers data, such that a select
system call will not reveal the presence of available buffered data.
IO::Socket::SSL is the only one with a pending method, but I'm sure
the other ones have the same problem.  So, this hack to Net::Cmd
allows it to work with interchangeably with IO::Socket::SSL and
IO::Socket::INET, but probably not IO::Compress::Bzip2 (although due
to the very large buffer in the bzip2 protocol I can't imagine it
being used for command line protocols like pop3).

Seems to me the IO::xxx api needs to handle this, but I'm not sure who
to put this to.  Any ideas what to do with this?  Is there a better
way to do this?

-rob