[Chicago-talk] open3 and passing large amounts of data

[Andrew Rodland]

On Monday 17 November 2008 07:49:13 pm Zane C.B. wrote:
> Any one ever run into a issue before when passing a large amount of
> data through a pipe?
>
> I am currently trying to print '1022163' bytes to the STDIN spamc,
> but it is never finishing.
>
> What I found I had to do is set spamc's max size arguement to what my
> incoming mail server to something large or as large as what my mail
> server allows for. Which is odd as spamc does not hang if I cat
> something that large through it.

It's a deadlock. spamc stopped reading its input and sent you a message that 
went something along the lines of "whoa, bucko, that's way too much data!", 
but your app didn't receive that message because it was too busy trying to 
stuff data through the pipe going the other way, and wasn't checking for 
input. So each end was blocked waiting on the other. To avoid problems like 
that, you need to use nonblocking I/O plus a select loop or POE or Event or 
something like that, or threads (but Perl isn't very thread-friendly, so 
forget I mentioned that one).

Andrew