[tpm] How to pipe variable content to shell scripts

[Antonio Sun]

Hi,

Thanks everyone for your replies.

On Tue, Jan 21, 2014 at 1:21 PM, Matthew Phillips wrote:

>
> On Tue, Jan 21, 2014 at 12:29 PM, Antonio Sun <tpm.ats at spamgourmet.com>wrote:
>
>> Hi,
>>
>> I have a script that works on xml content. A thousand time simplified
>> version is:
>>
>>  xml_output | perl -n000e 's,(?<=">)(.*?)(?=</HttpBody>),`echo $1 | wc
>> -c`,eg; print'
>>
>>
why are you doing `echo $1 | wc -c` instead of simply doing «length($1)»?


As said before, this is only thousand time simplified version, to stress
out what the problems is, instead of distract you with unrelated code.


> I.e., I need to pipe the matching string ($1, content between tag HttpBody)
>> to an external program via shell. As you can tell, if the matching
>> content is too big for shell parameter length, my script will fail.
>>
>

Take a look at perldoc perlipc (search for Using open() for IPC)
> Alternatively, use IO::All and something like ...
>

Previously, I thought the last resort would be bring in the big gun and do
something like the following,


   1.     open <http://perldoc.perl.org/functions/open.html>(SPOOLER,
"| cat -v | lpr -h 2>/dev/null")
   2.                         || die
<http://perldoc.perl.org/functions/die.html> "can't fork: $!";
   3.     local <http://perldoc.perl.org/functions/local.html>
$SIG{PIPE} = sub <http://perldoc.perl.org/functions/sub.html> { die
<http://perldoc.perl.org/functions/die.html> "spooler pipe broke" };
   4.     print <http://perldoc.perl.org/functions/print.html> SPOOLER
"stuff\n";
   5.     close <http://perldoc.perl.org/functions/close.html> SPOOLER
      || die <http://perldoc.perl.org/functions/die.html> "bad spool:
$! $?";
   6.

I.e, instead of using shell's "echo $1 | ", I'll write to my own file
descriptor opened by Perl just like above.

However, after trying that, I realized that it is not working.


> I *have to* process the matching string via the external program. Is
>> there any way I can get around this?
>>
>
Why not just print the extracted $1 to stdout ... that gets piped into wc ?


See below.

The problem is  that I not only need to process the matching string
via the external
program, but I also need to replace the matching string with the result of
the external process. Putting two together is where the problem is.

This is my previous code:

 perl -n000e 's,(?<=">)(.*?)(?=</HttpBody>),`echo $1 | wc -c`,eg; print'

This is what I tried just now:

  perl -n000e 'BEGIN { open(SPOOLER, "| tee /dev/tty | wc -c") || die
"cannot fork: $!"; local $SIG{PIPE} = sub { die "spooler pipe broke" }; sub
process { print SPOOLER "$_[0]"; }; };
s,(?<=">)(.*?)(?=</HttpBody>),process $1,eg; print'

Can you see what's wrong? -- the replacement would not be the result of "wc
-c", but all are 1.

So to recap,

I need to pick out a big chunk of input string (>200K), feed it to external
program (which is pipe after pipe after pipe), then replace the matching
string with the processed result. what's the proper way to do it (for big
matching chunks)?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/toronto-pm/attachments/20140121/887613ab/attachment.html>