[Chicago-talk] core dump using DBI

[Brian Katzung]

Hi Michael.

If I'm following you correctly, you're talking about 1,080,000 queries 
and who knows how many results row fetches.

I would seriously try to take a step back and see if you can refactor. 
With that amount of iteration and number of database requests, would it 
make more sense to make a single linear pass through all the rows of all 
the relevant tables and build your local (in memory) data structures in 
parallel?

   - Brian

Michael Potter wrote:
> Monks,
> 
> I am core dumping in this fragment of code:
> ----------------------------------------------
>         foreach my $TableName (@TableNames)
>          {
>             my $SelectSub = qq{
>                SELECT * FROM
>                      $TableName
>                   WHERE ARTKEY1 = ?
>             };
> 
>             my $sth_ssa = $DbHandle->prepare($SelectSub);
>             $sth_ssa->execute($artkeyA);
>             $Counter = 0;
>             while (my $hsa = $sth_ssa->fetchrow_hashref())
>             {
>                $Counter++;
>                delete $hsa->{ARTKEY1};
>                $hsa->{TABLENAME} = $TableName;
>                push @aa, $hsa;
>             }
>             $sth_ssa->finish();
> 
>             my $sth_ssb = $DbHandle->prepare($SelectSub);
>             $sth_ssb->execute($artkeyB);
>             $Counter = 0;
>             while (my $hsb = $sth_ssb->fetchrow_hashref())
>             {
>                $Counter++;
>                delete $hsb->{ARTKEY1};
>                $hsb->{TABLENAME} = $TableName;
>                push @ab, $hsb;
>             }
>             $sth_ssb->finish();
>          }
> -------------------------------------
> 
> I pulled up the core dump in the debugger and it is failing with this error:
> malloc_common.extend_brk(internal error: assertion failed at line 3626
> in file frame.c
> 
> There is another loop that surrounds this loop and runs about 27,000
> times.  The loop you see runs about 20 times for each of the 27,000.
> It fails at about 6200.  I shifted the data to confirm that the actual
> data is not the problem.
> 
> I am on AIX/DB2/Perl 5.8.5.
> 
> I know it would be ideal if I could create a sample application that
> would reproduce this problem, but with all the data required to
> reproduce the problem this is not practical.  I also suspect the
> problem is in DBI and way over my head to debug.
> 
> My next action will be to move this loop to it's own script so that
> memory will be cleaned up by the termination of the script, then I
> will execute that script 27,000 times.
> 
> Any better ideas?
> 

-- 
Brian Katzung, Kappa Computer Solutions, LLC
Leveraging UNIX, GNU/Linux, open source, and custom
software solutions for business and beyond
Phone: 877.367.8837 x1  http://www.kappacs.com