From don at gamma.net.nz  Wed Mar  5 14:55:32 2003
From: don at gamma.net.nz (Don Jones)
Date: Thu Aug  5 00:24:18 2004
Subject: MYSQL Help!
Message-ID: <20030305205532.M39149@gamma.net.nz>

Hi

This is a plea for help!

This is a mysql question, with a perl script thrown in to keep it on topic, Im
asking here as I may get an intelligent response

I am not a DBA but unfortunately am the closest thing to one here, we have a
mysql db which I look after, its version: 3.22.26a on FreeBSD 4.0-RELEASE (ie
OLD). This db does a bunch of isp related stuff, ie dialup users info and ip
billing.

The problem is that there is a perl script which updates a particular table
each night. 

I ran this command:

mysql> delete from NeTraMet_Daily where date like '2001%';
ERROR 1034: Incorrect key file for table: 'NeTraMet_Daily'. Try to repair it

now when the script runs mysql will segfault

I asked for help on #mysql on irc.freenode.net where they politely told me I
should have upgraded 2 years ago, this is true but unfortunately doesnt help
my imediate problem. 

I was also advised to use the tool myisamchk, unfortunately my version seems
to old to include this tool.

Any suggestions on how to "Try to repair it" would be welcome.

Thanks

Don Jones


From Grantm at web.co.nz  Wed Mar  5 15:25:05 2003
From: Grantm at web.co.nz (Grant McLean)
Date: Thu Aug  5 00:24:18 2004
Subject: MYSQL Help!
Message-ID: <5FA042F680739D44951B00FED8BE1A7D42FD06@dasher.webdom1.web.co.nz>

Sorry Don, forensic MySQL isn't my specialty.  You might want
to try the WellyLUG mailing list.

===============================================================
Grant McLean        BearingPoint Inc - formerly The Web Limited
+64 4 495 9026                     135 Victoria Street, Box 920
gmclean@bearingpoint.biz                Wellington, New Zealand



> -----Original Message-----
> From: Don Jones [mailto:don@gamma.net.nz]
> Sent: Thursday, March 06, 2003 9:56 AM
> To: wellington-pm-list@pm.org
> Subject: MYSQL Help!
> 
> 
> Hi
> 
> This is a plea for help!
> 
> This is a mysql question, with a perl script thrown in to 
> keep it on topic, Im
> asking here as I may get an intelligent response
> 
> I am not a DBA but unfortunately am the closest thing to one 
> here, we have a
> mysql db which I look after, its version: 3.22.26a on FreeBSD 
> 4.0-RELEASE (ie
> OLD). This db does a bunch of isp related stuff, ie dialup 
> users info and ip
> billing.
> 
> The problem is that there is a perl script which updates a 
> particular table
> each night. 
> 
> I ran this command:
> 
> mysql> delete from NeTraMet_Daily where date like '2001%';
> ERROR 1034: Incorrect key file for table: 'NeTraMet_Daily'. 
> Try to repair it
> 
> now when the script runs mysql will segfault
> 
> I asked for help on #mysql on irc.freenode.net where they 
> politely told me I
> should have upgraded 2 years ago, this is true but 
> unfortunately doesnt help
> my imediate problem. 
> 
> I was also advised to use the tool myisamchk, unfortunately 
> my version seems
> to old to include this tool.
> 
> Any suggestions on how to "Try to repair it" would be welcome.
> 
> Thanks
> 
> Don Jones
> 
> 

From shaunm at web.co.nz  Wed Mar  5 17:50:35 2003
From: shaunm at web.co.nz (Shaun McCarthy)
Date: Thu Aug  5 00:24:18 2004
Subject: [off-topic] MYSQL Help!
Message-ID: <5FA042F680739D44951B00FED8BE1A7D3581D9@dasher.webdom1.web.co.nz>

I have had a similar problem once before. In my case it was because MySQL had a cached version of the schema which I had changed, and everything worked fine after I restarted the daemon. But I guess you have already done that?
 
MySQL can also throw weird errors if your SQL isn't correct. Have you done a "select from NeTraMet_Daily where date like '2001%'" to make sure the query is all good? Is "date" a date field? In which case you should really be using "where year(date)=2002"
 
Assuming it is a corrupt table, which it does sound like - do you have isamchk instead of myisamchk? (if you are using ISAM tables you should be using this anyway) but my memory of MySQL is fading (it might just be an internal command in older versions?) and my dev computer is on the other side of the world at the moment and I don't fancy trying to use it from an internet cafe :)
 
All else fails: RTFM:
http://www.mysql.com/doc/en/Repair.html
 
Or install a new version of mysql and make sure your fingers are crossed
 
Or leave
 
Shaun

	-----Original Message----- 
	From: Don Jones [mailto:don@gamma.net.nz] 
	Sent: Thu 6/03/2003 7:55 AM 
	To: wellington-pm-list@pm.org 
	Cc: 
	Subject: MYSQL Help!
	
	

	Hi
	
	This is a plea for help!
	
	This is a mysql question, with a perl script thrown in to keep it on topic, Im
	asking here as I may get an intelligent response
	
	I am not a DBA but unfortunately am the closest thing to one here, we have a
	mysql db which I look after, its version: 3.22.26a on FreeBSD 4.0-RELEASE (ie
	OLD). This db does a bunch of isp related stuff, ie dialup users info and ip
	billing.
	
	The problem is that there is a perl script which updates a particular table
	each night.
	
	I ran this command:
	
	mysql> delete from NeTraMet_Daily where date like '2001%';
	ERROR 1034: Incorrect key file for table: 'NeTraMet_Daily'. Try to repair it
	
	now when the script runs mysql will segfault
	
	I asked for help on #mysql on irc.freenode.net where they politely told me I
	should have upgraded 2 years ago, this is true but unfortunately doesnt help
	my imediate problem.
	
	I was also advised to use the tool myisamchk, unfortunately my version seems
	to old to include this tool.
	
	Any suggestions on how to "Try to repair it" would be welcome.
	
	Thanks
	
	Don Jones
	
	


From don at gamma.net.nz  Thu Mar 13 15:53:26 2003
From: don at gamma.net.nz (Don Jones)
Date: Thu Aug  5 00:24:18 2004
Subject: script causing high cpu usage
Message-ID: <20030313215326.M25562@gamma.net.nz>

Hi again

I have another question based on some problems I have with some important, old
code from my organisation, that I am the lucky maintenance programmer for.
This code is from one of 3 scripts that gets run from cron once a month to do
billing for internet usage, it interfaces with a mysql database to get its
data and outsput's  various csv's html and pdf's. 

The problem is as soon as the script runs mysqld quickly gobbles up all the
cpu usage on this box (which happens to perform a lot of other important
functions). The code that seemsto be generating the excessive usage is
included below. the initial $Team places approximately 1700 ip addresses in
@Addresses. It then does a foreach $address (sort byIpAddress @Addresses)
block, there are numerous db accesses in this block (I havent included the
whole block, theres a bunch more code).

The problem as i see it is that there are _NO_

$sth->finish();
$dbh->disconnect();

statements in this whole script (and there are 2 more large subs doing similar
things to this one).

I think the excessive cpu usage by mysqld is being caused by many db
connectionsbeing made to the bd and then not being closed properly.

Obvious sloution is close the connections

QUESTIONS:

1. Do you agree with my assessment or is there something else fundamental Im
missing?

2. If i am correct how is it best to use $sth->finish(); and
$dbh->disconnect();. Is it best to use a finish after each query (after the
fetchrow_array )and a disconnect at the very end of the script (my guess)? or
some other configuration of finish and disconnect?

As usual any advice/help appreciated. The scary thing is this code has been
running "successfully" each month for the last few years.

Thanks

Don Jones

Offending Code Follows:
------------------------------------------------------------
sub createTeamOutput {
        ($CurrentOrganisation,$Team,$StartDate,$EndDate) = @_;
        my %CurrentTeamValues;

        # Need a list of all the ipaddresses in this team (but only this
organisation).
        print STDERR "select address from addressinfo,personinfo,subnetinfo
where team = '$Team' and addressinfo.person = personinfo.person and
organisation = '$CurrentOrganisation' and subnetinfo.subnet =
addressinfo.subnet\n";
        $sth = $dbh->prepare("select address from
addressinfo,personinfo,subnetinfo where team = '$Team' and addressinfo.person
= personinfo.person and organisation = '$CurrentOrganisation' and
subnetinfo.subnet = addressinfo.subnet");
        $sth->execute;
        my @Addresses;
        while(@currentAddress = $sth->fetchrow_array) {
                print STDERR "push \@Addresses, $currentAddress[0]\n";
                push @Addresses, $currentAddress[0];
        }
        foreach $address (sort byIpAddress @Addresses) {
                my %CurrentAddressValues;
                # Now here we need to get the total traffic for the address,
so we can check
                # that it is non-zero, like we do for the subnet
                print STDERR "\nCreating output for address $address...\n";
                print STDERR "select (sum(tooctets+fromoctets)/(1024*1024))
from NeTraMet_Daily1 where sourcepeeraddress = '$address' and (date >=
'$StartDate' and date <= '$EndDate') and not (type = 6 or type = 17)\n";
                $sth = $dbh->prepare("select
(sum(tooctets+fromoctets)/(1024*1024)) from NeTraMet_Daily1 where
sourcepeeraddress = '$address' and (date >= '$StartDate' and date <=
'$EndDate') and not (type = 6 or type = 17)");
                $sth->execute;
                @UnknownTotal = $sth->fetchrow_array;
                $CurrentAddressValues{'Unknown'} = $UnknownTotal[0];
                print STDERR "Address Unknown at
$CurrentAddressValues{'Unknown'}\t";
                $CurrentTeamValues{'Unknown'} += $UnknownTotal[0];
                print STEDRR "Team Unknown at $CurrentTeamValues{'Unknown'}\n";
                print STDERR "select (sum(tooctets+fromoctets)/(1024*1024))
from NeTraMet_Daily1 where sourcepeeraddress = '$address' and (date >=
'$StartDate' and date <= '$EndDate') and (type = 6 or type = 17)\n";
                $sth = $dbh->prepare("select
(sum(tooctets+fromoctets)/(1024*1024)) from NeTraMet_Daily1 where
sourcepeeraddress = '$address' and (date >= '$StartDate' and date <=
'$EndDate') and (type = 6 or type = 17)");
                $sth->execute;
                @IPTotal = $sth->fetchrow_array;
                $CurrentAddressValues{'IP'} = $IPTotal[0];
                $CurrentAddressValues{'Total'} = $CurrentAddressValues{'IP'} +
$CurrentAddressValues{'Unknown'};
                $CurrentTeamValues{'IP'} += $IPTotal[0];
                $CurrentTeamValues{'Total'} += $CurrentAddressValues{'IP'} +
$CurrentAddressValues{'Unknown'};
                print LOG "Address IP at $CurrentAddressValues{'IP'}\tAddress
Total at $CurrentAddressValues{'Total'}\tTeam IP at
$CurrentTeamValues{'IP'}\tTeam Total at $CurrentTeamValues{'Total'}\n";
                # Here we ignore the address if the value is zero,
                # we also want to ignore 'team 00' values of less than 5 MB
                next unless ($CurrentAddressValues{'Total'});
                

From douglas at katipo.co.nz  Thu Mar 13 16:49:02 2003
From: douglas at katipo.co.nz (Douglas Bagnall)
Date: Thu Aug  5 00:24:18 2004
Subject: script causing high cpu usage
In-Reply-To: <20030313215326.M25562@gamma.net.nz>
References: <20030313215326.M25562@gamma.net.nz>
Message-ID: <3E710ADE.2040102@katipo.co.nz>


hi Don,

You won't be getting more database connections than you ask for, so 
unless $dbh->connect() is being runs several times, that won't be the 
problem. One database handler is enough.

For the statement handlers, I believe perl's garbage handler will finish 
them up when (or soon after) they are no longer accessible (ie, when 
$sth is redefined for the next statement), but it is of course good 
practice to do it explicitly.

I suspect you will do better by adding indexes to the MySQL table - 
we've had hundredfold improvements for some things.

You'd probably want to look first at the NeTraMet_Daily1 table, because 
it's used in the inner loop. What you want is an index over the columns 
in the WHERE clause.

MySQL's EXPLAIN function is useful for this kind of thing.
see http://mysql.com/doc

Now a slightly irrelevent tip:

Don Jones wrote:
>         print STDERR "select address from addressinfo,personinfo,subnetinfo
> where team = '$Team' and addressinfo.person = personinfo.person and
> organisation = '$CurrentOrganisation' and subnetinfo.subnet =
> addressinfo.subnet\n";
>         $sth = $dbh->prepare("select address from
> addressinfo,personinfo,subnetinfo where team = '$Team' and addressinfo.person
> = personinfo.person and organisation = '$CurrentOrganisation' and
> subnetinfo.subnet = addressinfo.subnet");
>         $sth->execute;

I'd rewrite this as:

   my $query = "SELECT address FROM [etc]";
   warn $query;
   $sth = $dbh->prepare( $query );

so you know the print STDERR / warn is showing exactly what you're 
using. It's otherwise easy to change one and not the other and spend 
hours wondering what's happening.

Also, if you in anyway distrust your variables, you can replace '$Team' 
and '$CurrentOrganisation' with question marks, and put the variables in 
the execute call. Thus, (paraphrasing):

   my $query = "select x from y where team = ? and organisation = ?";
   $sth = $dbh->prepare( $query);
   $sth->execute( $Team, $CurrentOrganisation );

which just does the escaping of quotes for you, lest someone slip 
malicious strings in those variables.

hope this helps,

Douglas