From jeremygwa at hotmail.com Sat Mar 12 17:27:09 2011 From: jeremygwa at hotmail.com (Jer A) Date: Sat, 12 Mar 2011 18:27:09 -0700 Subject: [VPM] sqlite and perl for performance In-Reply-To: References: Message-ID: I want to get data from multiple databases using sqlite. instead of processing command at a time, or a chain sequence, I would like to execute them in multiple threads, and have them all join. this is going to be used in a cgi script, and possibly a cron job. are there any issues using sqlite and perl in this manner? thanks in advance for your input. Regards, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From brendan at ming.org Sat Mar 12 18:46:11 2011 From: brendan at ming.org (Brendan) Date: Sat, 12 Mar 2011 18:46:11 -0800 Subject: [VPM] sqlite and perl for performance In-Reply-To: References:

Message-ID: <4D7C2FF3.1030404@ming.org> On 11-03-12 05:27 PM, Jer A wrote: > I want to get data from multiple databases using sqlite. > instead of processing command at a time, or a chain sequence, > I would like to execute them in multiple threads, and have them all join. > this is going to be used in a cgi script, and possibly a cron job. > are there any issues using sqlite and perl in this manner? it's probably safe, so long as you declare the dbi object inside each thread (if you declare it prior to starting up the worker threads, you could end up with problems). dbd::sqlite is probably thread safe, but if you are worried DBIx::Threaded purports to make any DBD module thread safe (though i'm sure you'll pay some price in memory usage). Threaded::Queue or Threaded::Queue::Duplex can help you get the data back to your master thread (instead of using a perl shared variable - the two queue modules use threadshared objects, but they are without any of the problems you might accidently cause yourself using :shared) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mock at obscurity.org Sat Mar 12 20:04:36 2011 From: mock at obscurity.org (will) Date: Sat, 12 Mar 2011 20:04:36 -0800 Subject: [VPM] sqlite and perl for performance In-Reply-To: <4D7C2FF3.1030404@ming.org> References:

<4D7C2FF3.1030404@ming.org> Message-ID: I've done this with the dbd::sqlite module before, but rather than using threads, I used POE and POE::Component::EasyDBI. If I recall correctly, you needed to be careful about sqlite locking the database file for write (I seem to recall implementing some retry logic to deal with that) and thus preventing other processes from writing at the same time. But that shouldn't be an issue for you if you're reading. mock On Sat, Mar 12, 2011 at 6:46 PM, Brendan wrote: > On 11-03-12 05:27 PM, Jer A wrote: > > I want to get data from multiple databases using sqlite. > instead of processing command at a time, or a chain sequence, > I would like to execute them in multiple threads, and have them all join. > this is going to be used in a cgi script, and possibly a cron job. > are there any issues using sqlite and perl in this manner? > > > it's probably safe, so long as you declare the dbi object inside each > thread (if you declare it prior to starting up the worker threads, you could > end up with problems). dbd::sqlite is probably thread safe, but if you are > worried DBIx::Threaded purports to make any DBD module thread safe (though > i'm sure you'll pay some price in memory usage). > > Threaded::Queue or Threaded::Queue::Duplex can help you get the data back > to your master thread (instead of using a perl shared variable - the two > queue modules use threadshared objects, but they are without any of the > problems you might accidently cause yourself using :shared) > > _______________________________________________ > Victoria-pm mailing list > Victoria-pm at pm.org > http://mail.pm.org/mailman/listinfo/victoria-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From darren at darrenduncan.net Sat Mar 12 21:36:43 2011 From: darren at darrenduncan.net (Darren Duncan) Date: Sat, 12 Mar 2011 21:36:43 -0800 Subject: [VPM] sqlite and perl for performance In-Reply-To: References:

Message-ID: <4D7C57EB.1050509@darrenduncan.net> Jer A wrote: > I want to get data from multiple databases using sqlite. > > instead of processing command at a time, or a chain sequence, > > I would like to execute them in multiple threads, and have them all join. > > this is going to be used in a cgi script, and possibly a cron job. > > are there any issues using sqlite and perl in this manner? > > thanks in advance for your input. Do you realize that you can use multiple SQLite databases simultaneously through a single DBI connection? You just CONNECT to one of them as usual and then issue "ATTACH" SQL to make others visible at the same time, as if they were parts of a larger database. You can even do cross-database queries, natively in SQLite; you don't have to join them in your application. -- Darren Duncan From szabgab at gmail.com Mon Mar 21 09:55:04 2011 From: szabgab at gmail.com (Gabor Szabo) Date: Mon, 21 Mar 2011 18:55:04 +0200 Subject: [VPM] LinuxFest Northwest 30 April - 1 May Message-ID: Hi, just a reminder to the PM groups in the neighborhood. LinuxFest Northwest is going to be between 30 April - 1 May in Bellingham, Washington. http://linuxfestnorthwest.org/ As I can see they are still accepting talk proposals and you can still ask for a table to present Perl or your Perl Monger group. As I just wrote to another PM group a booth is basically just a fixed place in the hallway track to attract discussions in a specific topic. So it is the same thing you would do at any other event just you don't have to move around so much. If you are attending please let other Perl Mongers know as going in a group might be more fun. regards Gabor -- Gabor Szabo? ? ? ? ? ? ? ? ? ?? http://szabgab.com/ Perl Ecosystem Group ? ? ? http://perl-ecosystem.org/