[Pdx-pm] queueing, avoiding race conditions

Austin Schutz tex at off.org
Wed Aug 10 15:37:06 PDT 2005


On Wed, Aug 10, 2005 at 01:03:48PM -0700, Tom Phoenix wrote:
> Maybe there's a good reason not to do it this way, but it's simple and
> straightforward....
> 
> 1. Every request gets put into a queue.
> 
> The queue could be kept in a file (locked with flock), or in a
> database, or on a filesystem. (One simple implementation on a
> unix-like filesystem would be to write the request as a complete file
> in directory X, then hard-link the file into directory Y, which is the
> actual queue directory. By using the hard link, the complete request
> "appears" in the queue at once, avoiding race conditions.)
> 
> 2. A simple process drains the queue, one request at a time.
> 
> This process is started automatically just after an item is queued,
> unless there's already a process running.
> 
> 3. When the queue is empty, the process simply quits. Or it sleeps and
> awaits SIGCONT to be sent after an item is queued, perhaps.
> 
> > the bugbear of this problem seems to be race conditions.
> 
> If a single process drains the queue, there are relatively few race
> conditions to deal with.
> 
> Maybe you don't want the overhead of queueing for every item when
> there's usually no need for it. I can't blame you, but the simpler
> algorithm has fewer places for race conditions to hide.
> 
> Or have I missed something important? Hope this helps!
> 

	I would suggest the use of a lock file to make sure there is only
a single process draining the queue (perldoc flock).
	Even if you separate the queue flushing from the requesting code,
it's still handy to make sure you don't accidentally start two queue
processors. I do this with a bunch of my code, especially those things
which run from cron and I don't want to ever have a zillion copies running
at once.

	Austin


More information about the Pdx-pm-list mailing list